I juggled a bit with Greg Young's sample application and stumbled upon the problem in a multi-threadded environment that the order of messages in a bus might not be guaranteed or the processing of an event might not be completed before the next arrives.
Because of this, the ItemCreated message might occur after the ItemChangedSomething message, or at least the first message is not processed completely. This leads to problems in the "read side", because I want to update data that is not (yet) available.
How to go around this? (Assuming CQRS fits for the Domain Design case.)
Do I have to create a Saga or is there some other way of doing this?
You should choose a messaging infrastructure that guarantees delivery of events in-order on a per-consumer basis, even if multiple threads are delivering in parallel to different consumers. I.e., if you feed the events in order on the sending side, consumers will receive them in-order.
Then there are two basic approaches to handle this situation:
Infrastructure: in a small CQRS application without distributed data storage, you can record a global and increasing unique id for each event. Then make sure events are delivered by the messaging architecture in order of their id. That will completely eliminate out-of-order event delivery. Similarly you can record the time stamp of events and deliver them in order of their time stamp. While this might cause race conditions for some cases, for most applications and use cases time stamp based ordering is sufficient (in particular, if ItemCreated and ItemChanged are based on human actions).
State machines: For larger (typically distributed) setups, you can use an explicit or implicit automata/state machine model to cope for out-of-order arrival of messages. With a proper messaging infrastructure, you'll never receive ItemCreated and ItemChanged out of order if they originate from the same stream, but it might happen that events from two different sources (streams/aggregate roots) are consumed by some projection or saga in arbitrary order. Since these events are independent, there usually is a way (think state machine) to keep the projections in a valid state for either order.
Related
I have a number of domain events that can be dispatched in our enterprise system. For example if someone creates or removes an address.
Should i be passing the entire entity as part of the event or should i just pass the ID. The messages are sent via service bus and consumed in parallel.
If i just send the ID then the entity may not be available on the consumer side, if a delete happening in the meanwhile. I can always just use an active flag and set that to false but what about if the entity was updated meanwhile and it changed something important.
How would i go about handling these cases?
This I believe to be a common dilemma on a service bus, and I believe there is no one perfect solution. I'm assuming the scope here is JUST the Events raised when important domain objects change states (i.e. not Transactional Commands, nor Read / Data Services)
The decision on sending just Event metadata, versus sending a full Reference Message (e.g. a new Customer Aggregate Root) probably has wider implications than just on concurrency / versioning issues relating to latency, e.g. some pros and cons of either approach:
The minimal Event metadata:
Has a much smaller payload (especially useful if you audit all messages on the bus)
Fits nicely into standard envelopes
Is reasonably secure if a delivered to an unauthorised bus endpoint (all the system gets is the knowledge that Customer XYZ has changed, not the actual details).
Whereas a full "aggregate" root Message reference update
Can be overkill if most subscribers aren't interested in the full payload.
Potential security concerns - not all subscribers on the bus may be entitled to the full payload
But is great for replenishing CQRS readstore caches, as endpoints don't need to go back to the source of truth to fetch data once they know their data is out of date - the data has already been provided.
So I guess the final decision will go with what you primarily intend doing with your EDA Events (Keeping CQRS caches updated vs Triggering BPM Workflows vs Monitoring CEP Rules etc). You might decide to go with a hybrid e.g. broadcast Event Data widely, but then route full Messages to only trusted endpoints (The event meta data can likely be projected from the full payload, so the Originating / Source of Truth system can just send one message payload to the bus after each state change).
To answer your data consistency question, I believe you will need to accept that the data will only be Eventually Consistent, and that latencies will cause temporary inconsistencies across the enterprise. I believe the best pattern here is to add a hash or timestamp to each Message obtained from the originating Source of Truth, which needs to be added to any Commands which have used this version of the data as an assumption.
Then, when the command handling system processes the command, it can then check this hash against the current 'true' version (based on the actual line of business system database, NOT against a readstore Cache), and will need to fail the command if the hash / timestamps do not match up - i.e. the optimistic concurrency pattern.
I'm trying to figure out how my event storage and my read model are related in terms of actual nuts and bolts implementations.
My limited understanding of the event store leads me to believe:
Event is committed to event store
Dispatcher runs
If I'm using a queue, I send the message to a queue (lets say mass transit)
My read model is subscribed to the queue, so my read database gets the message (mysql)
My read model is updated with the new change to my data
This would mean that if anything happened to mass transit, my read database will be out of sync and I have to figure out how to sync it back.
Some stuff I've read/watched that's been published by greg young suggest using the event store itself as a queue, and maintain consistency by keeping an auto increment number on the event store side in order to maintain eventual consistency. I'm wondering if that is implemented in joliver's project?
so my read database gets the message (mysql)
I'd re-state that as "my event processor(s) for a given event get the message and (in my case) will typically manipulate state in a mysql database" (Or do you mean something else?).
This would mean that if anything happened to mass transit, my read database will be out of sync and I have to figure out how to sync it back.
Yes, your queue becomes part of the state of your app and it needs to be backed up and resilient. Note that the Dispatcher does not mark the Commit dispatched until it has successfully put it onto the Queue, and the queuing system won't remove the message until you've confirmed completion of the processing to do the necessary updates to sync the state in your Read Model.
Remember that you can consider multiple web service calls to all be part of the necessary work to process an event.
The other thing to bear in mind is that you'll want to have your event processors be idempotent (i.e. be able to handle At Least Once delivery).
Further down the line, you'll have fun considering what you're going to do if an event cannot complete processing - are you going to Dead Letter the message? Who is going to monitor that?
BTW depending on your hosting arrangements, Azure (or the on-premise Windows) ServiceBus might be worth considering)
Some stuff I've read/watched that's been published by greg young suggest using the event store itself as a queue, and maintain consistency by keeping an auto increment number on the event store side in order to maintain eventual consistency. I'm wondering if that is implemented in joliver's project?
No, JOES provides you a Dispatcher hook and you decide what's right for you after that. This is good and bad. There are systems that don't have a Dispatcher tied to a stateful Read Model at all - they simply query the Event Store for events and build an in-memory Read Model to short circuit all this.
Not sure what you mean by auto increment numbers.
Beware that the Projection stuff in the GES is not fully 1.0 yet (but it goes without saying its extremely deserving of your strong consideration - it intrinsically deals with the bulk of the concerns you're touching on with these questions)
At our organization we have a SQL Server 2005 database and a fair number of database clients: web sites (php, zope, asp.net), rich clients (legacy fox pro). Now we need to pass certain events from the core database with other systems (MongoDb, LDAP and others). Messaging paradigm seems pretty capable of solving this kind of problem. So we decided to use RabbitMQ broker as a middleware.
The problem of consuming events from the database at first seemed to have only two possible solutions:
Poll the database for outgoing messages and pass them to a message broker.
Use triggers on certain tables to pass messages to a broker on the same machine.
I disliked the first idea due to latency issues which arise when periodical execution of sql is involved.
But event-based trigger approach has a problem which seems unsolvable to me at the moment. Consider this scenario:
A row is inserted into a table.
Trigger fires and sends a message (using a CLR Stored Procedure written in C#)
Everything is ok unless transaction which writes data is rolled back. In this case data will be consistent, but the message has already been sent and cannot be rolled back because trigger fires at the moment of writing to the database log, not at the time of transaction commit (which is a correct behaviour of a RDBMS).
I realize now that I'm asking too much of triggers and they are not suitable for tasks other than working with data.
So my questions are:
Has anyone managed to extract data events using triggers?
What other methods of consuming data events can you advise?
Is Query Notification (built on top of Service Broker) suitable in my situation?
Thanks in advance!
Lest first cut out of the of the equation the obvious misfit: Query Notification is not right technology for this, because is designed to address cache invalidation of relatively stable data. With QN you'll only know that table has changed, but you won't be able to know what had changed.
Kudos to you for figuring out why triggers invoking SQLCRL won't work: the consistency is broken on rollback.
So what does work? Consider this: BizTalk Server. In other words, there is an entire business built around this problem space, and solutions are far from trivial (otherwise nobody would buy such products).
You can get quite far though following a few principles:
decoupling. Event based triggers are OK, but do not send the message from the trigger. Aside from the consistency issue on rollback you also have the latency issue of having every DML operation now wait for an external API call (the RabbitMQ send) and the availability issue of the external API call failure (if RabbitMQ is unavailable, your DB is unavailable). The solution is to have the trigger use ordinary tables as queues, the trigger will enqueue a message in the local db queue (ie. will insert into this table) and and external process will service this queue by dequeueing the messages (ie. delete from the table) and forwarding them to RabbitMQ. This decouples the transaction from the RabbitMQ operation (the external process is able to see the message only if the original xact commits), but the cost is some obvious added latency (there is an extra hop involved, the local table acting as a queue).
idempotency. Since RabbitMQ cannot enroll in distributed transactions with the database you cannot guarantee atomicity of the DB operation (the dequeue from local table acting as queue) and the RabbitMQ operation (the send). Either one can succeed when the other failed, and there is simply no way around it w/o explicit distributed transaction enrollment support. Which implies that the application will send duplicate messages every once in a while (usually when things already go bad for some reason). And a quick heads up: enrolling into the act of explicit 'acknowledge' messages and send sequence numbers is a loosing battle as you'll quickly discover that you're reinventing TCP on top of messaging, that road is paved with bodies.
tolerance. For the same reasons as the item above every now in a while a message you believe was sent will never make it. Again, what damage this causes is entirely business specific. The issue is not how to prevent this situation (is almost impossible...) but how to detect this situation, and what to do about it. No silver bullet, I'm afraid.
You do mention in passing Service Broker (the fact that is powering Query Notification is the least interestign aspect of it...). As a messaging platform built into SQL Server which offers Exactly Once In Order delivery guarantees and is fully transacted it would solve all the above pain points (you can SEND from triggers withouth impunity, you can use Activation to solve the latency issue, you'll never see a duplicate or a missing message, there are clear error semantics) and some other pain points I did not mention before (consistency of backup/restore as the data and the messages are on the same unit of storage - the database, cosnsitnecy of HA/DR failover as SSB support both database mirroring and clustering etc). The draw back though is that SSB is only capable of talking to another SSB service, in other words it can only be used to exchange messages between two (or more) SQL Server instances. Any other use requires the parties to use a SQL Server to exchange messages. But if your endpoints are all SQL Server, then consider that there are some large scale deployments using Service Broker. Note that endpoints like php or asp.net can be considered SQL Server endpoints, they are just programming layers on top of the DB API, a different endpoint would, say, the need to send messages from handheld devices (phones) directly to the database (and eve those 99% of the time go through a web service, which means they can reach a SQL Server ultimately). Another consideration is that SSB is geared toward throughput and reliable delivery, not toward low latency. Is definitely not the technology to use to get back the response in a HTTP web request, for instance. IS the technology to use to submit for processing something triggered by a web request.
Remus's answer lays out some sound principals for generating and handling events. You can initiate the pushing of events from a trigger to achieve low latency.
You can achieve everything necessary from a trigger. We will still decouple this into two components: a trigger that generates the events and a local reader that reads the events.
The first component is the trigger.
Make a CLR trigger that prepares what needs to be done when the transaction commits.
Create a System.Transactions.IEnlistmentNotification that always agrees to be prepared, and whose void Commit(System.Transactions.Enlistment) method executes the prepared action.
In the trigger, call System.Transactions.Transaction.Current.EnlistVolatile(enlistmentNotification, System.Transactions.EnlistmentOptions.None)
You'll want your action to be short and sweet, like appending the data to a lockless queue in memory or updating some other state in memory. Don't try to communicate with other machines or processes. Don't write to a disk (if you wanted to write to a disk, just make an ordinary trigger that inserts into a queue table). You'll need to be careful to make sure your assembly is loaded only once so that any shared static state will be unique; this is easiest to do if your static state is in a top level assembly that isn't referenced by other assemblies, so no other assemblies will try to load it.
You will also need to either
initialize your state in such a way that it will be correct even if the system was restarted without sending all the previously queued messages (since a short, in memory queue will not be durable). This means you might be resending messages, so they will need to be idempotent. or
rely on the tolerance of another component to pick up on missed messages
The second component reads the state that is update by the trigger. Make a separate CLR component that reads from your queue or state, and does whatever you need done (like send an idempotent message to a messaging system, record that it was sent, whatever). If this component can fail (hint: it can), you will need some form of tolerance, which may belong in another system. You can achieve low latency by having the trigger signal the second component when new state is available.
One architectural possibility is to have the trigger put the event in memory on commit for another low-latency component to pick up and have the second component send a low-latency, low-reliability copy of an idempotent message. You can pair that with a more reliably or durable messaging system, such as SSB, that will reliably and durably, but with grater latency, send the same idempotent message later.
I have a scenario where about 10 different messages will need to be enqueued and then dequeued / processed. One subscriber will need all 10 messages, but another will only need 8 of the 10 messages. I am trying to understand what the best way is to setup this type of architecture. Do you create a queue for each message type so the subscriber(s) can just subscribe to the relevant queues or do you dump them all to the same queue and ignore the messages that are not relevant to that subscriber? I want to ensure the solution is flexible / scalable, etc.
Process:
10 different xml messages will be enqueued to an IBM WebSphere MQ server.
We will use .Net (Most likely WCF since WebSphere MQ 7.1 has added in WCF support)
We will dequeue the messages and load them into another backend DB (Most likely SQL Server).
Solution needs to scale well because we will be processing a very large number of messages and this could grow (Probably 40-50,000 / hr). At least large amount for us.
As always greatly appreciate the info.
--S
Creating queues is relatively 'cheap' from a resource perspective, plus yes, it's better to use a queue for each specific purpose, so it's probably better in this case to separate them by target client if possible. Using a queue to pull messages selectively based on some criteria (correlation ID or some other thing) is usually a bad idea. The best performing scenario in messaging is the most straightforward one: simply pull messages from the queue as they arrive, rather than peeking and receiving selectively.
As to scaling, I can't speak for Websphere MQ or other IBM products, but 40-50K messages per hour isn't particularly hard for MSMQ on Windows Server to handle, so I'd assume IBM can do that as well. Usually the bottleneck isn't the queuing platform itself but rather the process of dequeuing and processing individual messages.
OK, based on the comments, here's a suggestion that will scale and doesn't require much change on the apps.
On the producer side, I'd copy the message selection criteria to a message property and then publish the message to a topic. The only change that is required here to the app is the message property. If for some reason you don't want to make it publish using the native functionality, you can define an alias over a topic. The app thinks it is sending messages but they are really publications.
On the consumer side you have a couple of choices. One is to create administrative subscriptions for each app and use a selector in the subscription. The messages are then funneled to a dedicated queue per consumer, based on the selection criteria. The apps think that they are simply consuming messages.
Alternatively the app can simply subscribe to the topic. This gives you the option of a dynamic subscription that doesn't receive messages when the app is disconnected (if in fact you wanted that) or a durable subscription that is functionally equivalent to the administrative subscription.
This solution will easily scale to the volumes you cited. Another option is that the producer doesn't use properties. Here, the consumer application consumes all messages, breaks open the message payload on each and decides whether to process or ignore the message. In this solution the producer is still publishing to a topic. Any solution involving straight queueing forces the producer to know all the destinations. Add another consumer, change the producer. Also, there's a PUT for each destination.
The worst case is a producer putting multiple messages and a consumer having to read each one to decide if it's going to be ignored. That option might have problems scaling, depending on how deep in the payload the selection criteria field lies. Really long XPath expression = poor performance and no way to tune WMQ to make up for it since the latency is all in the application at that point.
Best case, producer sets a message property and publishes. Consumers select on property in their subscription or an administrative subscription does this for them. Whether this solution uses application subscriptions or administrative subscriptions doesn't make any difference as far as scalability is concerned.
I'm trying to design a system which reports activity events to a database via a web service. The web service and database have already been built (COTS software) - all I have to do is provide the event source.
The catch, though, is that the event source needs to be fault tolerant. We have multiple replicated databases that I can talk to, so if the web service or database I'm talking to goes down, the software can quickly switch to another one that's up.
What I need help with though is the case when all the databases are down. I've already designed a queue that will hold on to the events as they pile in (and burst them out once the connection is restored), but the queue is an in-memory structure: if my app crashes in this state, or if power is lost, etc., then all the events in the queue are lost. This is unacceptable. What I need is a way to persist the events so that when a database comes back online I can send a burst of queued-up events, even in the event of power loss or crash.
I know that I don't want to re-implement the queue itself to use the file system as a backing store. This would work (and I've tried it) - but that method slows the system down dramatically as the hard drive becomes a bottleneck. Aside from this though, I can't think of a single way to design this system such that all the events are safely stored on the hard drive only when access to the database isn't available.
Does anyone have any ideas? =)
When I need messaging with fault tolerance (and/or guaranteed delivery, which based on your description I am guessing you also need), I usually turn to MSMQ. It provides both fault tolerance (messages are stored on disk in case of machine restart) and guaranteed delivery (messages will automatically and continually resend until they are received), as well as transactional sends and receives, message journaling, poison message handling, and other features.
I have been able to achieve a throughput of several thousand messages per second using MSMQ. Frankly, I am not sure that you will get too much better than that while still being fault tolerant.
msmq. I think you could also take a look at the notion of Job object.
I would agree with guys that better to use out of the box system like MSMQ with a set of messaging patterns in hand.
Anyway, if you have to do it yourself, you can use in memory database instead of serializing data yourself, I believe it should be faster enough.