I am setting up a system where we will transport messages between several internal services on ServiceBus Topics. The messages will hold serialized objects. The model objects are defined as quite complex trees of classes. This means it is not practical to maintain duplet versions of the model structures in the code.
We expect the model structure to change so I have exposed the model version as a property on the brokered message.
What is the best way to handle the transition when we need to upgrade the model version?
I don't think we will really need to support two parallell model versions. But I am concerned we don't loose messages during the transition. I assume it is a good strategy to upgrade the sending services first and let all subscribers continue to process messages. When all messages of the previous version are processed, then it is time to upgrade the subscribing services.
What is the best mechanism for skipping messages with a new version that the listening service is currently not handling?
I know I could go back to the old school and define parallell model versions by using schemas for json or xml, thus making it possible for the listening service to handle parallell versions. But that would be cumbersome, so I really want to avoid that.
I noticed the BrokeredMessage has a Defer method. Would that be useful? It looked promising until I realized the messages will be "moved" from the live queue into a separate state where they need to be pulled by referencing them by key. Not practical.
Is it possible to postpone the message by modifying delivery time? A couple of minutes would be fine. If the same service is still running by that time it can be postponed once again. (A working code example would be appreciated!)
Do I need to create separate subscriptions based on model version? So far we allow different message types to travel on the same topic so that would call for some redesign.
As a rule of thumb: upgrades on a live system are difficult. The easiest option that minimises risks of system downtime is:
add next message version support to the current code base
run two message versions concurrently
ensure all versions are supported and system runs without a problem
remove previous version
I have been looking at something similar but am yet to implement it so can't provide full guidance, but to answer your question on #3... I have messages which have a flag to re-queue the message to run again, e.g. to get a process to run every 5 minutes.
So during the process I extract the object from the BrokeredMessage:
var myObject = receivedMessage.GetBody<MyModel>();
I then complete that message to remove it from the queue and create a new BrokeredMessage based on that object and you can then set the ScheduledEnqueueTimeUtc field to something in the future.
BrokeredMessage brokeredMsg = new BrokeredMessage(myObject);
brokeredMsg.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddMinutes(5);
Client.Send(brokeredMsg);
So if you only want to process one model version at a time, you could assign a version number to your Model and code something in to your processor to look for a certain model number. If the model is higher, then re-queue it for a future time (Until you have updated your code). If it is lower (a missed message), then perhaps have some exception handling.
Use custom MessageProperty on message, say Version.
Under SB topic - create new subscription that will accept only messages with new version (using Rules), and modify existing subscription(s) to NOT accept new version messages.
Then you can upgrade senders - new messages will be stored only in new 'temporary' subscription.
After that, you upgrade listeners, change rules on subscriptions (remove version rule from 'main' subscription, disable receive on temporary subscription).
And now you have choice:
using any tool read messages from temporary subscription and write them back to topic - they will arrive to upgraded listeners.
temporarily start one more listener that will read temporary subscription and process all messages in it
other ways, specific to your architecture
Related
I have a worker attached to a subscription processing messages. Depending on the message payload, the worker could possibly take days to completely process the message.
I'm a bit confused by different properties that the client can set that control how the Pub/Sub client library automatically extends the deadline so that the worker can process the message without fearing that the message will be redelivered.
Properties from documentation which is not very clear :
MinimumAckDeadline, MaximumAckDeadline, DefaultAckDeadline, MinimumAckExtensionWindow, DefaultAckExtensionWindow, MinimumLeaseExtensionDelay, and DefaultMaxTotalAckExtension
I believe I want to set DefaultMaxTotalAckExtension to a large value (several days) to allow my subscriber to continue working on the message without getting some kind of time out.
But I think I also want to modify the AckDeadline so that Pub/Sub knows that the client is still alive. Not sure which one I would want to modify: MinimumAckDeadline, MaximumAckDeadline, DefaultAckDeadline.
Aside from those properties, I don't know if I need to set MinimumAckExtensionWindow, DefaultAckExtensionWindow, or MinimumLeaseExtensionDelay.
All of the properties you mentioned are default/limit properties for the SubscriberClient class itself. Note that they are static only have getters, not setters. What you want to set is MaxTotalAckExtension, which controls the maximum amount of time a message's lease will be extended.
However, taking days to process a message is considered an anti-pattern for Cloud Pub/Sub and will very likely result in duplicate deliveries. If you are going to take that long to process a message, you probably need to look at other options like persisting it locally (in a file or in a database), acking it, and then processing it. At that point, you may consider just writing directly to a database instead of to Pub/Sub and scanning for rows that need to be processed by your subscribers.
We are in the process of migrating a legacy system to nservicebus 5.0. What is generally the best way to migrate our business data to saga data? For example, if we had an OrderCancellationPolicy saga, that only allowed cancellation within 2 days, how would past orders from the legacy system create these new sagas in the correct state?
I see two options. The first being to write some sql script to prepopulate the saga persistence tables (we are using nhibernate persistence). The other being to create some kind of special import message, such as MigrateOrderDataCmd, that contains the data from the legacy order. An import script could send out these messages which the sagas could handle and set the saga data that way.
Any guidance in this area is appreciated.
Theoretically I'd go for option two, or a version of it. Imagine your saga being down for a day and messages are piling up in the queue. With messages from a day coming in, you'd want to verify when the message was sent, or add a custom DateTime to the message yourself.
When the saga finally picks up the message, it knows it should not set the timeout two days in the future, but rather the time the message took to be delivered into the saga. That way when you migrate your current state, all messages get a proper timeout set.
On the other hand, if you've got a really large set of running processes, you might want to investigate how to propagate the tables. I have no hands on experience adding records to those tables myself, just changing them. :)
I am trying to create a message based application based with ActiveMQ, using .NET Clients.
Client 1: A Web Service (producer)
Client 2: A Windows Service (consumer)
My question is: Is it possible to prevent messages of a certain type or content from being queued by a Client?
The reason why I want to do this is Version Updating.
I think there will be a time, when I need to extend or change the message type.
My plan is to do that update in the following order:
Prevent messages of the old version to be queued.
Wait until the consumer has processed all messages of the old version.
Update producer and consumer software.
I would like the Web Service to be still available during the update process to report back to the call. But it should not be able to queue new messages.
Of course if there is a better way of solving this problem altogether, please let me know.
As a general rule it is a good idea to only have one type of payload per queue. An easy way to do this is to use two different queues for the two different message versions. Something like:
mysystem.orders.1_0
mysystem.orders.1_1
The version should be the last part of the queue name, as it makes it easy to work with wildcards, which are used for a lot of the config options in ActiveMQ.
Splitting up different versions into different queues gets you around the problem of having to upgrade the producer and consumer at the same time, and also gives you some visibility as whether all of the 1_0 messages have been consumed.
I have a scenario where about 10 different messages will need to be enqueued and then dequeued / processed. One subscriber will need all 10 messages, but another will only need 8 of the 10 messages. I am trying to understand what the best way is to setup this type of architecture. Do you create a queue for each message type so the subscriber(s) can just subscribe to the relevant queues or do you dump them all to the same queue and ignore the messages that are not relevant to that subscriber? I want to ensure the solution is flexible / scalable, etc.
Process:
10 different xml messages will be enqueued to an IBM WebSphere MQ server.
We will use .Net (Most likely WCF since WebSphere MQ 7.1 has added in WCF support)
We will dequeue the messages and load them into another backend DB (Most likely SQL Server).
Solution needs to scale well because we will be processing a very large number of messages and this could grow (Probably 40-50,000 / hr). At least large amount for us.
As always greatly appreciate the info.
--S
Creating queues is relatively 'cheap' from a resource perspective, plus yes, it's better to use a queue for each specific purpose, so it's probably better in this case to separate them by target client if possible. Using a queue to pull messages selectively based on some criteria (correlation ID or some other thing) is usually a bad idea. The best performing scenario in messaging is the most straightforward one: simply pull messages from the queue as they arrive, rather than peeking and receiving selectively.
As to scaling, I can't speak for Websphere MQ or other IBM products, but 40-50K messages per hour isn't particularly hard for MSMQ on Windows Server to handle, so I'd assume IBM can do that as well. Usually the bottleneck isn't the queuing platform itself but rather the process of dequeuing and processing individual messages.
OK, based on the comments, here's a suggestion that will scale and doesn't require much change on the apps.
On the producer side, I'd copy the message selection criteria to a message property and then publish the message to a topic. The only change that is required here to the app is the message property. If for some reason you don't want to make it publish using the native functionality, you can define an alias over a topic. The app thinks it is sending messages but they are really publications.
On the consumer side you have a couple of choices. One is to create administrative subscriptions for each app and use a selector in the subscription. The messages are then funneled to a dedicated queue per consumer, based on the selection criteria. The apps think that they are simply consuming messages.
Alternatively the app can simply subscribe to the topic. This gives you the option of a dynamic subscription that doesn't receive messages when the app is disconnected (if in fact you wanted that) or a durable subscription that is functionally equivalent to the administrative subscription.
This solution will easily scale to the volumes you cited. Another option is that the producer doesn't use properties. Here, the consumer application consumes all messages, breaks open the message payload on each and decides whether to process or ignore the message. In this solution the producer is still publishing to a topic. Any solution involving straight queueing forces the producer to know all the destinations. Add another consumer, change the producer. Also, there's a PUT for each destination.
The worst case is a producer putting multiple messages and a consumer having to read each one to decide if it's going to be ignored. That option might have problems scaling, depending on how deep in the payload the selection criteria field lies. Really long XPath expression = poor performance and no way to tune WMQ to make up for it since the latency is all in the application at that point.
Best case, producer sets a message property and publishes. Consumers select on property in their subscription or an administrative subscription does this for them. Whether this solution uses application subscriptions or administrative subscriptions doesn't make any difference as far as scalability is concerned.
Just wondering if Workflow Foundation is suitable for my problem.
The system will be used to process messages in XML format.
Based on message contents the system can send other messages to related systems or, in case of incomplete data, query them for additional information. Some messages have quite elaborate workflow with multiple states so this is why am I thinking of leveraging WF as then the message processing workflows would be easier to modify and understand due to visual workflow designer
In some cases missing data has to be entered by users manually. ASP.NET MVC application with EF will be used for that.
Is the WF suitable for this task?
I see these possible problems:
Getting all messages that are in certain state;
Editing the messages directly and notifying WF to resume the workflow, possibly even from different step;
WF should not persist the message content within its state and persist/load them from database when needed; In fact I do not think it should persist anything that cannot be changed by EF directly. I guess I would need custom persistence service for that, wouldn't this take too much effort?
Everything must be unit-testable.