I am working on two apps that use an MSMQ as a message bus mechanism so that A transfers messages to B. This clearly has to be robust so initially we chose MSMQ to store and transfer the messages.
When testing the app we noticed that in real-world conditions, where msmq is called to handle approximately 50.000 messages a minute (which sounds quite low to me) then we quickly reach the max storage size of the msmq /storage directory (defaults to 1.2gb i think).
We can increase that but I was wondering whether there is a better approach to handle slow receivers and fast senders. Is there a better queue or a better approach to use in this case?
Actually it isnt so much a problem of slow receivers since msmq will maintain the (received) messages in the storage dir for something like 6 hours or until the service is restarted. So essentially if in 5 minutes we reach the 1gb threshold then in a few hours we will reach terratybes of data!
Please read this blog to understand how MSMQ uses resources which I put together after years of supporting MSMQ at Microsoft.
It really does cover all the areas you need to know about.
If you have heard something about MSMQ that isn't in the blog then it is alomost certainly wrong - such as the 1.2GB storage limit for MSMQ. The maximum size of the msmq\storage directory is the hard disk capacity - it's an NTFS folder!
You should be able to have a queue with millions of messages in it (assuming you have enough kernel memory, as mentioned in the blog)
Cheers
John Breakwell
You should apply an SLA to your subscribers, they have to read their messages with in X amount of time or they lose them. You can scale this SLA to match the volume of messages that arrive.
For subscribers that cannot meet their SLA then simply put, they don't really care about receiving their messages that quickly (if they did, they would be available). For these subscribers you can offer a slower channel, such as an XML dump of the messages in the last hour (or what ever granularity is required). You probably wouldn't store each individual message here, but just an aggregate of changes (eg, something that can be queried from a DB).
Use separate queues for each message type, this way you can apply different priorities depending on the importance of the message, if one queue becomes full, messages of other types won't be blocked. It also makes it simpler to monitor if each message is being processed within its SLA by looking at the first message in the queue and seeing when it was added to determine how long it was waiting (see NServiceBus).
From your above metrics of 1GB in 5 minutes at 50,000 messages/minute I calculate each message to be about 4kb. This is quite large for a message since messages should normally only be carrying top level details about something happening, mostly IDs of what was changed, and the intent of what was changed. Larger data is better served from some other out-of-band channel for transferring large blobs (eg, file share, sftp, etc).
Also, since a service should encapsulate its own data, you shouldn't need to share much data between services. So large data within a service using messages to say what happened isn't unusual, large data between separate services using messages indicates that some boundaries are probably leaking.
Related
I'm writing an ASP .NET Core application where I'm using the Google PubSub emulator where I can both publishe and subscribe to a topic. However, when I publish a "large" amount of messages 1000+, I would like to pull as many as possible.
I use the Google.Cloud.PubSub.V1 library which provides SubscriberServiceApiClient to interact with their API. I pull asynchronously with the PullAsync method which has the parameter maxMessages. According to their documentation this decides the max number of messages that can be pulled by each request, however it may return fewer. If I provide an argument that specifies a maxMessages number above 100, it will not make a difference. This means the maximum number of messages I can receive from each request is always 100, which seems low. I've also tried to pull through their REST Api, which is also limited to 100 messages per pull.
I'm unsure whether it is due to some limit or if I'm doing something wrong. I have tried searching in their documentation and elsewhere, but without luck.
In general, Google Cloud Pub/Sub cannot return more than 1,000 messages to a single PullAsync call. This may be even smaller when running through the emulator. The value of returnImmediately can also affect how many messages are returned. If you want to maximize the number of messages returned, then you'll want to set returnImmediately to false. However, even in this scenario, you'll not necessarily get maxMessages in each response; Cloud Pub/Sub tries to balance returning fuller responses with minimizing end-to-end latency by waiting too long.
In general, to maximize throughput, you'll need to have multiple PullAsync calls active at once. However, even better is to use SubscriberClient, which handles the underlying requests behind the scenes for you and delivers messages to the function you specify as they arrive.
Max messages is still capped at 1,000 messages in November 2019. Pubsub does not allow to get more messages at a time. As seen in the picture below, I tried to pull messages in a loop, with 1,000 at a time. In half of the requests it gets a lot less than the maximum amount of messages. I managed to pull around 50,000 messages within the 9 minutes maximum runtime of a Cloud Function.
An alternative solution is async subscribing to a pubsub topic with google.cloud.pubsub_v1.SubscriberClient.subscribe(). However, this solution is better suited in a long running process which you could describe as a sort of collector sitting on a server.
I have a question.
I thought that Microsoft's Azure Queue was design to support a lot of messages as suggested by this post and I read you are able to store 100TB of messages (capacity max of an account). So you can use only one queue for all a worker role.
But I found an article which shows us only a queue by worker role instance.
I think the rest of the article is serious so I don't know what think about this.
Are multiple queues better in some cases?
If you're looking at storage it doesn't matter if you have 1 queue or 100, the limits are the same. But you have to know the performance implications. In a storage account performance is based on partitions (the Fabric Controller is able to assign more resources to a hot partition for example).
When you're working with Windows Azure Storage Queues, 1 queue is 1 partition. And this comes with some scalablity targets. For queues this means that 1 queue should be able to handle 500 messages / second. Read more about it here: Windows Azure Storage Abstractions and their Scalability Targets
Now this is an important factor for when you're defining your architecture and it depends on what you're going to use the queues for. If you never expect to reach 500 messages / second you should be fine with one queue for all your workers. But if you think to exceed this limit (even if it won't happen right away) you'll need to set up a good architecture to handle this. And this doesn't simply mean 1 queue for 1 worker. Think about different queues for different processes, splitting up processes in sub processes and use queues per subprocess, have your workers poll different queues, think of a round robin system, ...
Also consider looking at an existing implementation like this one:
Partitioned CloudQueue - Azure Storage Queue without scalability limits. I also suggest you look at the website of the Customer Advisory Team, they have a number of interesting articles regarding queue scalablity (the first article you referenced comes from the CAT website).
This is a question about message passing. This relates specifically to an in-house application written in C#. But it has a home grown "message passing" system resembling erlang.
Okay, we hope that it will be possible to learn from erlang folks or documentation to find an elegant solution to a couple of message passing challenges. But alas, after reading erlang documentation online and forums these topics don't seem to be addressed--that we can find.
So the question is: In erlang, when does the queue to send messages to a process get full? And does erlang handle the queue full situation? Or are the queues for message passing in erlang boundless--that is only limited by system memory?
Well in our system it involves processing a stream of financial data with potentially billions of tuples of information being read from disk, each tuple of financial information is called a "tick" in the financial world.
So it was necessary to set a limit to the queue size of each "process" in our system. We arbitrarily selected 1000 items max in each queue. Now those queues quickly get filled entirely by the tick messages.
The problem is that the processes also need to send other types of messages to each other besides just ticks but the ticks fill up the queues preventing any other types of message from getting passed.
As a "band aid" solution (which is messy) allow multiple queues per process for each message type. So a process will have a tick queue, and a command queue, and fill queue, and so on.
But erlang seems so much cleaner by having a single queue to each "process" that carries different message types. But again, how does it deal with the the queue getting hogged by a flood of only one of the message types?
So perhaps this is a question about the internals of erlang. Does erlang internally have separate limits on message types in a queue? Or does it internally have a separate queue per type of message?
In any case, how are sending processes aware when a queue is too full to receive certain types of message? Does the send fail? Does that mean error handling becomes necessary in erlang for inability to send?
In our system, it tracks when queues get full and then prevents any processes from running which will attempt to add to a full queue until that queue has more space. This avoids messy error handling logic since processes, once invoked, are guaranteed to have room to send one message.
But again, if we put multiple types of messages on that same queue. Other message types will be blocked that must get through.
It has become my perhaps mistaken impression that erlang wasn't designed to handle this situation so perhaps it doesn't address the problem of a queues getting filled with a flood of a single message type.
But we hope someone know how to answer this point to good reference information or book that covers this specific scenario.
Erlang sends all messages to a single queue with the system memory being the upper limit on the queue size. If you want to prioritize messages you have to scan the entire queue for the high priority messages before fetching a low priority one.
There are ways to get around this by spawning handler processes which throttle and prioritize traffic, but the erlang VM as such has no support for it.
Answer to the additional question in the comment:
Even at Safari books online, the main ones never say how messages are passed on erlang. It's clear they don't used "shared memory". So how do they communicate? is it via loopback tcp/ip when on the same machine?
Within one virtual machine, messages are simply copied (except for big enough binaries; for them, pointers are copied) between memory areas assigned to processes. If you start several Erlang VMs on the same machine, they can communicate over TCP/IP.
I have a scenario where about 10 different messages will need to be enqueued and then dequeued / processed. One subscriber will need all 10 messages, but another will only need 8 of the 10 messages. I am trying to understand what the best way is to setup this type of architecture. Do you create a queue for each message type so the subscriber(s) can just subscribe to the relevant queues or do you dump them all to the same queue and ignore the messages that are not relevant to that subscriber? I want to ensure the solution is flexible / scalable, etc.
Process:
10 different xml messages will be enqueued to an IBM WebSphere MQ server.
We will use .Net (Most likely WCF since WebSphere MQ 7.1 has added in WCF support)
We will dequeue the messages and load them into another backend DB (Most likely SQL Server).
Solution needs to scale well because we will be processing a very large number of messages and this could grow (Probably 40-50,000 / hr). At least large amount for us.
As always greatly appreciate the info.
--S
Creating queues is relatively 'cheap' from a resource perspective, plus yes, it's better to use a queue for each specific purpose, so it's probably better in this case to separate them by target client if possible. Using a queue to pull messages selectively based on some criteria (correlation ID or some other thing) is usually a bad idea. The best performing scenario in messaging is the most straightforward one: simply pull messages from the queue as they arrive, rather than peeking and receiving selectively.
As to scaling, I can't speak for Websphere MQ or other IBM products, but 40-50K messages per hour isn't particularly hard for MSMQ on Windows Server to handle, so I'd assume IBM can do that as well. Usually the bottleneck isn't the queuing platform itself but rather the process of dequeuing and processing individual messages.
OK, based on the comments, here's a suggestion that will scale and doesn't require much change on the apps.
On the producer side, I'd copy the message selection criteria to a message property and then publish the message to a topic. The only change that is required here to the app is the message property. If for some reason you don't want to make it publish using the native functionality, you can define an alias over a topic. The app thinks it is sending messages but they are really publications.
On the consumer side you have a couple of choices. One is to create administrative subscriptions for each app and use a selector in the subscription. The messages are then funneled to a dedicated queue per consumer, based on the selection criteria. The apps think that they are simply consuming messages.
Alternatively the app can simply subscribe to the topic. This gives you the option of a dynamic subscription that doesn't receive messages when the app is disconnected (if in fact you wanted that) or a durable subscription that is functionally equivalent to the administrative subscription.
This solution will easily scale to the volumes you cited. Another option is that the producer doesn't use properties. Here, the consumer application consumes all messages, breaks open the message payload on each and decides whether to process or ignore the message. In this solution the producer is still publishing to a topic. Any solution involving straight queueing forces the producer to know all the destinations. Add another consumer, change the producer. Also, there's a PUT for each destination.
The worst case is a producer putting multiple messages and a consumer having to read each one to decide if it's going to be ignored. That option might have problems scaling, depending on how deep in the payload the selection criteria field lies. Really long XPath expression = poor performance and no way to tune WMQ to make up for it since the latency is all in the application at that point.
Best case, producer sets a message property and publishes. Consumers select on property in their subscription or an administrative subscription does this for them. Whether this solution uses application subscriptions or administrative subscriptions doesn't make any difference as far as scalability is concerned.
I'm trying to design a system which reports activity events to a database via a web service. The web service and database have already been built (COTS software) - all I have to do is provide the event source.
The catch, though, is that the event source needs to be fault tolerant. We have multiple replicated databases that I can talk to, so if the web service or database I'm talking to goes down, the software can quickly switch to another one that's up.
What I need help with though is the case when all the databases are down. I've already designed a queue that will hold on to the events as they pile in (and burst them out once the connection is restored), but the queue is an in-memory structure: if my app crashes in this state, or if power is lost, etc., then all the events in the queue are lost. This is unacceptable. What I need is a way to persist the events so that when a database comes back online I can send a burst of queued-up events, even in the event of power loss or crash.
I know that I don't want to re-implement the queue itself to use the file system as a backing store. This would work (and I've tried it) - but that method slows the system down dramatically as the hard drive becomes a bottleneck. Aside from this though, I can't think of a single way to design this system such that all the events are safely stored on the hard drive only when access to the database isn't available.
Does anyone have any ideas? =)
When I need messaging with fault tolerance (and/or guaranteed delivery, which based on your description I am guessing you also need), I usually turn to MSMQ. It provides both fault tolerance (messages are stored on disk in case of machine restart) and guaranteed delivery (messages will automatically and continually resend until they are received), as well as transactional sends and receives, message journaling, poison message handling, and other features.
I have been able to achieve a throughput of several thousand messages per second using MSMQ. Frankly, I am not sure that you will get too much better than that while still being fault tolerant.
msmq. I think you could also take a look at the notion of Job object.
I would agree with guys that better to use out of the box system like MSMQ with a set of messaging patterns in hand.
Anyway, if you have to do it yourself, you can use in memory database instead of serializing data yourself, I believe it should be faster enough.