Develop Reference

RabbitMQ async Request-Reply pattern, client side, .NET C#

I'm kinda new to RabbitMQ, so please bear with me. I use standard RabbitMQ package: https://www.rabbitmq.com/dotnet.html
When you implement client side Request-Reply pattern, you use same methods as usual. Message consumption:
_consumer.Received += (model, ea) =>
{
var reply = JsonSerializer.Deserialize<TReply>
(
Encoding.UTF8.GetString(ea.Body.ToArray())
);
/* Consume the message */;
};
_channel.BasicConsume
(
queue: _replyQueue.QueueName,
autoAck: true,
consumer: _consumer
);
Message publishment:
var properties = _channel.CreateBasicProperties();
properties.ReplyTo = _replyQueue.QueueName;
properties.CorrelationId = Guid.NewGuid().ToString();
var body = Encoding.UTF8.GetBytes
(
JsonSerializer.Serialize(request)
);
_channel.BasicPublish("", _requestQueue.QueueName, properties, body);
So, message publishment and message consumption are still implemented in an independent way, however, in addition, you have properties.CorrelationId property which allows you to connect reply to its request on your own. I feel like this is too low-level approach, which shouldn't be used directly on the application level. Instead, this should be wrapped into some higher-level library client. On the application level I want to see something like this:
var client = MyRabbitMQRequestReplyClient();
var reply = await client.RequestAsync(request);
All request-reply CorrelationId-matching, as I think, should be hidden from the application-level developer.
This approach may look kinda unsafe, because we wait in a sequential way until the reply is received insted of doing everything in parallel. But sometimes we really want exactly such a behavior. For example, when a user clicks a button, and the client requests for something on a remote service, and once a reply is received, a user should get this reply right away, and this user is OK to wait for this reply - it feels like a natural behavior for this particular UI.
So, the questions are: is this a good approach for the cases when we really want to wait for the reply? If it's not - then why? What is the better approach? I don't want to mess with CorrelationId-matching every time I do request-reply on the application level. If this approach is good - then what is the best way to implement it?

If you're using something like direct reply-to, then this would be useful. I'd hesitate to do this approach if the reply queue was persistent or shared, because you want to ensure that only that one client will get the response.
The usual solution is to have a concurrent dictionary mapping from id to TaskCompletionSource<TReply>. When sending a new request, insert a new item in that dictionary and then return the TCS's Task property. When replies come in, retrieve the TCS from the dictionary and complete it.

Azure Service bus - pass parameter to message handler

In Azure Service Bus I need to listen for messages arriving from multiple subscriptions from different services busses at once.
To do this I created a list that contains objects with a connection string, a topic, a subscription name and some other information (the list is called 'jobs').
For each item in this list I am then creating a different task that creates the ServiceBusClient and the processor.
var jobs = GetAllServiceBusTopics();
Parallel.ForEach(jobs, async job =>
{
var client = new ServiceBusClient(job.Environment.ServiceBusConnectionString);
var options = new ServiceBusProcessorOptions();
var processor = client.CreateProcessor(job.Environment.TopicName, _subscriptionName, new ServiceBusProcessorOptions());
try
{
processor.ProcessMessageAsync += MessageHandler;
//Pass the job object somehow to the "MessageHandler" below.
processor.ProcessErrorAsync += ErrorHandler;
await processor.StartProcessingAsync();
Console.WriteLine("Wait for a minute and then press any key to end the processing");
Console.ReadKey();
Console.WriteLine("\nStopping the receiver...");
await processor.StopProcessingAsync();
Console.WriteLine("Stopped receiving messages");
}
finally
{
await processor.DisposeAsync();
await client.DisposeAsync();
}
});
And the handler that is called if a new message arrives:
static async Task MessageHandler(ProcessMessageEventArgs args)
{
//I need the "job" object from my loop above here.
}
How the concept generally works I learned on this website of Microsoft.
My first question:
Is this approach okay, or am I running in the wrong direction? Can I do it like this?
But even if this is okay, I have another more important task:
I need to pass the "job" object from my loop somehow to the message handler - as a parameter.
But I have currently no idea how to archvie this. Any proposals on this?

Is this approach okay, or am I running in the wrong direction? Can I do it like this?
Yes, you can do this. One thing to keep in mind is that you instantiate multiple ServiceBusClient instances, each causing a new connection to be established rather than using the same connection. I don't know how big the number of topics (jobs) might be but if it's large, you'll end up with connections starvation.
I need to pass the "job" object from my loop somehow to the message handler - as a parameter. But I have currently no idea how to archvie this. Any proposals on this?
That's not how ServiceBusProcessor is designed. It doesn't receive anything other than the incoming message that needs to be processed. If you need to have a job ID, that should be part of the message payload/metadata. If you need to know the entity it arrived from, you could add a subscription filter action to add a custom header with the identifier. An alternative approach would require wrapping the ServiceBusProcessor to retain the job ID/subscription identifier and use that in the event handler.

Observable stream from StackExchange Redis Pub Sub subscription

OBJECTIVE:
I am using StackExchange Redis Client. My objective is to create an Observable stream from the Pub Sub Subscriber exposed by the Client, that can then in turn support 1-n subscriptions by Observables, that each have their own filter via LINQ. (Publishing is working as planned, the issue is purely around Subscription to the Event Stream on a specific channel.)
BACKGROUND:
I am using Redis Pub Sub as part of an Event Sourced CQRS application. The specific use case is to publish Events to multiple subscribers that then update the various Read Models, send emails etc.
Each of these Subscribers need to filter the Event Types that they handle, and for this I am looking to use Rx .Net (Reactive Extensions), with LINQ, to
provide a filter criteria on the Event Stream, to efficiently handle reacting only to Events of interest. Using this approach removes the need for registering Handlers with an event bus implementation, and allows me to add new projections to the system by deploying 1-n Microservices that each have 1-n Observables subscribed to the Event Stream with their own specific filters.
WHAT I HAVE TRIED:
1) I have created a class inheriting from ObservableBase, overriding the SubscribeCore method, which receives subscription requests from Observables, stores them in a ConcurrentDictionary, and as each Redis notification arrives from the channel, loops through the registered Observable subscribers and calls their OnNext method passing the RedisValue.
2) I have created a Subject, that also accepts subscriptions from Observables, and calls their OnNext method. Again, the use of Subjects appears to be frowned upon by many.
THE ISSUE:
The approaches I have tried do function (at least superficially), with varying levels of performance, but feel like a hack, and that I am not using Rx in the way it was intended.
I see many comments that the built-in Observable methods should be used where at all possible, for example Observable.FromEvent, but that seems to be impossible to do with the StackExchange Redis Clients Subscription API, at least to my eyes.
I also understand that the preferred method for receiving a stream and forwarding to multiple Observers is to use a ConnectableObservable, which would seem to be designed for the very scenario I face (Each Microservice will internally have 1-n Observables subscribed). At the moment, I cannot get my head around how to connect a ConnectableObservable to the notifications from StackExchange Redis, or if it offers real benefit over an Observable.
UPDATE:
Although completion is not an issue in my scenario (Disposal is fine), error handling is important; e.g. isolating errors detected in one subscriber to prevent all subscriptions terminating.

Here is an extension method you can use to create an IObservable<RedisValue> from an ISubscriber and a RedisChannel:
public static IObservable<RedisValue> WhenMessageReceived(this ISubscriber subscriber, RedisChannel channel)
{
return Observable.Create<RedisValue>(async (obs, ct) =>
{
// as the SubscribeAsync callback can be invoked concurrently
// a thread-safe wrapper for OnNext is needed
var syncObs = Observer.Synchronize(obs);
await subscriber.SubscribeAsync(channel, (_, message) =>
{
syncObs.OnNext(message);
}).ConfigureAwait(false);
return Disposable.Create(() => subscriber.Unsubscribe(channel));
});
}
As there is no completion of Redis channels the resulting IObservable will never complete, however you may drop the IDisposable subscription to unsubscribe from the Redis channel (this will be done automatically by many Rx operators).
Usage could be like so:
var subscriber = connectionMultiplexer.GetSubscriber();
var gotMessage = await subscriber.WhenMessageReceived("my_channel")
.AnyAsync(msg => msg == "expected_message")
.ToTask()
.ConfigureAwait(false);
Or as per your example:
var subscriber = connectionMultiplexer.GetSubscriber();
var sendEmailEvents = subscriber.WhenMessageReceived("my_channel")
.Select(msg => ParseEventFromMessage(msg))
.Where(evt => evt.Type == EventType.SendEmails);
await sendEmailEvents.ForEachAsync(evt =>
{
SendEmails(evt);
}).ConfigureAwait(false);
Other microservices may filter differently.

Serial processing of a certain message type in Rebus

We have a Rebus message handler that talks to a third party webservice. Due to reasons beyond our immediate control, this WCF service frequently throws an exception because it encountered a database deadlock in its own database. Rebus will then try to process this message five times, which in most cases means that one of those five times will be lucky and not get a deadlock. But it frequently happens that a message does get deadlock after deadlock and ends up in our error queue.
Besides fixing the source of the deadlocks, which would be a longterm goal, I can think of two options:
Keep trying with only this particular message type until it succeeds. Preferably I would be able to set a timeout, so "if five deadlocks then try again in 5 minutes" rather than choke the process up even more by trying continuously. I already do a Thread.Sleep(random) to spread the messages somewhat, but it will still give up after five tries.
Send this particular message type to a different queue that has only one worker that processes the message, so that this happens serially rather than in parallel. Our current configuration uses 8 worker threads, but this just makes the deadlock situation worse as the webservice now gets called concurrently and the messages get in each other's way.
Option #2 has my preference, but I'm not sure if this is possible. Our configuration on the receiving side currently looks like this:
var adapter = new Rebus.Ninject.NinjectContainerAdapter(this.Kernel);
var bus = Rebus.Configuration.Configure.With(adapter)
.Logging(x => x.Log4Net())
.Transport(t => t.UseMsmqAndGetInputQueueNameFromAppConfig())
.MessageOwnership(d => d.FromRebusConfigurationSection())
.CreateBus().Start();
And the .config for the receiving side:
<rebus inputQueue="app.msg.input" errorQueue="app.msg.error" workers="8">
<endpoints>
</endpoints>
</rebus>
From what I can tell from the config, it's only possible to set one input queue to 'listen' to. I can't really find a way to do this via the fluent mapping API either. That seems to take only one input- and error queue as well:
.Transport(t =>t.UseMsmq("input", "error"))
Basically, what I'm looking for is something along the lines of:
<rebus workers="8">
<input name="app.msg.input" error="app.msg.error" />
<input name="another.input.queue" error="app.msg.error" />
</rebus>
Any tips on how to handle my requirements?

I suggest you make use of a saga and Rebus' timeout service to implement a retry strategy that fits your needs. This way, in your Rebus-enabled web service facade, you could do something like this:
public void Handle(TryMakeWebServiceCall message)
{
try
{
var result = client.MakeWebServiceCall(whatever);
bus.Reply(new ResponseWithTheResult{ ... });
}
catch(Exception e)
{
Data.FailedAttempts++;
if (Data.FailedAttempts < 10)
{
bus.Defer(TimeSpan.FromSeconds(1), message);
return;
}
// oh no! we failed 10 times... this is probably where we'd
// go and do something like this:
emailService.NotifyAdministrator("Something went wrong!");
}
}
where Data is the saga data that is made magically available to you and persisted between calls.
For inspiration on how to create a saga, check out the wiki page on coordinating stuff that happens over time where you can see an example on how a service might have some state (i.e. number of failed attempts in your case) stored locally that is made available between handling messages.
When the time comes to make bus.Defer work, you have two options: 1) use an external timeout service (which I usually have installed one of on each server), or 2) just use "yourself" as a timeout service.
At configuration time, you go
Configure.With(...)
.(...)
.Timeouts(t => // configure it here)
where you can either StoreInMemory, StoreInSqlServer, StoreInMongoDb, StoreInRavenDb, or UseExternalTimeoutManager.
If you choose (1), you need to check out the Rebus code and build Rebus.Timeout yourself - it's basically just a configurable, Topshelf-enabled console application that has a Rebus endpoint inside.
Please let me know if you need more help making this work - bus.Defer is where your system becomes awesome, and will be capable of overcoming all of the little glitches that make all others' go down :)

Bloomberg API request timing out

Having set up a ReferenceDataRequest I send it along to an EventQueue
Service refdata = _session.GetService("//blp/refdata");
Request request = refdata.CreateRequest("ReferenceDataRequest");
// append the appropriate symbol and field data to the request
EventQueue eventQueue = new EventQueue();
Guid guid = Guid.NewGuid();
CorrelationID id = new CorrelationID(guid);
_session.SendRequest(request, eventQueue, id);
long _eventWaitTimeout = 60000;
myEvent = eventQueue.NextEvent(_eventWaitTimeout);
Normally I can grab the message from the queue, but I'm hitting the situation now that if I'm making a number of requests in the same run of the app (normally around the tenth), I see a TIMEOUT EventType
if (myEvent.Type == Event.EventType.TIMEOUT)
throw new Exception("Timed Out - need to rethink this strategy");
else
msg = myEvent.GetMessages().First();
These are being made on the same thread, but I'm assuming that there's something somewhere along the line that I'm consuming and not releasing.
Anyone have any clues or advice?
There aren't many references on SO to BLP's API, but hopefully we can start to rectify that situation.

I just wanted to share something, thanks to the code you included in your initial post.
If you make a request for historical intraday data for a long duration (which results in many events generated by Bloomberg API), do not use the pattern specified in the API documentation, as it may end up making your application very slow to retrieve all events.
Basically, do not call NextEvent() on a Session object! Use a dedicated EventQueue instead.
Instead of doing this:
var cID = new CorrelationID(1);
session.SendRequest(request, cID);
do {
Event eventObj = session.NextEvent();
...
}
Do this:
var cID = new CorrelationID(1);
var eventQueue = new EventQueue();
session.SendRequest(request, eventQueue, cID);
do {
Event eventObj = eventQueue.NextEvent();
...
}
This can result in some performance improvement, though the API is known to not be particularly deterministic...

I didn't really ever get around to solving this question, but we did find a workaround.
Based on a small, apparently throwaway, comment in the Server API documentation, we opted to create a second session. One session is responsible for static requests, the other for real-time. e.g.
_marketDataSession.OpenService("//blp/mktdata");
_staticSession.OpenService("//blp/refdata");
The means one session operates in subscription mode, the other more synchronously - I think it was this duality which was at the root of our problems.
Since making that change, we've not had any problems.

My reading of the docs agrees that you need separate sessions for the "//blp/mktdata" and "//blp/refdata" services.

A client appeared to have a similar problem. I solved it by making hundreds of sessions rather than passing in hundreds of requests in one session. Bloomberg may not be to happy with this BFI (brute force and ignorance) approach as we are sending the field requests for each session but it works.

Nice to see another person on stackoverflow enjoying the pain of bloomberg API :-)
I'm ashamed to say I use the following pattern (I suspect copied from the example code). It seems to work reasonably robustly, but probably ignores some important messages. But I don't get your time-out problem. It's Java, but all the languages work basically the same.
cid = session.sendRequest(request, null);
while (true) {
Event event = session.nextEvent();
MessageIterator msgIter = event.messageIterator();
while (msgIter.hasNext()) {
Message msg = msgIter.next();
if (msg.correlationID() == cid) {
processMessage(msg, fieldStrings, result);
}
}
if (event.eventType() == Event.EventType.RESPONSE) {
break;
}
}
This may work because it consumes all messages off each event.

It sounds like you are making too many requests at once. BB will only process a certain number of requests per connection at any given time. Note that opening more and more connections will not help because there are limits per subscription as well. If you make a large number of time consuming requests simultaneously, some may timeout. Also, you should process the request completely(until you receive RESPONSE message), or cancel them. A partial request that is outstanding is wasting a slot. Since splitting into two sessions, seems to have helped you, it sounds like you are also making a lot of subscription requests at the same time. Are you using subscriptions as a way to take snapshots? That is subscribe to an instrument, get initial values, and de-subscribe. If so, you should try to find a different design. This is not the way the subscriptions are intended to be used. An outstanding subscription request also uses a request slot. That is why it is best to batch as many subscriptions as possible in a single subscription list instead of making many individual requests. Hope this helps with your use of the api.

By the way, I can't tell from your sample code, but while you are blocked on messages from the event queue, are you also reading from the main event queue while(in a seperate event queue)? You must process all the messages out of the queue, especially if you have outstanding subscriptions. Responses can queue up really fast. If you are not processing messages, the session may hit some queue limits which may be why you are getting timeouts. Also, if you don't read messages, you may be marked a slow consumer and not receive more data until you start consuming the pending messages. The api is async. Event queues are just a way to block on specific requests without having to process all messages from the main queue in a context where blocking is ok, and it would otherwise be be difficult to interrupt the logic flow to process parts asynchronously.