OBJECTIVE:
I am using StackExchange Redis Client. My objective is to create an Observable stream from the Pub Sub Subscriber exposed by the Client, that can then in turn support 1-n subscriptions by Observables, that each have their own filter via LINQ. (Publishing is working as planned, the issue is purely around Subscription to the Event Stream on a specific channel.)
BACKGROUND:
I am using Redis Pub Sub as part of an Event Sourced CQRS application. The specific use case is to publish Events to multiple subscribers that then update the various Read Models, send emails etc.
Each of these Subscribers need to filter the Event Types that they handle, and for this I am looking to use Rx .Net (Reactive Extensions), with LINQ, to
provide a filter criteria on the Event Stream, to efficiently handle reacting only to Events of interest. Using this approach removes the need for registering Handlers with an event bus implementation, and allows me to add new projections to the system by deploying 1-n Microservices that each have 1-n Observables subscribed to the Event Stream with their own specific filters.
WHAT I HAVE TRIED:
1) I have created a class inheriting from ObservableBase, overriding the SubscribeCore method, which receives subscription requests from Observables, stores them in a ConcurrentDictionary, and as each Redis notification arrives from the channel, loops through the registered Observable subscribers and calls their OnNext method passing the RedisValue.
2) I have created a Subject, that also accepts subscriptions from Observables, and calls their OnNext method. Again, the use of Subjects appears to be frowned upon by many.
THE ISSUE:
The approaches I have tried do function (at least superficially), with varying levels of performance, but feel like a hack, and that I am not using Rx in the way it was intended.
I see many comments that the built-in Observable methods should be used where at all possible, for example Observable.FromEvent, but that seems to be impossible to do with the StackExchange Redis Clients Subscription API, at least to my eyes.
I also understand that the preferred method for receiving a stream and forwarding to multiple Observers is to use a ConnectableObservable, which would seem to be designed for the very scenario I face (Each Microservice will internally have 1-n Observables subscribed). At the moment, I cannot get my head around how to connect a ConnectableObservable to the notifications from StackExchange Redis, or if it offers real benefit over an Observable.
UPDATE:
Although completion is not an issue in my scenario (Disposal is fine), error handling is important; e.g. isolating errors detected in one subscriber to prevent all subscriptions terminating.
Here is an extension method you can use to create an IObservable<RedisValue> from an ISubscriber and a RedisChannel:
public static IObservable<RedisValue> WhenMessageReceived(this ISubscriber subscriber, RedisChannel channel)
{
return Observable.Create<RedisValue>(async (obs, ct) =>
{
// as the SubscribeAsync callback can be invoked concurrently
// a thread-safe wrapper for OnNext is needed
var syncObs = Observer.Synchronize(obs);
await subscriber.SubscribeAsync(channel, (_, message) =>
{
syncObs.OnNext(message);
}).ConfigureAwait(false);
return Disposable.Create(() => subscriber.Unsubscribe(channel));
});
}
As there is no completion of Redis channels the resulting IObservable will never complete, however you may drop the IDisposable subscription to unsubscribe from the Redis channel (this will be done automatically by many Rx operators).
Usage could be like so:
var subscriber = connectionMultiplexer.GetSubscriber();
var gotMessage = await subscriber.WhenMessageReceived("my_channel")
.AnyAsync(msg => msg == "expected_message")
.ToTask()
.ConfigureAwait(false);
Or as per your example:
var subscriber = connectionMultiplexer.GetSubscriber();
var sendEmailEvents = subscriber.WhenMessageReceived("my_channel")
.Select(msg => ParseEventFromMessage(msg))
.Where(evt => evt.Type == EventType.SendEmails);
await sendEmailEvents.ForEachAsync(evt =>
{
SendEmails(evt);
}).ConfigureAwait(false);
Other microservices may filter differently.
Related
I just started to learn about TPL Dataflow and have a question as described below:
"Block"s 1, 2, 3 holds references to states. They modify the states and send messages downstream each time they receive a message. The number of such blocks varies.
The "Aggregator" receives messages from the Blocks and check all the messages for errors. After all source blocks are Completed and aggregator passes a single message to the "Releaser".
"Releaser" holds a reference to the state. It will know from "Aggregator" whether the updating is done correctly and will send the state with a success message or a failure message downstream.
public static void Run()
{
var sourceBlock1 = new TransformBlock<int, int>(x => x * 2);
var sourceBlock2 = new TransformBlock<int, int>(x => x * 3);
//How to implement the aggregator that aggregates messages from an unknown number of sources and then return a message
//when all sources are complete?
var aggregater = new TransformBlock<int, int[]>(x => ?);
var releaser = new TransformBlock<int[], int>(xs => xs.Sum());
sourceBlock1.LinkTo(aggregater);
sourceBlock2.LinkTo(aggregater);
aggregater.LinkTo(releaser);
sourceBlock1.Post(10);
sourceBlock2.Post(20);
targetBlock.Completion.Wait();
}
In this line:
sourceBlock1.LinkTo(aggregater);
...the aggregater receives no notification that it has become the linked target of the sourceBlock1. The ISourceBlock<TOutput>.LinkTo method changes only the state of the source, not the target. The target will only become aware that it has been linked when it is offered the first message, via the ITargetBlock<TInput>.OfferMessage method:
public DataflowMessageStatus OfferMessage (
DataflowMessageHeader messageHeader,
TInput messageValue,
ISourceBlock<TInput> source,
bool consumeToAccept);
Even then, it's not guaranteed that the source argument will be a reference to the sourceBlock1, since the sourceBlock1 can opt to intercept an internal proxy ISourceBlock<TInput> implementation between itself and its linked target. So I don't think that you can achieve what you want using solely the methods of the existing interfaces. Maybe you could equip your custom aggregator block with an API that allows for bidirectional link-awareness. For example:
aggregater.Observe(sourceBlock1);
aggregater.Observe(sourceBlock2);
As for how to propagate the completion of multiple blocks to a single target block, take a look at these links:
Many to Many TPL Dataflow does not process all inputs
Dataflow unreliably completes before processing all items
In Azure Service Bus I need to listen for messages arriving from multiple subscriptions from different services busses at once.
To do this I created a list that contains objects with a connection string, a topic, a subscription name and some other information (the list is called 'jobs').
For each item in this list I am then creating a different task that creates the ServiceBusClient and the processor.
var jobs = GetAllServiceBusTopics();
Parallel.ForEach(jobs, async job =>
{
var client = new ServiceBusClient(job.Environment.ServiceBusConnectionString);
var options = new ServiceBusProcessorOptions();
var processor = client.CreateProcessor(job.Environment.TopicName, _subscriptionName, new ServiceBusProcessorOptions());
try
{
processor.ProcessMessageAsync += MessageHandler;
//Pass the job object somehow to the "MessageHandler" below.
processor.ProcessErrorAsync += ErrorHandler;
await processor.StartProcessingAsync();
Console.WriteLine("Wait for a minute and then press any key to end the processing");
Console.ReadKey();
Console.WriteLine("\nStopping the receiver...");
await processor.StopProcessingAsync();
Console.WriteLine("Stopped receiving messages");
}
finally
{
await processor.DisposeAsync();
await client.DisposeAsync();
}
});
And the handler that is called if a new message arrives:
static async Task MessageHandler(ProcessMessageEventArgs args)
{
//I need the "job" object from my loop above here.
}
How the concept generally works I learned on this website of Microsoft.
My first question:
Is this approach okay, or am I running in the wrong direction? Can I do it like this?
But even if this is okay, I have another more important task:
I need to pass the "job" object from my loop somehow to the message handler - as a parameter.
But I have currently no idea how to archvie this. Any proposals on this?
Is this approach okay, or am I running in the wrong direction? Can I do it like this?
Yes, you can do this. One thing to keep in mind is that you instantiate multiple ServiceBusClient instances, each causing a new connection to be established rather than using the same connection. I don't know how big the number of topics (jobs) might be but if it's large, you'll end up with connections starvation.
I need to pass the "job" object from my loop somehow to the message handler - as a parameter. But I have currently no idea how to archvie this. Any proposals on this?
That's not how ServiceBusProcessor is designed. It doesn't receive anything other than the incoming message that needs to be processed. If you need to have a job ID, that should be part of the message payload/metadata. If you need to know the entity it arrived from, you could add a subscription filter action to add a custom header with the identifier. An alternative approach would require wrapping the ServiceBusProcessor to retain the job ID/subscription identifier and use that in the event handler.
I subscribe to topics like that:
var topics = new string[]{
"TOP/IC/*/A/>",
"TOP/IC/*/B/>",
"ANY/*/STRU/CTURE"
// etc...
};
foreach(var t in topics)
{
var topic = ContextFactory.Instance.CreateTopic(t);
var sessionReturnCode = _session.Subscribe(topic, false); // _session is ISession
}
Incomming message has full topic name information. E.g. TOP/IC/ANY/A/N123/XFDJK985.
I would like to know from which subscription the message comes. Is it possible? (correlationId, whatever...). If yes, sample please.
When Solace .NET API, by default, Direct messages are dispatched to the same receive callback or delegate that is configured for the Session. However, it is possible to use special receive callbacks or delegates to handle Direct messages that are published to specific Topics. This functionality is also known as topic dispatching because messages with a specific Topic are dispatched to a specific callback or delegate.
More information is available here: https://docs.solace.com/Solace-PubSub-Messaging-APIs/API-Developer-Guide/Receiving-Direct-Message.htm
I'm trying to understand the scoping of using SE.Redis objects specifically in the area of subscribing for (and unsubscribing) to notifications.
I'd like to do something like the following to wait for a remote node to indicate it has changed/freed a resource (a very dumb distributed semaphore):
var t = new TaskCompletionSource<bool>();
sub.Subscribe(key, (c, v) =>
{
t.TrySetResult(true);
sub.UnsubscribeAll();
});
return t.Task;
I am fairly certain that is wrong :) and in a multithreaded environment with the ConnectionMultiplexer shared I'm probably going to end up in a race where one thread subscribes to a particular RedisChannel whilst another unsubscribes from it.
Is it possible to safely/efficiently implement this pattern or am I trying to simplify this problem too much and need a per process 'subscription manager' to coordinate my subscriptions?
while (true)
{
BasicDeliverEventArgs e = (BasicDeliverEventArgs)Consumer.Queue.Dequeue();
IBasicProperties properties = e.BasicProperties;
byte[] body = e.Body;
Console.WriteLine("Recieved Message : " + Encoding.UTF8.GetString(body));
ch.BasicAck(e.DeliveryTag, false);
}
This is what we do when we Retrieve Message by subscription..We use While Loop because we want Consumer to listen Continously..what if i want to make this even based..that is when a new message arrives in the queue at that time only Consumer should Consume the message..or on any such similar event..
use the RabbitMQ.Client.Events.EventingBasicConsumer for a eventing consumer instead of a blocking one.
You're currently blocking on the Consumer.Queue.Dequeue(). If I understand your question correctly, you want to asynchronously consume messages.
The standard way of doing this would be to write your own IBasicConsumer (probably by subclassing DefaultBasicConsumer) and set it as the consumer for the channel.
The trouble with this is that you have to be very careful about what you do in IBasicConsumer.HandleBasicDelivery. If you use any synchronous AMQP methods, such as basic.publish, you'll get a dead-lock. If you do anything that takes a long time, you'll run into some other problems.
If you do need synchronous methods or long-running actions, what you're doing is about the right way to do it. Have a look at Subscription; it's an IBasicConsumer that consumes messages and puts them on a queue for you.
If you need any more help, a great place to ask is the rabbitmq-discuss mailing list.
I had this problem and could not find an answer so created a demonstration project to have the RabbitMQ subscription raise .Net events when a message is received. The subscription runs on its own thread leaving the UI (in mycase) free to do it thing.
I amusing call my project RabbitEar as it listens out for messages from the mighty RabbitMQ
I intend to share this with the RabbitMQ site so if they think its of value they can include a link / code in there examples.
Check it out at http://rabbitears.codeplex.com/
Thanks
Simon