Develop Reference

Azure Service bus - pass parameter to message handler

In Azure Service Bus I need to listen for messages arriving from multiple subscriptions from different services busses at once.
To do this I created a list that contains objects with a connection string, a topic, a subscription name and some other information (the list is called 'jobs').
For each item in this list I am then creating a different task that creates the ServiceBusClient and the processor.
var jobs = GetAllServiceBusTopics();
Parallel.ForEach(jobs, async job =>
{
var client = new ServiceBusClient(job.Environment.ServiceBusConnectionString);
var options = new ServiceBusProcessorOptions();
var processor = client.CreateProcessor(job.Environment.TopicName, _subscriptionName, new ServiceBusProcessorOptions());
try
{
processor.ProcessMessageAsync += MessageHandler;
//Pass the job object somehow to the "MessageHandler" below.
processor.ProcessErrorAsync += ErrorHandler;
await processor.StartProcessingAsync();
Console.WriteLine("Wait for a minute and then press any key to end the processing");
Console.ReadKey();
Console.WriteLine("\nStopping the receiver...");
await processor.StopProcessingAsync();
Console.WriteLine("Stopped receiving messages");
}
finally
{
await processor.DisposeAsync();
await client.DisposeAsync();
}
});
And the handler that is called if a new message arrives:
static async Task MessageHandler(ProcessMessageEventArgs args)
{
//I need the "job" object from my loop above here.
}
How the concept generally works I learned on this website of Microsoft.
My first question:
Is this approach okay, or am I running in the wrong direction? Can I do it like this?
But even if this is okay, I have another more important task:
I need to pass the "job" object from my loop somehow to the message handler - as a parameter.
But I have currently no idea how to archvie this. Any proposals on this?

Is this approach okay, or am I running in the wrong direction? Can I do it like this?
Yes, you can do this. One thing to keep in mind is that you instantiate multiple ServiceBusClient instances, each causing a new connection to be established rather than using the same connection. I don't know how big the number of topics (jobs) might be but if it's large, you'll end up with connections starvation.
I need to pass the "job" object from my loop somehow to the message handler - as a parameter. But I have currently no idea how to archvie this. Any proposals on this?
That's not how ServiceBusProcessor is designed. It doesn't receive anything other than the incoming message that needs to be processed. If you need to have a job ID, that should be part of the message payload/metadata. If you need to know the entity it arrived from, you could add a subscription filter action to add a custom header with the identifier. An alternative approach would require wrapping the ServiceBusProcessor to retain the job ID/subscription identifier and use that in the event handler.

In an Azure Web Job, when is an item in the Azure queue removed?

I have created a triggered web job using the Azure queue. For brevity, my function looks like this:
public static void ProcessQueueMessage([QueueTrigger("myqueue")] string message, TextWriter log)
{
log.WriteLine(message);
System.Threading.Thread.Sleep(10000);
}
I have another function that adds items to the queue for processing. However, I want to check the last xx messages to determine if an item already exists in the queue (with the same message). Something like this:
while (true)
{
var myMessageId = "98uasd98fusad8";
CloudQueue queue = queueClient.GetQueueReference("myqueue");
queue.CreateIfNotExists();
var maxMessagesToCheck = 99;
var oldMessages = queue.GetMessages(maxMessagesToCheck, null);
if (!oldMessages.Any(x => x.AsString == myMessageId))
{
CloudQueueMessage message = new CloudQueueMessage(myMessageId);
queue.AddMessage(message);
}
}
However, I'm just not sure when an item is "popped" off the queue. Does it get popped off after processing is complete and the ProcessQueueMessage function returns (without error)? Or, does it get popped off right as the processing starts?
If the items are popped off after the processing completes, then my second function above will work fine. However, if items are popped off as soon as the process starts, then I will have to find another solution.
This might be a more concise way of asking the question:
public static void ProcessQueueMessage([QueueTrigger("myqueue")] string message, TextWriter log)
{
//IS THE ITEM IN THE QUEUE POPPED OFF HERE?
log.WriteLine(message);
System.Threading.Thread.Sleep(10000);
} //OR IS THE ITEM IN THE QUEUE POPPED OFF HERE?

According to Microsoft doc
When a message is locked, other clients receiving from the same queue or subscription can take on locks and retrieve the next available messages not under active lock. When the lock on a message is explicitly released or when the lock expires, the message pops back up at or near the front of the retrieval order for redelivery.
For that reason your solution will not work, let's say you have 10 messages in queue, and the first message is currently being processed by your first function, therefore when the checking function runs it can only peek messages after it, let's say it is peeking message 2,3,4 in a batch, same as the first function it needs to put lock for the three messages, so when the first function finishes message 1, it can only pick up message 5, which has't checked.

NamedPipeServerStream/NamedPipeClientStream wrapper

I'm currently writing a small wrapper for NamedPipeServerStream/NamedPipeClientStream that is fully Event based as oppose to using AsyncCallbacks.
I expose sync and async methods for pretty much everything possible (connecting/waiting for connection, writing, etc) so if a consumer wanted to, for example, start a server instance and send a message when a client connects he could either go full sync route and do something like ...
var server = new NamedPipeServer("helloWorld");
server.StartAndWait();
server.Write("Welcome!");
or the async way like...
var server = new NamedPipeServer("helloWorld);
server.ClientConnected += x => x.WriteAsync("Welcome!");
server.Start(); //Start() returns immediately
However I'm struggling with finding a good way to do the same for reading messages. Currently when a message is read I fire a MessageAvailable event and pass the message in as one of the arguments.
I just can't come up with a proper way of implementing synchronous reads.
What I've considered is the following:
Having a GetNextMessage() sync method that gets the message. Internally, this could be handled in two different ways:
I could keep an IEnumerable<Message> with all of the not yet consumed messages. So as soon as the other side sends a message, I'd read it from the stream and store it in memory so they can be later consumed by GetNextMessage(). The advantage is that it frees up the stream pretty much as soon as the message is written, so it doesn't block the other side from sending other messages. The disadvantage is that I have absolutely no control over how many messages I'll be holding or the size of them. My IEnumerable<Message> might end up having 10GB worth of non-consumed messages, and there's nothing I can do about it since I can't force the consumer to retrieve messages.
I could take the view that I only ever store one message in an internal buffer, and only ever start reading again once that one was consumed via GetNextMessage(). If I do that though, the other side would be prevented from writing other messages until the previous one was consumed. To be more exact, the other side would be able to write until the stream is full. Which could be either multiple small complete messages or a single incomplete message. In the case of an incomplete single message I think this is a worse approach because in between part 1 of the message being sent and subsequent parts, the other end might end up disconnecting and the whole message will be lost.
To make things harder, in either of the approaches above there's always the chance that the consumer is using events for receiving messages (remember the event contains the message received) and therefore has no need for GetNextMessage(). I'd either need to stop sending the message in the event altogether, or find a way of not pushing the event to the internal buffer if the message is consumed via the event. And while I can easily tell whether there is an event handler or not, there's no way of knowing if the message is actually being handled there (i.e., consider a class implementing this one and listens to that event, yet does nothing with it). The only real approach I can see here is to remove the message from the event, force consumers to always call GetNextMessage(), but am open to other ideas.
There's also another problem with either of the approaches, which is the fact that I can't control the order in which the messages are sent if WriteAsync() is used (or Write() is used from different threads).
Can anyone think of a better way of tackling this problem?

I'd suggest the following approach. Create interface:
public interface ISubscription : IDisposable {
Message NextMessage(TimeSpan? timeout);
}
public class Message {
}
And then implement like that:
public class NamedPipeServer {
public void StartAndWait() {
}
public ISubscription StartAndSubscribe() {
// prevent race condition before Start and subscribing to MessageAvailable
var subscription = new Subscription(this);
StartAndWait();
return subscription;
}
public ISubscription Subscribe() {
// if user wants to subscribe and some point after start - why not
return new Subscription(this);
}
public event Action<Message> MessageAvailable;
private class Subscription : ISubscription {
// buffer
private readonly BlockingCollection<Message> _queue = new BlockingCollection<Message>(
new ConcurrentQueue<Message>());
private readonly NamedPipeServer _server;
public Subscription(NamedPipeServer server) {
// subscribe to event
_server = server;
_server.MessageAvailable += OnMessageAvailable;
}
public Message NextMessage(TimeSpan? timeout) {
// this is blocking call
if (timeout == null)
return _queue.Take();
else {
Message tmp;
if (_queue.TryTake(out tmp, timeout.Value))
return tmp;
return null;
}
}
private void OnMessageAvailable(Message msg) {
// add to buffer
_queue.Add(msg);
}
public void Dispose() {
// clean up
_server.MessageAvailable -= OnMessageAvailable;
_queue.CompleteAdding();
_queue.Dispose();
}
}
}
Client then either calls Subscribe or StartAndSubscribe.
var sub = server.StartAndSubscribe();
var message = sub.NextMessage();
var messageOrNull = sub.NextMessage(TimeSpan.FromSeconds(1));
sub.Dispose();
That way if no one subscribes - you buffer no messages. And if someone subscribes and then does not consume - it's their problem, not yours, because buffering happens in subscription they now own. You can also limit size of _queue blocking collection, then adding to it will block if limit is reached, blocking your MessageAvailable event, but I won't recommend doing that.

Cant get MSMQ to work properly

I keep receiving an error when i try to Send to the queues
The specified format name does not support the requested operation. For example, a direct queue format name cannot be deleted.
My queue name is stored in a config file and looks like this
"FormatName:Direct=OS:MyComputerName\PRIVATE$\MyQueue"
There are 2 queues that exist, each queue has permissions set so that EVERYONE has Full Control.
SyncUser you can assume = "EVERYONE" or "My_Domain\operator"
//init the queues
qSync = new MessageQueue(queueName + "-sync")
{Formatter = new XmlMessageFormatter(new Type[] {typeof (String)})};
qClient = new MessageQueue(queueName + "-client")
{Formatter = new XmlMessageFormatter(new Type[] { typeof(String) })};
Creating the queues doesn't cause any errors, but I receive errors when i call this Send function
**Send(qSync, "This is a message");** //Example
public void Send(MessageQueue queue, string msg)
{
try
{
queue.Send(msg);
}
catch (MessageQueueException mqx)
{
var e = string.Format("Failed to Send - {0} | {1}", queue.QueueName, mqx.Message);
OnSynchronizerMonitor(new SyncEvent(e));
}
}
Any help would be greatly appreciated.
EDIT
It seems that my messages ARE making it to their remote queue destination, but the application is still throwing that error on every message sent. If I look at the MessageQueues (qSync|qClient) while debugging, that error shows up in several of the inner fields, before it is even used.

I hope someone can confirm this for me as I cant seem to find any forum questions or documentation that support this, but it seems that the MessageQueue has an issue with my Event setup.
I removed the event call and simply hardcoded the queue creation and send in its place within my main process and everything worked fine.
My setup is like so:
A Service - Creates a Processor, a Synchronizer, and sets the events.
Processor starts a thread in which it processes string transactions, on a successful process the transaction needs to be sent via the Synchronizer, so the Processor triggers event.
Synchronizer has already created the queues and is simply waiting for events to be triggered to send messages.
If anyone could shed some light on if this could be a cross thread issue or just a quirk in my system it would be appreciated.
**After a little more research and some trial and error, it seems I get the error due to a cross thread complication with the event. My guess is that it's a similar issue to UI objects being updated from a different thread than the one it was created on. The error is essentially an ownership issue. The messages still go though but the queue complains that its original owner isn't the one that initiated the call.

Edit MSMQ messages in a queue

I need to be able to edit messages in my error queue (so that they can be resent to the actual queue for reprocessing).
I would like to make my own custom tool to do this (because my messages require specific formatting to make them easily readable by support personnel).
I know that this is possible because the application "QueueExplorer" does this.
Does anyone know how I can load an MSMQ message (that is not the first one in the queue), edit it, and save it back to the queue?

Iterate through the messages, using something like this:
List<Message> msgList = new List<Message>();
using (MessageEnumerator me = queue.GetMessageEnumerator2())
{
while (me.MoveNext(new TimeSpan(0, 0, 0)))
{
Message message = me.Current;
msgList.Add(message)
}
}
You can then iterate through the list, processing each message.
Create a new message, based on the original. Then remove the existing message, and add the new one.
foreach (Message message in msgList)
{
//Create a new message as required, add it, then remove the old message
MessageQueue.ReceiveById(message.MessageId);
}

MSMQ messages are supposed to be immutable. The best you can do is read the message and send an edited copy of the message back to the queue.