Following the MS documentation it was not difficult to receive message(s) from subscription. However, if I'd like my application to receive a message every time new message is posted - a constant polling. Hence the OnMessage() method of the SubscriptionClient class.
MS documentation says: "...When calling OnMessage, the client starts an internal message pump that constantly polls the queue or subscription. This message pump consists of an infinite loop that issues a Receive() call. If the call times out, it issues the next Receive() call. ..."
But when the application is running, the moment OnMessage() method is called only latest message(s) is received. When new messages are posted the constant polling does not seem to be working. After trying many different approaches the only way I could make this work and have the application react the moment new message is received is to place the code into a separate task with infinite loop. This seems totally wrong on so many levels! (see code below).
Can anyone help me to correct my code or post a working sample to accomplish the same functionality without the loop? Thank you!
public void ReceiveMessageFromSubscription(string topicName, string subscriptionFilter)
{
var newMessage = new MessageQueue();
int i = 0;
Task listener = Task.Factory.StartNew(() =>
{
while (true)
{
SubscriptionClient Client = SubscriptionClient.CreateFromConnectionString(connectionString, topicName, subscriptionFilter);
Dictionary<string, string> retrievedMessage = new Dictionary<string, string>();
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = false;
options.AutoRenewTimeout = TimeSpan.FromMinutes(1);
Client.OnMessage((message) =>
{
try
{
retrievedMessage.Add("messageGuid", message.Properties["MessageGuid"].ToString());
retrievedMessage.Add("instanceId", message.Properties["InstanceId"].ToString());
retrievedMessage.Add("pId", message.Properties["ProcessId"].ToString());
retrievedMessage.Add("processKey", message.Properties["ProcessKey"].ToString());
retrievedMessage.Add("message", message.Properties["Message"].ToString());
newMessage.AnnounceNewMessage(retrievedMessage); // event ->
message.Complete(); // Remove message from subscription.
}
catch (Exception ex)
{
string exmes = ex.Message;
message.Abandon();
}
}, options);
retrievedMessage.Clear();
i++;
Thread.Sleep(3000);
}
});
}
Your code has a few issues to iron out -
It falls through and I assume your application then exits - or at
least the thread that is listening for the messages terminates.
Your while loop keeps repeating the code to hook up the message handler,
you only need to do this once.
You need a way to keep the call stack alive and prevent your app from garbage collecting your object.
The below should see you on your way to success. Good luck.
ManualResetEvent CompletedResetEvent = new ManualResetEvent(false);
SubscriptionClient Client;
public void ReceiveMessagesFromSubscription(string topicName, string subscriptionFilter, string connectionString)
{
Task listener = Task.Factory.StartNew(() =>
{
// You only need to set up the below once.
Client = SubscriptionClient.CreateFromConnectionString(connectionString, topicName, subscriptionFilter);
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = false;
options.AutoRenewTimeout = TimeSpan.FromMinutes(1);
options.ExceptionReceived += LogErrors;
Client.OnMessage((message) =>
{
try
{
Trace.WriteLine("Got the message with ID {0}", message.MessageId);
message.Complete(); // Remove message from subscription.
}
catch (Exception ex)
{
Trace.WriteLine("Exception occurred receiving a message: {0}" + ex.ToString());
message.Abandon(); // Failed. Leave the message for retry or max deliveries is exceeded and it dead letters.
}
}, options);
CompletedResetEvent.WaitOne();
});
}
/// <summary>
/// Added in rudimentary exception handling .
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="ex">The <see cref="ExceptionReceivedEventArgs"/> instance containing the event data.</param>
private void LogErrors(object sender, ExceptionReceivedEventArgs ex)
{
Trace.WriteLine("Exception occurred in OnMessage: {0}" + ex.ToString());
}
/// <summary>
/// Call this to stop the messages arriving from subscription.
/// </summary>
public void StopMessagesFromSubscription()
{
Client.Close(); // Close the message pump down gracefully
CompletedResetEvent.Set(); // Let the execution of the listener complete and terminate gracefully
}
Alternatively you can chunk the message off in a more traditional fashion yourself using ReceiveBatch:
var messages = await queueClient.ReceiveBatchAsync(10, TimeSpan.FromSeconds(30),
cancellationToken);
Related
I have implemented a rabbitMQ listener, which essentially just sits in a loop waiting for messages to arrive. When the message arrives I wish to fire an event and run some code.
However I don't always seem to receive this event, and I am wondering if this is because I am running the queue polling part of the code on a different thread.
It does seem to work initially though, so it is possible that the threading is not the problem. Can anyone give me an opinion on this?
QueueListener:
public void CreateQueueListener<T>() where T : IPubSubEvent
{
var mqServer = new RabbitMqServer(m_RabbitMqAddress);
var mqClient = (RabbitMqQueueClient)mqServer.MessageFactory.CreateMessageQueueClient();
var channel = mqClient.Channel;
string queueName = mqClient.GetTempQueueName();
channel.QueueBind(queueName, m_EventExchange, routingKey: QueueNames<T>.In);
var consumer = new RabbitMqBasicConsumer(channel);
channel.BasicConsume(queue: queueName, autoAck: true, consumer: consumer);
Task.Run(() =>
{
while (true)
{
BasicGetResult basicGetResult;
try
{
basicGetResult = consumer.Queue.Dequeue();
}
catch (Exception ex)
{
throw;
}
var message = basicGetResult.ToMessage<T>();
PublishEvent?.Invoke(this, new PubSubEventArgs { EventData = message.GetBody().EventName });
}
});
}
Consuming Class
public class MyClass
{
public MyClass(IEventClient eventClient)
{
eventClient.CreateQueueListener<AuthoriseEvent>();
eventClient.PublishEvent += OnPublishEvent;
}
private async void OnPublishEvent(object sender, PubSubEventArgs e)
{
if (e.EventData == "AuthoriseEvent")
//dostuff
}
}
I am running the queue polling part of the code on a different thread
As far as I know, this isn't supported by the .NET client.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
I've got a WCF windows service running. It basically registers a number of clients and broadcasts messages to them as and when needed.
Generally it works fine, but recently I've been getting an error trying to register a new client, there are about 24 clients connected already, but when I try to register a 25th, I get "Index was outside the bounds of the array." being returned.
If I restart the service, all the clients reconnect, and the new client is able to register.
The NotifyServer method is called to broadcast a message to all the registered clients. This runs through the clients dictionary creating an async task that sends the message. This is done so that any issue happening with sending to one client does not impact the sending of the message to others.
The service is set up as to use a Reliable Connection.
// List of connected Clients
//
private static Dictionary<string, IBroadcastorCallBack> clients = new Dictionary<string, IBroadcastorCallBack>();
// lock indicator object
//
private static object locker = new object();
public string RegisterClient(string clientName)
{
string returnValue = "";
if (!string.IsNullOrEmpty(clientName))
{
try
{
var callback = OperationContext.Current.GetCallbackChannel<IBroadcastorCallBack>();
lock (locker)
{
// Remove the old client if its already there
//
if (clients.Keys.Contains(clientName))
{
clients.Remove(clientName);
}
// Add the new one
//
clients.Add(clientName, callback);
}
}
catch (Exception ex)
{
// Return any error message
//
returnValue = ex.Message;
}
}
return returnValue;
}
/// <summary>
/// Notify the service of a message that can be broadcast
/// </summary>
/// <param name="eventData">Message details</param>
///
public async void NotifyServer(EventDataType eventData)
{
DateTime Start = DateTime.Now;
// Get a list copy of the dictionary for all clients bar the one sending it. This is so we can't update the list inside the loop.
//
var clientlist = clients.Where(x => x.Key != eventData.ClientName).ToList();
// Logging
//
if (MetricBroadcast.Properties.Settings.Default.LogXML)
{
log.Debug("XML Broadcast from " + eventData.ClientName + " to " + clientlist.Count.ToString() + " clients:\n" + eventData.EventMessage + "\n");
}
// Broadcast to all the valid clients
//
var BroadcastToClientList = clientlist.Select(client => BroadcastMessage(client.Value, client.Key, eventData)).ToList();
// Wait until they are all done
//
await Task.WhenAll(BroadcastToClientList);
// If we are logging and the broadcast time is > 1 second, we make a log entry
//
DateTime End = DateTime.Now;
if (MetricBroadcast.Properties.Settings.Default.LogXML)
{
TimeSpan res = End - Start;
if (res.TotalSeconds > 1)
{
var timetaken = string.Format("XML Broadcast Time : {0,2:00}:{1,2:00}.{2,3:000}", res.Minutes, res.Seconds, res.Milliseconds);
log.Debug(timetaken);
}
}
}
private async Task<bool> BroadcastMessage(IBroadcastorCallBack clientCallback, string ClientKey, EventDataType eventData)
{
bool retval = true;
Exception savedEx = null;
DateTime BroadStart = DateTime.Now;
try
{
// Send the message to the current client
//
clientCallback.BroadcastToClient(eventData);
}
catch (Exception e)
{
// If we can't access the current clients callback method,
// we remove them from the clients list, as they've probably lost their connection.
//
clients.Remove(ClientKey);
savedEx = e;
retval = false;
}
// Log any broadcast that took > .5 seconds
//
DateTime BroadEnd = DateTime.Now;
if (MetricBroadcast.Properties.Settings.Default.LogXML)
{
TimeSpan res = BroadEnd - BroadStart;
if (res.TotalSeconds > .5)
{
var timetaken = string.Format("Single XML Broadcast Time to " + ClientKey + " : {0,2:00}:{1,2:00}.{2,3:000}", res.Minutes, res.Seconds, res.Milliseconds);
log.Debug(timetaken, savedEx);
}
}
return retval;
}
Most likely the problem is incorrect usage of static clients Dictionary in multithreaded environment. New client can register at any time, including in the middle of your BroadcastMessage and NotifyServer functions. Dictionary was not designed for access from multiple threads. Take for example this:
clients.Where(x => x.Key != eventData.ClientName).ToList();
What happens if client is removed in the middle of this enumeration? Who knows, because it was not designed for this, but most likely some exception (like your "index out of bounds") will be thrown. That is why you need to lock your dictionary for both writes and reads, not just for writes:
List<KeyValuePair<string, IBroadcastorCallback>> clientList;
lock (locker)
clientList = clients.Where(x => x.Key != eventData.ClientName).ToList();
Another option is to use ConcurrentDictionary class. It was designed for concurrent access and you can safely read and write to it from multiple threads.
I have a MQTT calls inside a loop and in each iteration, it should return a response from the subscriber so that I could use the value being forwarded after I published. But the problem is I don't know how would I do it.
I hope you have an idea there or maybe if I'm just not implementing it right, may you guide me through this. Thanks.
Here's my code:
// MyClientMgr
class MyClientMgr{
public long CurrentOutput { get; set; }
public void GetCurrentOutput(MyObjectParameters parameters, MqttClient client)
{
MyMessageObject msg = new MyMessageObject
{
Action = MyEnum.GetOutput,
Data = JsonConvert.SerializeObject(parameters)
}
mq_GetCurrentOutput(msg, client);
}
private void mq_GetCurrentOutput(MyMessageObject msg, MqttClient client)
{
string msgStr = JsonConvert.SerializeObject(msg);
client.Publish("getOutput", Encoding.UTF8.GetBytes(msgStr),
MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, false);
client.MqttMsgPublishReceived += (sender, e) =>{
MyObjectOutput output = JsonConvert.DeserializeObject<MyObjectOutput>(Encoding.UTF8.GetString(e.Message));
CurrentOutput = output;
};
}
}
// MyServerMgr
class MyServerMgr
{
public void InitSubscriptions()
{
mq_GetOutput();
}
private void mq_GetOutput()
{
MqttClient clientSubscribe = new MqttClient(host);
string clientId = Guid.NewGuid().ToString();
clientSubscribe.Connect(clientId);
clientSubscribe.Subscribe(new string[] { "getOutput" }, new byte[] { MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE });
MqttClient clientPublish = new MqttClient(host);
string clientIdPub = Guid.NewGuid().ToString();
clientPublish.Connect(clientIdPub);
clientSubscribe.MqttMsgPublishReceived += (sender, e) => {
MyMessageObj msg = JsonConvert.DeserializeObject<MyMessageObj>(Encoding.UTF8.GetString(e.Message));
var output = msg.Output;
clientPublish.Publish("getOutput", Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(output)), MqttMsgBase.QOS_LEVEL_EXACTLY_ONCE, false);
}
}
}
// MyCallerClass
class MyCallerClass
{
var host = "test.mqtt.org";
var myServer = new MyServerMgr(host);
var myClient = new MyClientMgr();
myServer.InitSubscriptions();
MqttClient client = new MqttClient(host);
for(int i = 0; i < 10; i++)
{
long output = 0;
MyObjectParameters parameters = {};
myClient.GetCurrentOutput(parameters, client) // here I call the method from my client manager
// to publish the getting of the output and assigned
// below for use, but the problem is the value doesn't
// being passed to the output variable because it is not
// yet returned by the server.
// Is there a way I could wait the process to
// get the response before assigning the output?
output = myClient.CurrentOutput; // output here will always be null
// because the response is not yet forwarded by the server
}
}
I have a loop in my caller class to call the mqtt publish for getting the output, but I have no idea how to get the output before it was assigned, I want to wait for the response first before going to the next.
I've already tried doing a while loop inside like this:
while(output == 0)
{
output = myClient.CurrentOutput;
}
Yes, I can get the output here, but it will slow down the process that much. And sometimes it will fail.
Please help me. Thanks.
It looks like you are trying to do synchronous communication over an asynchronous protocol (MQTT).
By this I mean you want to send a message and then wait for a response, this is not how MQTT works as there is no concept of a reply to a message at the protocol level.
I'm not that familiar with C# so I'll just give an abstract description of possible solution.
My suggestion would be to use a publishing thread, wait/pulse (Look at the Monitor class) to have this block after each publish and have the message handler call pulse when it has received the response.
If the response doesn't contain a wait to identify the original request you will also need a state machine variable to record which request is in progress.
You may want to look at putting a time out on the wait in case the other end does not respond for some reasons.
You can use AutoResetEvent class that has WaitOne() and Set() methods. Using WaitOne() after publish will wait until the message is published and using Set() under client_MqttMsgPublishReceived event will release the wait when the subscriber received the message he subscribed for.
I have an Azure worker role that is responsible for checking 4 service bus queues. Currently, I just the looping method to manually check the queues.
while(true)
{
//loop through my queues to check for messages
}
With the Azure SDK 2.0 came the ability to listen for messages rather than polling for them. But Every example I've seen uses a console app with Console.ReadKey(). Is there a way to have the worker role sit and wait on messages too?
I tried:
public override void Run()
{
_queueProcessors.ForEach(x => x.OnMessage(Process);
}
where _queueProcessors is a list of QueueClients and Process is a private method that handles the messages. However, the worker role would register them and then restart.
So anyone know how to make a queue client sit and wait on a message?
Following is a code sample for this:
using Microsoft.ServiceBus;
using Microsoft.ServiceBus.Messaging;
using Microsoft.WindowsAzure.ServiceRuntime;
using System.Diagnostics;
using System.Net;
using System.Threading;
namespace WorkerRoleWithSBQueue1
{
public class WorkerRole : RoleEntryPoint
{
// The name of your queue
const string QueueName = "demoapp";
ManualResetEvent CompletedEvent = new ManualResetEvent(false);
// QueueClient is thread-safe. Recommended that you cache
// rather than recreating it on every request
QueueClient Client;
public override void Run()
{
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = true; // Indicates if the message-pump should call complete on messages after the callback has completed processing.
options.MaxConcurrentCalls = 1; // Indicates the maximum number of concurrent calls to the callback the pump should initiate
options.ExceptionReceived += LogErrors; // Allows users to get notified of any errors encountered by the message pump
Trace.WriteLine("Starting processing of messages");
// Start receiveing messages
Client.OnMessage((receivedMessage) => // Initiates the message pump and callback is invoked for each message that is recieved, calling close on the client will stop the pump.
{
try
{
// Process the message
Trace.WriteLine("Processing Service Bus message: " + receivedMessage.SequenceNumber.ToString());
}
catch
{
// Handle any message processing specific exceptions here
}
}, options);
CompletedEvent.WaitOne();
}
private void LogErrors(object sender, ExceptionReceivedEventArgs e)
{
if (e.Exception != null)
{
Trace.WriteLine("Error: " + e.Exception.Message);
}
}
public override bool OnStart()
{
// Set the maximum number of concurrent connections
ServicePointManager.DefaultConnectionLimit = 12;
// Create the queue if it does not exist already
Trace.WriteLine("Creating Queue");
string connectionString = "*** provide your connection string here***";
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
if (!namespaceManager.QueueExists(QueueName))
{
namespaceManager.CreateQueue(QueueName);
}
// Initialize the connection to Service Bus Queue
Client = QueueClient.CreateFromConnectionString(connectionString, QueueName);
Trace.WriteLine("Sending messages...");
// populate some messages
for (int ctr = 0; ctr < 10; ctr++)
{
Client.Send(new BrokeredMessage());
}
return base.OnStart();
}
public override void OnStop()
{
// Close the connection to Service Bus Queue
Client.Close();
CompletedEvent.Set(); // complete the Run function
base.OnStop();
}
}
}
We have a service that receives messages from n message queues. However, if the Message Queuing service is restarted, the message retrieval service stops receiving messages even after the Message Queuing service has restarted successfully.
I have tried to specifically catch the MessageQueueException that is thrown in the message retrieval service and invoke the queue's BeginReceive method again. However, in the 2 seconds or so that it takes the Message Queuing service to restart, I get about 1875 instances of the exception and then the service stops functioning when another MessageQueueException is thrown in our StartListening method.
Is there an elegant way to recover from a Message Queuing service restart?
private void OnReceiveCompleted(object sender, ReceiveCompletedEventArgs e)
{
MessageQueue queue = (MessageQueue)sender;
try
{
Message message = queue.EndReceive(e.AsyncResult);
this.StartListening(queue);
if (this.MessageReceived != null)
this.MessageReceived(this, new MessageReceivedEventArgs(message));
}
catch (MessageQueueException)
{
LogUtility.LogError(String.Format(CultureInfo.InvariantCulture, StringResource.LogMessage_QueueManager_MessageQueueException, queue.MachineName, queue.QueueName, queue.Path));
this.StartListening(queue);
}
}
public void StartListening(MessageQueue queue)
{
queue.BeginReceive();
}
I need to deal with the infinite loop issue this causes and clean it up a bit but you get the idea.
When the MessageQueueException occurs, invoke the RecoverQueue method.
private void RecoverQueue(MessageQueue queue)
{
string queuePath = queue.Path;
bool queueRecovered = false;
while (!queueRecovered)
{
try
{
this.StopListening(queue);
queue.Close();
queue.Dispose();
Thread.Sleep(2000);
MessageQueue newQueue = this.CreateQueue(queuePath);
newQueue.ReceiveCompleted += new ReceiveCompletedEventHandler(this.OnReceiveCompleted);
this.StartListening(newQueue);
LogUtility.LogInformation(String.Format(CultureInfo.InvariantCulture, "Message queue {0} recovered successfully.", newQueue.QueueName));
queueRecovered = true;
}
catch (Exception ex)
{
LogUtility.LogError(String.Format(CultureInfo.InvariantCulture, "The following error occurred while trying to recover queue: {0} error: {1}", queue.QueueName, ex.Message));
}
}
}
public void StopListening(MessageQueue queue)
{
queue.ReceiveCompleted -= new ReceiveCompletedEventHandler(this.OnReceiveCompleted);
}
Upon receiving the exception that is the result of the service restarting, you have to release the old MessageQueue, i.e. unwiring your ReceiveCompleted event, disposing the MessageQueue, etc. Then create a new instance of the MessageQueue and hook up to the ReceiveCompleted event again on the new MessageQueue instance.
Alternatively, you can use a polling method that creates a new instance on a certain interval, calls MessageQueue.Receive(TimeSpan), will wait for an incoming message or until the timeout occurs. In which case you handle the message and destroy the MessageQueue instance and start the iteration again.
By recreating the MessageQueue each time, you ensure a built in recovery. Also, the overhead of creating the MessageQueue is minimal due to internal caching of the underlying queue.
Pseudo-code...
while (!notDone)// or use a timer or periodic task of some sort...
{
try
{
using (MessageQueue queue = new MessageQueue(queuePath))
{
Message message = queue.Receive(TimeSpan.FromMilliseconds(500));
// process message
}
}
catch (MessageQueueException ex)
{
// handle exceptions
}
}