Develop Reference

Reading messages from IBM MQ asynchronously - download many messages

Below is the code responsible for creating the connection:
IConnection connectionMQ;
factoryFactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
IConnectionFactory cf = factoryFactory.CreateConnectionFactory();
cf.SetStringProperty(XMSC.WMQ_HOST_NAME, "host");
cf.SetIntProperty(XMSC.WMQ_PORT, port);
cf.SetStringProperty(XMSC.WMQ_CHANNEL, "channel");
cf.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT);
cf.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, "queueManager");
cf.SetStringProperty(XMSC.WMQ_SSL_PEER_NAME, "sslPeerName");
cf.SetStringProperty(XMSC.WMQ_SSL_CIPHER_SPEC, "TLS_RSA_WITH_AES_256_CBC_SHA256");
cf.SetStringProperty(XMSC.WMQ_CCSID, "ccSid");
cf.SetStringProperty(XMSC.WMQ_SSL_KEY_REPOSITORY, "*USER");
cf.SetStringProperty(XMSC.WMQ_SSL_CLIENT_CERT_LABEL, "clientCertLabel");
connectionMQ = cf.CreateConnection();
Consumer :
using (sessionWMQ = connectionWMQ.CreateSession(true, AcknowledgeMode.AutoAcknowledge))
{
var destination = sessionWMQ.CreateQueue("queueName");
consumerAsync = sessionWMQ.CreateConsumer(destination);
var messageListener = new MessageListener(OnMessageCallback);
consumerAsync.MessageListener = messageListener;
connectionWMQ.Start();
}
OnMessageCallback:
public void OnMessageCallback(IMessage message)
{
if (message != null)
{
numMessage++;
sessionWMQ.Commit();
}
}
This implementation causes me to download exactly one message and not the next ones.
What should I change to download all messages and even if the queue is empty it will wait for the message?
I will add that there is no error and this message is correctly taken from the queue and the next time I connect this message is gone.

Redis Pub/Sub doesn't publish message

I'm using StackExchange.Redis .NET client for Redis (installed on Windows 7).
Hostname - 127.0.0.1, port - 6379
Subscriber:
using (var connection = ConnectionMultiplexer.Connect(string.Format("{0}:{1},abortConnect=false,ConnectTimeout=10000", m_HostName, m_Port)))
{
var sub = connection.GetSubscriber();
sub.Subscribe("tasks", (channel, value) =>
{
// processing
});
}
Publisher:
using (var connection = ConnectionMultiplexer.Connect(string.Format("{0}:{1},abortConnect=false,ConnectTimeout=10000", m_HostName, m_Port)))
{
var subscriber = connection.GetSubscriber();
Logger.Debug(subscriber.IsConnected().ToString());
subscriber.Publish("tasks", message);
}
In logs I can see that subscriber is connected and there is no exception on this row:
subscriber.Publish("tasks", message);
But subscriber doesn't catch any message and Redis Desktop Manager shows that DB is empty.
In redis cli the command PUBSUB CHANNELS displays next result, that means that channel exists:
"tasks"
"__Booksleeve_MasterChanged"
Also, I pushed string to DB and it was successfull:
var db = connection.GetDatabase();
db.StringSet("key","message");
Any ideas or suggestions?

The connection of your subscriber will be disposed unless you block your client at the end of the using-statement. Therefore your client won't receive any pubsub messages after the using-statement.

NetMQ subscriber blocking with a published message

I have a Message Publishing Server class which creates a PUB socket at construction, with the following code:
this.context = NetMQContext.Create();
this.pubSocket = this.context.CreatePublisherSocket();
var portNumber = this.installerSettings.PublisherPort;
this.pubSocket.Bind("tcp://127.0.0.1:" + portNumber);
Sending a message using messagePublishingServer.Publish(message) executes:
this.pubSocket.SendMoreFrame(string.Empty).SendFrame(message);
The following xBehave test...
[Scenario]
public void PublishMessageScenario()
{
MessagePublishingServer messagePublishingServer = null;
NetMQContext context;
NetMQ.Sockets.SubscriberSocket subSocket = null;
string receivedMessage = null;
"Given a running message publishing server"._(() =>
{
var installerSettingsManager = A.Fake<IInstallerSettingsManager>();
var settings = new InstallerSettings { PublisherPort = "5348" };
A.CallTo(() => installerSettingsManager.Settings).Returns(settings);
messagePublishingServer = new MessagePublishingServer(installerSettingsManager);
});
"And a subscriber connected to the publishing server"._(() =>
{
context = NetMQContext.Create();
subSocket = context.CreateSubscriberSocket();
subSocket.Options.ReceiveHighWatermark = 1000;
subSocket.Connect("tcp://127.0.0.1:5348");
subSocket.Subscribe(string.Empty);
});
"When publishing a message"._(() =>
{
messagePublishingServer.Publish("test message");
// Receive the topic
subSocket.ReceiveFrameString();
// and the message
receivedMessage = subSocket.ReceiveFrameString();
});
"Then the subscriber must have received it"._(() =>
{
receivedMessage.Should().NotBeNullOrEmpty();
receivedMessage.Should().Be("test message");
});
}
... blocks in the first subSocket.ReceiveFrameString() which I find unexpected. Shouldn't the subscriber socket have queued the published message until the receive is called?

Publisher is like radio, if you was not connected and subscribed when the publisher published you miss the message. My tip is to put 100ms sleep after subscriber connect (only for testing).

From the source (ReceivingSocketExtensions.cs
):
/// Receive a single frame from socket, blocking until one arrives, and decode as a string using ...
public static string ReceiveFrameString([NotNull] this IReceivingSocket socket)
and
/// If no message is immediately available, return <c>false</c>.
public static bool TryReceiveFrameString([NotNull] this IReceivingSocket socket, out string frameString)

How can I consume RabbitMq messages with different type of consumers in .Net?

MessageType: "PublishX"
Consumers:
Type1ConsumerX
Type2ConsumerX
Type3ConsumerX
All of the consumers must catch messages immediately, but consume synchronously inside themselves..
For example there are 100 "PublishX" messages in the queue. Type1ConsumerX consumed 30 messages (synchronously), Type2ConsumerX consumed 50 messages(synchronously) , Type3ConsumerX consumed 100 messages(synchronously).
How can I know the message is consumed by "all type of consumers" ?
Could RabbitMQ/MassTransit PUSH messages to consumers?
Could RabbitMQ/MassTransit push messages (merging them) with intervals (1s) for decrease network traffic?
Could RabbitMQ/MassTransit push same messages to the different type of Consumers?

If I've understood the question correctly you just need to set up a basic pub/sub pattern. This will allow you to deliver the same message to multiple consumers.
Example publisher:
public static void PublishMessageToFanout()
{
var factory = new ConnectionFactory { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.ExchangeDeclare("messages", "fanout");
var message = new Message { Text = "This is a message to send" };
var json = JsonConvert.SerializeObject(message);
var body = Encoding.UTF8.GetBytes(json);
channel.BasicPublish("messages", string.Empty, null, body);
}
}
Example consumers:
SubscribeToMessages("sms-messages", (s) => Console.WriteLine("SMS Message: {0}", s));
SubscribeToMessages("email-messages", (s) => Console.WriteLine("Email Message: {0}", s));
public static void SubscribeToMessages(string queueName, Action<string> messageAction)
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.ExchangeDeclare("messages", "fanout");
channel.QueueDeclare(queueName, true, false, false, null);
channel.QueueBind(queueName, "messages", string.Empty);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume(queueName, true, consumer);
while (true)
{
var ea = consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
messageAction(message);
}
}
}
If you run the SubscribeToMessages loops in separate processes or console apps you'll see that they both print out the message whenever you call the PublishMessageToFanout. You'll also see that both queues exist in RabbitMQ Management under Queues.

Regarding the MassTransit part of your question
RabbitMQ/MassTransit PUSH messages to consumers?
Yes, MassTransit publishes messages to the bus, and then a consumer processes them
Could RabbitMQ/MassTransit push messages (merging them) with intervals (1s) for decrease network traffic?
Don't know if there is a feature for this, you could write your own but you would have to be very careful about losing the messages.
Could RabbitMQ/MassTransit push same messages to the different type of Consumers?
Yes, multiple consumers can consume the same type of message.
I've written a simple hello world app that shows the basics - http://nodogmablog.bryanhogan.net/2015/04/mass-transit-with-rabbitmq-hello-world/

RabbitMQ C# driver stops receiving messages

Do you have any pointers how to determine when a subscription problem has occurred so I can reconnect?
My service uses RabbitMQ.Client.MessagePatterns.Subscription for it's subscription. After some time, my client silently stops receiving messages. I suspect network issues as I our VPN connection is not the most reliable.
I've read through the docs for awhile looking for a key to find out when this subscription might be broken due to a network issue without much luck. I've tried checking that the connection and channel are still open, but it always seems to report that it is still open.
The messages it does process work quite well and are acknowledged back to the queue so I don't think it's an issue with the "ack".
I'm sure I must be just missing something simple, but I haven't yet found it.
public void Run(string brokerUri, Action<byte[]> handler)
{
log.Debug("Connecting to broker: {0}".Fill(brokerUri));
ConnectionFactory factory = new ConnectionFactory { Uri = brokerUri };
using (IConnection connection = factory.CreateConnection())
{
using (IModel channel = connection.CreateModel())
{
channel.QueueDeclare(queueName, true, false, false, null);
using (Subscription subscription = new Subscription(channel, queueName, false))
{
while (!Cancelled)
{
BasicDeliverEventArgs args;
if (!channel.IsOpen)
{
log.Error("The channel is no longer open, but we are still trying to process messages.");
throw new InvalidOperationException("Channel is closed.");
}
else if (!connection.IsOpen)
{
log.Error("The connection is no longer open, but we are still trying to process message.");
throw new InvalidOperationException("Connection is closed.");
}
bool gotMessage = subscription.Next(250, out args);
if (gotMessage)
{
log.Debug("Received message");
try
{
handler(args.Body);
}
catch (Exception e)
{
log.Debug("Exception caught while processing message. Will be bubbled up.", e);
throw;
}
log.Debug("Acknowledging message completion");
subscription.Ack(args);
}
}
}
}
}
}
UPDATE:
I simulated a network failure by running the server in a virtual machine and I do get an exception (RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted) when I break the connection for long enough so perhaps it isn't a network issue. Now I don't know what it would be but it fails after just a couple hours of running.

EDIT: Since I'm sill getting upvotes on this, I should point out that the .NET RabbitMQ client now has this functionality built in: https://www.rabbitmq.com/dotnet-api-guide.html#connection-recovery
Ideally, you should be able to use this and avoid manually implementing reconnection logic.
I recently had to implement nearly the same thing. From what I can tell, most of the available information on RabbitMQ assumes that either your network is very reliable or that you run a RabbitMQ broker on the same machine as any client sending or receiving messages, allowing Rabbit to deal with any connection issues.
It's really not that hard to set up the Rabbit client to be robust against dropped connections, but there are a few idiosyncrasies that you need to deal with.
The first thing you need to do turn on the heartbeat:
ConnectionFactory factory = new ConnectionFactory()
{
Uri = brokerUri,
RequestedHeartbeat = 30,
};
Setting the "RequestedHeartbeat" to 30 will make the client check every 30 seconds if the connection is still alive. Without this turned on, the message subscriber will sit there happily waiting for another message to come in without a clue that its connection has gone bad.
Turning the heartbeat on also makes the server check to see if the connection is still up, which can be very important. If a connection goes bad after a message has been picked up by the subscriber but before it's been acknowledged, the server just assumes that the client is taking a long time, and the message gets "stuck" on the dead connection until it gets closed. With the heartbeat turned on, the server will recognize when the connection goes bad and close it, putting the message back in the queue so another subscriber can handle it. Without the heartbeat, I've had to go in manually and close the connection in the Rabbit management UI so that the stuck message can get passed to a subscriber.
Second, you will need to handle OperationInterruptedException. As you noticed, this is usually the exception the Rabbit client will throw when it notices the connection has been interrupted. If IModel.QueueDeclare() is called when the connection has been interrupted, this is the exception you will get. Handle this exception by disposing of your subscription, channel, and connection and creating new ones.
Finally, you will have to handle what your consumer does when trying to consume messages from a closed connection. Unfortunately, each different way of consuming messages from a queue in the Rabbit client seems to react differently. QueueingBasicConsumer throws EndOfStreamException if you call QueueingBasicConsumer.Queue.Dequeue on a closed connection. EventingBasicConsumer does nothing, since it's just waiting for a message. From what I can tell from trying it, the Subscription class you're using seems to return true from a call to Subscription.Next, but the value of args is null. Once again, handle this by disposing of your connection, channel, and subscription and recreating them.
The value of connection.IsOpen will be updated to False when the connection fails with the heartbeat on, so you can check that if you would like. However, since the heartbeat runs on a separate thread, you will still need to handle the case where the connection is open when you check it, but closes before subscription.Next() is called.
One final thing to watch out for is IConnection.Dispose(). This call will throw a EndOfStreamException if you call dispose after the connection has been closed. This seems like a bug to me, and I don't like not calling dispose on an IDisposable object, so I call it and swallow the exception.
Putting that all together in a quick and dirty example:
public bool Cancelled { get; set; }
IConnection _connection = null;
IModel _channel = null;
Subscription _subscription = null;
public void Run(string brokerUri, string queueName, Action<byte[]> handler)
{
ConnectionFactory factory = new ConnectionFactory()
{
Uri = brokerUri,
RequestedHeartbeat = 30,
};
while (!Cancelled)
{
try
{
if(_subscription == null)
{
try
{
_connection = factory.CreateConnection();
}
catch(BrokerUnreachableException)
{
//You probably want to log the error and cancel after N tries,
//otherwise start the loop over to try to connect again after a second or so.
continue;
}
_channel = _connection.CreateModel();
_channel.QueueDeclare(queueName, true, false, false, null);
_subscription = new Subscription(_channel, queueName, false);
}
BasicDeliverEventArgs args;
bool gotMessage = _subscription.Next(250, out args);
if (gotMessage)
{
if(args == null)
{
//This means the connection is closed.
DisposeAllConnectionObjects();
continue;
}
handler(args.Body);
_subscription.Ack(args);
}
}
catch(OperationInterruptedException ex)
{
DisposeAllConnectionObjects();
}
}
DisposeAllConnectionObjects();
}
private void DisposeAllConnectionObjects()
{
if(_subscription != null)
{
//IDisposable is implemented explicitly for some reason.
((IDisposable)_subscription).Dispose();
_subscription = null;
}
if(_channel != null)
{
_channel.Dispose();
_channel = null;
}
if(_connection != null)
{
try
{
_connection.Dispose();
}
catch(EndOfStreamException)
{
}
_connection = null;
}
}