Develop Reference

Performance limits for NetMQ

I am sending messages from various places on my 10G network. I am using a Pub/Sub pattern.
The messages I send are serialized using zeroFormatter and have length of approx 270 bytes.
Once I start to send over 150K message per sec, I notice the subscriber starting to miss messages.
How do I work out what the limits I can expect to be able to send are?
EDIT 1:
I am sending just under 1 billion bits/sec, which is a 10th of the capacity of my network. After this point I start to miss messages. Would this be due to CPU issues? Neither sender or receiver seem highly utilized...
private void BackgroundProcess()
{
int msgSeqNum = 0;
using (var server = new PublisherSocket())
{
server.Options.SendHighWatermark = 1000;
server.Bind(Connection);
var address = Key;
FastTickData fastTickData;
while (true)
{
if (O.TryTake(out fastTickData, 60000))
{
msgSeqNum++;
server.SendMoreFrame(address).SendMoreFrame(msgSeqNum.ToString()).SendMoreFrame(DateTime.UtcNow.ToString("yyyyMMddTHHmmssffffff")).SendFrame(ZeroFormatterSerializer.Serialize(fastTickData));
}
}
}
}

How can I both Send and Receive from a Router socket in ZeroMQ or NetMQ?

I have a Dealer <--> Router setup in NetMQ v4 where I can asynchronously send and receive messages in any direction with no problems.
I now want to formalize that into an abstraction where the server (Router) listens for any incoming message but it also need to on demand broadcasts messages to any of the connected clients (Dealers).
I am trying to avoid using Pub <--> Sub sockets as I need the subscribers to also send messages to the server. The closest pattern to what I am trying to achieve is a WebSocket client-server communication.
The first part of listening to the client messages are done in something like:
using (var server = new RouterSocket("#tcp://*:80"))
{
var addresses = new HashSet<string>();
while (true)
{
var msg = server.ReceiveMultipartMessage();
var address = Encoding.UTF8.GetString(msg[0].Buffer);
var payload = Encoding.UTF8.GetString(msg[2].Buffer);
Console.WriteLine("[Server] - Client: {0} Says: {1}", address, payload);
var contains = addresses.Contains(address);
if (!contains) { addresses.Add(address); }
msg.Clear();
msg.Append(address);
msg.AppendEmptyFrame();
msg.Append("Reply for: " + address);
server.SendMultipartMessage(msg);
}
}
Now given that the sockets are not Thread-Safe, I am stuck on finding a way to broadcasts messages (coming from a different thread on demand) to all the clients.
I can probably use the TryReceiveMultipartMessage method in the loop instead with a set timeout after which I can check a queue for any broadcast messages and then loop through each client sending such message. Something like:
using (var server = new RouterSocket("#tcp://*:80"))
{
var addresses = new HashSet<string>();
var msg = new NetMQMessage();
while (true)
{
var clientHasMsg = server.TryReceiveMultipartMessage(TimeSpan.FromSeconds(1), ref msg);
if (!clientHasMsg)
{
// Check any incoming broacast then loop through all the clients
// sending each the brodcast msg
var broadMsg = new NetMQMessage();
foreach (var item in addresses)
{
broadMsg.Append(item);
broadMsg.AppendEmptyFrame();
broadMsg.Append("This is a broadcast");
server.SendMultipartMessage(broadMsg);
broadMsg.Clear();
}
// Go back into the loop waiting for client messages
continue;
}
var address = Encoding.UTF8.GetString(msg[0].Buffer);
var payload = Encoding.UTF8.GetString(msg[2].Buffer);
Console.WriteLine("[Server] - Client: {0} Says: {1}", address, payload);
var contains = addresses.Contains(address);
if (!contains) { addresses.Add(address); }
msg.Clear();
msg.Append(address);
msg.AppendEmptyFrame();
msg.Append("Reply for: " + address);
server.SendMultipartMessage(msg);
}
}
This somehow does not feel right mainly due to:
What value for the timeout is a good value? 1 sec, 100 ms etc;
Is this the most efficient/performing solution as this program will be used to have 100k+ clients connected with each sending thousands of messages per second.
Any pointers on what's the best approach to this is very much appreciated.

You can use netmqqueue, it multi producer single consumer queue. You can add it to NetMQPoller and enqueue from multiple threads without lock.

I think PUB/SUB is a proper approach for your requirements of 100k+ clients. Nevertheless, it doesn't mean that you can't communicate back to server: use DEALER/ROUTER. Why do you think this solution is not acceptable?

MessageReceiver.ReceiveBatch() not working as intended

I am trying to receive messages in batch from the ServiceBus using the ReceiveBatch method in the MessageReceiver:
IEnumerable<BrokeredMessage> messages;
var messagingfactory = MessagingFactory.CreateFromConnectionString("ConnectionString");
var msgrcvr = messagingfactory.CreateMessageReceiver("queueName", ReceiveMode.ReceiveAndDelete);
messages = msgrcvr.ReceiveBatch(20, timeoutInSecs);
I have checked that my queue contains 20 messages using the Service Bus Explorer.
This code returns only one message in the messages structure. Is there some property I am missing?

This is only a partial-answer or work-around; the following code reliably gets all elements, but doesn't use the "ReceiveBatch"; note, as far as I can discern, Peek(i) operates on a one-based index. Also: depending on which server one is running on, if you are charged by the message pull, this may (or may not) be more expensive, so use at your own risk:
List<BrokeredMessage> dlIE = new List<BrokeredMessage>();
BrokeredMessage potentialMessage = null;
int loopCount = 1;
while ((potentialMessage = deadletterSubscriptionClient.Peek(loopCount)) != null)
{
dlIE.Add(potentialMessage); loopCount++;
}

Websphere MQ Queue Depth with XMS.Net

Since we are experiencing some trouble with IBM's Websphere MQ using XMS.net (Windows service that sometimes seems to give up listening for messages on a queue) we would like to create a simple application to monitor the depths of some queues (or number of messages on the queue) to be able to alert someone when the queue depth exceeds a certain threshold. This application would be launched by the task scheduler on a specific interval and would "read out" for X queues their queue depth (and maybe some other statistics).
Our windows service is using the following code and I was hoping I could reuse that same "knowledge" for our "monitoring" application.
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
//Read config values
string QueueManager = ConfigurationManager.AppSettings["queuemanager"];
string Channel = ConfigurationManager.AppSettings["channel"];
string Queue = ConfigurationManager.AppSettings["queue"];
string HostIP = ConfigurationManager.AppSettings["host"];
int Port = int.Parse(ConfigurationManager.AppSettings["port"]);
//Create connection
var factoryfactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
var connectionfactory = factoryfactory.CreateConnectionFactory();
connectionfactory.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, QueueManager);
connectionfactory.SetStringProperty(XMSC.WMQ_HOST_NAME, HostIP);
connectionfactory.SetIntProperty(XMSC.WMQ_PORT, Port);
connectionfactory.SetStringProperty(XMSC.WMQ_CHANNEL, Channel);
connectionfactory.SetIntProperty(XMSC.WMQ_BROKER_VERSION, XMSC.WMQ_BROKER_V2);
connectionfactory.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT_UNMANAGED);
Console.WriteLine("Creating connection");
var connection = connectionfactory.CreateConnection();
connection.ExceptionListener = new ExceptionListener(OnXMSExceptionReceived);
//Create a_session
Console.WriteLine("Creating sessions");
var session = connection.CreateSession(false, AcknowledgeMode.ClientAcknowledge);
//Create queue
Console.WriteLine("Creating queue");
var queue = session.CreateQueue(string.Format("queue://{0}/{1}", QueueManager, Queue));
I have browsed through the properties of session, queue etc. but, ofcourse, there are no "current queue depth" properties. I could use GetIntProperty() or GetLongProperty() on these objects but I don't know which constant to use for that (I have seen IBM.XMS.MQC.MQIA_CURRENT_Q_DEPTH but that contains an int and Get...Property() expects a string as parameter).
Long story short: how would I go about retrieving a queues depth with the above code as a starting-point? Is it at all possible using XMS.Net?

I was able to solve it using, as Shashi suggested, the MQ API. For this you need to reference amqmdnet.dll (C:\Program Files (x86)\IBM\WebSphere MQ\bin\amqmdnet.dll) and use the following (example) code. Please note that this is a simple example, no exception handling etc. is included.
using System;
using System.Collections;
using System.Configuration;
using IBM.WMQ;
namespace Test
{
class Program
{
static void Main(string[] args)
{
//Connection properties
var properties = new Hashtable();
properties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_CLIENT);
properties.Add(MQC.CHANNEL_PROPERTY, "SOME.CHANNEL.TCP");
properties.Add(MQC.HOST_NAME_PROPERTY, "12.34.56.78");
properties.Add(MQC.PORT_PROPERTY, 1416);
var qmgr = new MQQueueManager("MYQMGR", properties);
Console.WriteLine("Local : {0}", GetQueueDepth(qmgr, "FOO.LOCALQ"));
Console.WriteLine("Report : {0}", GetQueueDepth(qmgr, "FOO.REPORTQ"));
}
public static int GetQueueDepth(MQQueueManager queuemgr, string queue)
{
return queuemgr.AccessQueue(queue,
MQC.MQOO_INPUT_AS_Q_DEF +
MQC.MQOO_FAIL_IF_QUIESCING +
MQC.MQOO_INQUIRE).CurrentDepth;
}
}
}
This performs way better than my initial "workaround".

Using XMS .NET queue depth can't be determined. Queue depth is specific to messaging providers and not JMS/XMS, so you will need to use MQ APIs to get the queue depth. You could use MQ .NET API to find the queue depth. MQQueue.CurrentDepth will give the number of message in the queue.
IMO it would be good to investigate why XMS .NET service stopped listening for messages than write another program to monitor queue depth.

ZeroMQ/0MQ Push/Pull memory and routing issues

I have played a while with ZeroMQ and have couple questions/problems that I came up with. Would appreciate if any contributer to ZeroMQ could chime in or anyone who has used or currently uses the library.
* Let's say I have one router/forwarder and 2 different clients(c1,c2). I want to push messages from client1 to client2 through the routing device. The router pulls messages from whichever client (here client1) and publishes them to any subscribed client (here client2). I currently the only way to route such messages to the appropriate client is through pub/sub, however , a) I want to decide how to route at runtime by sending a routingTo tag along with the message body, b) I want to use push/pull to forward to clients, not pub/sub because I want to implement blocking functionality when setting the high water mark property, c) I want to have c1 and c2 connect on exactly 1 port for pushing and 1 port for subscribing. Can I somehow make changes on the router side in order to not having to use pub/sub or is pub/sub the only way to route to clients even I know on the routing side where a message is supposed to be forwarded to? I read that pub/sub drops messages when queue size exceeds the hwm which I dont want. I also do not want to implement the request/reply patters because it adds unnecessary overhead as I do not need replies.
* After running below code (Push/Pull -> Pub/Sub) and sent all messages and have received confirmation that all messages were received the client that pushed messages out still displays a huge memory footprint, apparently there are still huge amounts of messages in the Push socket's queue. Why is that and what can I do to fix that?
Here is my code:
ROUTER:
class Program
{
static void Main(string[] args)
{
using (var context = new Context(1))
{
using (Socket socketIn = context.Socket(SocketType.PULL), socketOut = context.Socket(SocketType.XPUB))
{
socketIn.HWM = 10000;
socketOut.Bind("tcp://*:5560"); //forwards on this port
socketIn.Bind("tcp://*:5559"); //listens on this port
Console.WriteLine("Router started and running...");
while (true)
{
//Receive Message
byte[] address = socketIn.Recv();
byte[] body = socketIn.Recv();
//Forward Message
socketOut.SendMore(address);
socketOut.Send(body);
}
}
}
}
}
CLIENT1:
class Program
{
static void Main(string[] args)
{
using (var context = new Context(1))
{
using (Socket socketIn = context.Socket(SocketType.SUB), socketOut= context.Socket(SocketType.PUSH))
{
byte[] iAM = Encoding.Unicode.GetBytes("Client1");
byte[] youAre = Encoding.Unicode.GetBytes("Client2");
byte[] msgBody = new byte[16];
socketOut.HWM = 10000;
socketOut.Connect("tcp://localhost:5559");
socketIn.Connect("tcp://localhost:5560");
socketIn.Subscribe(iAM);
Console.WriteLine("Press key to kick off Test Client1 Sending Routine");
Console.ReadLine();
for (int counter = 1; counter <= 10000000; counter++)
{
//Send Message
socketOut.SendMore(youAre);
socketOut.Send(msgBody);
}
Console.WriteLine("Client1: Finished Sending");
Console.ReadLine();
}
}
}
}
CLIENT2:
class Program
{
public static int msgCounter;
static void Main(string[] args)
{
msgCounter = 0;
using (var context = new Context(1))
{
using (Socket socketIn = context.Socket(SocketType.SUB), socketOut = context.Socket(SocketType.PUSH))
{
byte[] iAM = Encoding.Unicode.GetBytes("Client2");
socketOut.Connect("tcp://localhost:5559");
socketIn.Connect("tcp://localhost:5560");
socketIn.Subscribe(iAM);
Console.WriteLine("Client2: Started Listening");
//Receive First Message
byte[] address = socketIn.Recv();
byte[] body = socketIn.Recv();
msgCounter += 1;
Console.WriteLine("Received first message");
Stopwatch watch = new Stopwatch();
watch.Start();
while (msgCounter < 10000000)
{
//Receive Message
address = socketIn.Recv();
body = socketIn.Recv();
msgCounter += 1;
}
watch.Stop();
Console.WriteLine("Elapsed Time: " + watch.ElapsedMilliseconds + "ms");
Console.ReadLine();
}
}
}
}

I'm going to suggest that your architecture may be a bit off here.
1) If you need exactly one PUSH and exactly one PULL, remove the device from the middle. Devices are added to an architecture explicitly to mange multiple consumers so that you don't have to update producers each time you add a node. When/If you do get to where you need multiple consumers and/or producers, you're going to need a connection to each node on your device - that's just how they work. In this case, it sounds as though the device is overly complicating your solution.
2) The idea of having the "route to" tag really boggles my mind. Probably the biggest reason to choose messaging over other integration options is to decouple your producers and consumers so that neither side has to know anything about the other (other than where to send the messages in the case of broker-less designs). Adding routing information directly to your logic breaks this.
As to the overhead, I've never experienced this. But then, I've never used the .Net driver for ZeroMQ before and so an uneducated guess would be to look at the .Net driver itself.