Develop Reference

NetMQ response socket poll fails after succeeding one time

I'm new to the world of ZeroMQ and I'm working through the documentation of both NetMQ and ZeroMQ as I go. I'm currently implementing (or preparing to implement) the Paranoid Pirate Pattern, and hit a snag. I have a single app which is running the server(s), clients, and eventually queue, though I haven't implemented the queue yet. Right now, there should only be one server at a time running. I can launch as many clients as I like, all communicating with the single server. I am able to have my server "crash" and restart it (manually for now, automatically soon). That all works. Or at least, restarting the server works once.
To enforce that there's only a single server running, I have a thread (which I'll call the WatchThread) which opens a response socket that binds to an address and polls for messages. When the server dies, it signals its demise and the WatchThread decrements the count when it receives the signal. Here's the code snippet that is failing:
//This is the server's main loop:
public void Start(object? count)
{
num = (int)(count ?? -1);
_model.WriteMessage($"Server {num} up");
var rng = new Random();
using ResponseSocket server = new();
server.Bind(tcpLocalhost); //This is for talking to the clients
int cycles = 0;
while (true)
{
var message = server.ReceiveFrameString();
if (message == "Kill")
{
server.SendFrame("Dying");
return;
}
if (cycles++ > 3 && rng.Next(0, 16) == 0)
{
_model.WriteMessage($"Server {num} \"Crashing\"");
RequestSocket sock = new(); //This is for talking to the WatchThread
sock.Connect(WatchThreadString);
sock.SendFrame("Dying"); //This isn't working correctly
return;
}
if(cycles > 3 && rng.Next(0, 10) == 0)
{
_model.WriteMessage($"Server {num}: Slowdown");
Thread.Sleep(1000);
}
server.SendFrame($"Server{num}: {message}");
}
}
And here's the WatchThread code:
public const string WatchThreadString = "tcp://localhost:5000";
private void WatchServers()
{
_watchThread = new ResponseSocket(WatchThreadString);
_watchThread.ReceiveReady += OnWatchThreadOnReceiveReady;
while (_listen)
{
bool result = _watchThread.Poll(TimeSpan.FromMilliseconds(1000));
}
}
private void OnWatchThreadOnReceiveReady(object? s, NetMQSocketEventArgs a)
{
lock (_countLock)
{
ServerCount--;
}
_watchThread.ReceiveFrameBytes();
}
As you can see, it's pretty straight forward. What am I missing? It seems like what should happen is exactly what happens the first time everything is instantiated: The server is supposed to go down, so it opens a new socket to the pre-existing WatchThread and sends a frame. The WatchThread receives the message and decrements the counter appropriately. It's only on the second server where things don't behave as expected...
Edit: I was able to get it to work by unbinding/closing _watchThread and recreating it... it's definitely suboptimal and it still seems like I'm missing something. It's almost as if for some reason I can only use that socket once, though I have other request sockets being used multiple times.
Additional Edit:
My netstat output with 6 clients running (kubernetes is in my host file as 127.0.0.1 as is detailed here):
TCP 127.0.0.1:5555 MyComputerName:0 LISTENING
TCP 127.0.0.1:5555 kubernetes:64243 ESTABLISHED
TCP 127.0.0.1:5555 kubernetes:64261 ESTABLISHED
TCP 127.0.0.1:5555 kubernetes:64264 ESTABLISHED
TCP 127.0.0.1:5555 kubernetes:64269 ESTABLISHED
TCP 127.0.0.1:5555 kubernetes:64272 ESTABLISHED
TCP 127.0.0.1:5555 kubernetes:64273 ESTABLISHED

Detect pipe server with same name in global namespace

Question: Is there a way how to quickly check whether particular pipename is being hosted in session 0 - preferabely during the ServiceHost.Open call?
Scenario:
Two processes PipeHost and PipeUser are trying to communicate on the system via pipe with name PeacePipe. They are not required to be started with special privileges.
PipeHost is started and hosts the PeacePipe without any problem.
PipeUser is started and connects to PeacePipe without any problem.
PipeUser tries to comunicate to PipeHost via PeacePipe, it sends messages but PipeHost doesn't see anything.
In fact PipeUser connected to DifferentPipeHostInSession0 that is hosting pipe with same name (but OS creates different pipe) in an elevated (or service) process.
Background:
ServiceHost.Open should throw AddressAlreadyInUseException when the selected pipename is already being hosted.
However it's not thrown if the pipe is hosted in session 0 and you are attempting to host the same pipe in different sessions. As windows named pipes are normally not te be used accross sessions. With the exception of pipe hosted in session 0. Any process can connect to such a pipe. This can lead to the above sceanrio.
Code:
[ServiceContract]
public interface IService
{
[OperationContract]
void Ping();
}
[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single, ConcurrencyMode = ConcurrencyMode.Multiple, IncludeExceptionDetailInFaults = true)]
public class Service: IService
{
public void Ping()
{
Console.WriteLine("Client Pinged me");
}
}
private static readonly Uri _pipeUri = new Uri("net.pipe://localhost/aaa");
private static readonly Binding _pipeBinding = new NetNamedPipeBinding(NetNamedPipeSecurityMode.None);
static void PipeHostTest()
{
ServiceHost serviceHost = new ServiceHost(new Service(), _pipeUri);
serviceHost.AddServiceEndpoint(typeof(IService), _pipeBinding, "");
try
{
//Fail here if same pipe already opened - even in Global space
serviceHost.Open();
Console.WriteLine("OPENED");
}
catch (AddressAlreadyInUseException e)
{
Console.WriteLine(e);
throw;
}
Console.ReadKey();
}
static void PipeClient()
{
ChannelFactory<IService> channelFactory =
new ChannelFactory<IService>(_pipeBinding, new EndpointAddress(_pipeUri));
var proxy = channelFactory.CreateChannel();
proxy.Ping();
}
static void Main(string[] args)
{
if (args.Any())
{
PipeClient();
}
else
{
PipeHostTest();
}
}
Run once without parameters elevated, once without parameters non-elevated. Both processes will host pipe with same name - but those are different pipes.
Then run once with any parameter. Client process will conect to the pipe hosted by elevated process.
Possible Solution:
Use a named mutex in global session new Mutex(true, "Global\\MutexForMyPipeName", out createdNew) to see if there is another process trying to do the same.
This however disqualifies even scenarios where the pipes are in 2 different sessions that do not colide.
Preferabely the ServiceHost.Open would take care about this for me as I'm using multiple bindings types (net.tcp, net.pipe, net.udp) and have single code for creating and hosting the ServiceHost. NamedPipes are the only ones that can allow creation of new host without AddressAlreadyInUseException exception while the address is actuall already in use.

TcpClient dispose order

I am writing a network layer on top of TCP and I am facing some troubles during my UnitTest phase.
Here is what I'm doing (My library is composed of multiple classes but I only show you the native instructions causing my problems, to limit the size of the post):
private const int SERVER_PORT = 15000;
private const int CLIENT_PORT = 16000;
private const string LOCALHOST = "127.0.0.1";
private TcpClient Client { get; set; }
private TcpListener ServerListener { get; set; }
private TcpClient Server { get; set; }
[TestInitialize]
public void MyTestInitialize()
{
this.ServerListener = new TcpListener(new IPEndPoint(IPAddress.Parse(LOCALHOST), SERVER_PORT));
this.Client = new TcpClient(new IPEndPoint(IPAddress.Parse(LOCALHOST), CLIENT_PORT));
this.ServerListener.Start();
}
// In this method, I just try to connect to the server
[TestMethod]
public void TestConnect1()
{
var connectionRequest = this.ServerListener.AcceptTcpClientAsync();
this.Client.Connect(LOCALHOST, SERVER_PORT);
connectionRequest.Wait();
this.Server = connectionRequest.Result;
}
// In this method, I assume there is an applicative error within the client and it is disposed
[TestMethod]
public void TestConnect2()
{
var connectionRequest = this.ServerListener.AcceptTcpClientAsync();
this.Client.Connect(LOCALHOST, SERVER_PORT);
connectionRequest.Wait();
this.Server = connectionRequest.Result;
this.Client.Dispose();
}
[TestCleanup]
public void MyTestCleanup()
{
this.ServerListener?.Stop();
this.Server?.Dispose();
this.Client?.Dispose();
}
First of all, I HAVE TO dispose the server first if I want to connect earlier to the server on the same port from the same endpoint:
If you run my tests like this, it will run successfully the first time.
The second time, it will throw an exception, in both tests, on the Connect method, arguing the port is already in use.
The only way I found to avoid this exception (and to be able to connect on the same listener from the same endpoint) is to provoke a SocketException within the Server by sending bytes to the disposed client twice (on the first sending, there is no problem, the exception is thrown only on the second sending).
I don't even need to Dispose the Server if I provoke an Exception ...
Why is the Server.Dispose() not closing the connection and freeing the port ??? Is there a better way to freeing the port than by provoking an Exception ?
Thanks in advance.
(Sorry for my English, I am not a native speaker)
Here is an example within a main fonction, to be checkout more easily:
private const int SERVER_PORT = 15000;
private const int CLIENT_PORT = 16000;
private const string LOCALHOST = "127.0.0.1";
static void Main(string[] args)
{
var serverListener = new TcpListener(new IPEndPoint(IPAddress.Parse(LOCALHOST), SERVER_PORT));
var client = new TcpClient(new IPEndPoint(IPAddress.Parse(LOCALHOST), CLIENT_PORT));
serverListener.Start();
var connectionRequest = client.ConnectAsync(LOCALHOST, SERVER_PORT);
var server = serverListener.AcceptTcpClient();
connectionRequest.Wait();
// Oops, something wrong append (wrong password for exemple), the client has to be disposed (I really want this behavior)
client.Dispose();
// Uncomment this to see the magic happens
//try
//{
//server.Client.Send(Encoding.ASCII.GetBytes("no problem"));
//server.Client.Send(Encoding.ASCII.GetBytes("oops looks like the client is disconnected"));
//}
//catch (Exception)
//{ }
// Lets try again, with a new password for example (as I said, I really want to close the connection in the first place, and I need to keep the same client EndPoint !)
client = new TcpClient(new IPEndPoint(IPAddress.Parse(LOCALHOST), CLIENT_PORT));
connectionRequest = client.ConnectAsync(LOCALHOST, SERVER_PORT);
// If the previous try/catch is commented, you will stay stuck here,
// because the ConnectAsync has thrown an exception that will be raised only during the Wait() instruction
server = serverListener.AcceptTcpClient();
connectionRequest.Wait();
Console.WriteLine("press a key");
Console.ReadKey();
}
You may need to restart Visual Studio (or wait some time) if you trigger the bug and the program refuse to let you connect.

Your port is already in use. Run netstat and see. You'll find ports still open in the TIME_WAIT state.
Because you have not gracefully closed the sockets, the network layer must keep these ports open, in case the remote endpoint sends more data. Were it to do otherwise, the sockets could receive spurious data meant for something else, corrupting the data stream.
The right way to fix this is to close the connections gracefully (i.e. use the Socket.Shutdown() method). If you want to include a test involving the remote endpoint crashing, then you'll need to handle that scenario correctly as well. For one, you should set up an independent remote process that you can actually crash. For another, your server should correctly accommodate the situation by not trying to use the port again until an appropriate time has passed (i.e. the port is actually closed and is no longer in TIME_WAIT).
On that latter point, you may want to consider actually using the work-around you've discovered: TIME_WAIT involves the scenario where the status of the remote endpoint is unknown. If you send data, the network layer can detect the failed connection and effect the socket cleanup earlier.
For additional insights, see e.g.:
Port Stuck in Time_Wait
Reconnect to the server
How can I forcibly close a TcpListener
How do I prevent Socket/Port Exhaustion?
(But do not use the recommendation found among the answers to use SO_REUSEADDR/SocketOptionName.ReuseAddress…all that does is hide the problem, and can result in corrupted data in real-world code.)

An operation on a socket could not be performed because the system lacked sufficient buffer

I might be missing something here but from what I can understand of WCF and using the ChannelFactory, I believe I am correctly creating, closing and disposing of connections.
I have created a simple service running on my PC to simply return a string. I then created a client application to rattle through creating channels, calling the service method, closing the channel and then disposing of it. Typically, when the loop gets to between 14,000 and 16,000, an error occurs saying, “An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full”. This leads me to think that I’m not disposing/cleaning up my connection but from what I’ve read I believe I’m doing everything correctly. I’ve also tried disposing and creating a new channel factory in each iteration and this has the same problem (and is noticeably slower).
Any help would be greatly appreciated. Thanks.
// Interface for the service
[ServiceContract]
public interface ITestService
{
[OperationContract, XmlSerializerFormat]
string HelloWorld(int counter);
}
//Implementation of the service
public class TestService : ITestService
{
public string HelloWorld(int counter)
{
return string.Format("{0,6:0}: Hello New World - {1}", counter, Guid.NewGuid().ToString());
}
}
// Console Application
static void Main(string[] args)
{
int count = Convert.ToInt32(args[0]);
System.Console.WriteLine("Generating " + args[0] + " calls");
using (ChannelFactory<ITestService> sharedChannelFactory = new ChannelFactory<ITestService>("BasicHttpBinding_ITestService"))
{
for (int i = 0; i < count; i++)
{
ITestService svc = null;
svc = sharedChannelFactory.CreateChannel();
string response = svc.HelloWorld(i);
((IClientChannel)svc).Close();
((IClientChannel)svc).Dispose();
System.Console.WriteLine(string.Format("{0}", response));
}
}
}

Websphere MQ Queue Depth with XMS.Net

Since we are experiencing some trouble with IBM's Websphere MQ using XMS.net (Windows service that sometimes seems to give up listening for messages on a queue) we would like to create a simple application to monitor the depths of some queues (or number of messages on the queue) to be able to alert someone when the queue depth exceeds a certain threshold. This application would be launched by the task scheduler on a specific interval and would "read out" for X queues their queue depth (and maybe some other statistics).
Our windows service is using the following code and I was hoping I could reuse that same "knowledge" for our "monitoring" application.
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
//Read config values
string QueueManager = ConfigurationManager.AppSettings["queuemanager"];
string Channel = ConfigurationManager.AppSettings["channel"];
string Queue = ConfigurationManager.AppSettings["queue"];
string HostIP = ConfigurationManager.AppSettings["host"];
int Port = int.Parse(ConfigurationManager.AppSettings["port"]);
//Create connection
var factoryfactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
var connectionfactory = factoryfactory.CreateConnectionFactory();
connectionfactory.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, QueueManager);
connectionfactory.SetStringProperty(XMSC.WMQ_HOST_NAME, HostIP);
connectionfactory.SetIntProperty(XMSC.WMQ_PORT, Port);
connectionfactory.SetStringProperty(XMSC.WMQ_CHANNEL, Channel);
connectionfactory.SetIntProperty(XMSC.WMQ_BROKER_VERSION, XMSC.WMQ_BROKER_V2);
connectionfactory.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT_UNMANAGED);
Console.WriteLine("Creating connection");
var connection = connectionfactory.CreateConnection();
connection.ExceptionListener = new ExceptionListener(OnXMSExceptionReceived);
//Create a_session
Console.WriteLine("Creating sessions");
var session = connection.CreateSession(false, AcknowledgeMode.ClientAcknowledge);
//Create queue
Console.WriteLine("Creating queue");
var queue = session.CreateQueue(string.Format("queue://{0}/{1}", QueueManager, Queue));
I have browsed through the properties of session, queue etc. but, ofcourse, there are no "current queue depth" properties. I could use GetIntProperty() or GetLongProperty() on these objects but I don't know which constant to use for that (I have seen IBM.XMS.MQC.MQIA_CURRENT_Q_DEPTH but that contains an int and Get...Property() expects a string as parameter).
Long story short: how would I go about retrieving a queues depth with the above code as a starting-point? Is it at all possible using XMS.Net?

I was able to solve it using, as Shashi suggested, the MQ API. For this you need to reference amqmdnet.dll (C:\Program Files (x86)\IBM\WebSphere MQ\bin\amqmdnet.dll) and use the following (example) code. Please note that this is a simple example, no exception handling etc. is included.
using System;
using System.Collections;
using System.Configuration;
using IBM.WMQ;
namespace Test
{
class Program
{
static void Main(string[] args)
{
//Connection properties
var properties = new Hashtable();
properties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_CLIENT);
properties.Add(MQC.CHANNEL_PROPERTY, "SOME.CHANNEL.TCP");
properties.Add(MQC.HOST_NAME_PROPERTY, "12.34.56.78");
properties.Add(MQC.PORT_PROPERTY, 1416);
var qmgr = new MQQueueManager("MYQMGR", properties);
Console.WriteLine("Local : {0}", GetQueueDepth(qmgr, "FOO.LOCALQ"));
Console.WriteLine("Report : {0}", GetQueueDepth(qmgr, "FOO.REPORTQ"));
}
public static int GetQueueDepth(MQQueueManager queuemgr, string queue)
{
return queuemgr.AccessQueue(queue,
MQC.MQOO_INPUT_AS_Q_DEF +
MQC.MQOO_FAIL_IF_QUIESCING +
MQC.MQOO_INQUIRE).CurrentDepth;
}
}
}
This performs way better than my initial "workaround".

Using XMS .NET queue depth can't be determined. Queue depth is specific to messaging providers and not JMS/XMS, so you will need to use MQ APIs to get the queue depth. You could use MQ .NET API to find the queue depth. MQQueue.CurrentDepth will give the number of message in the queue.
IMO it would be good to investigate why XMS .NET service stopped listening for messages than write another program to monitor queue depth.