Develop Reference

IBM XMS Receive method not returning messages immediately

I use IBM XMS to connect to a third party to send and receive messages.
UPDATE:
Client .Net Core 3.1
IBM XMS library version from Nuget. Tried 9.2.4 and 9.1.5 with same results
Same code used to work fine a week ago - so something must have changed in the MQ manager or somewhere in my infrastructure
SSL and client certificates
I have been using a receive with timeout for a while without problems but since last week I started to not see any messages to pick - even when they were there - but once I changed to the not timeout receive method I started again to pick messages every 5 minutes.
Looking at the XMS logs I can see the messages are actually read almost immediately with and without timeout but that XMS seems to be deciding to wait for those 5 minutes before returning the message...
I haven't changed anything in my side and the third party reassures they haven't either.
My question is: given the below code used to receive is there anything there that may be the cause of the 5 minutes wait? Any ideas on things I can try? I can share the XMS logs too if that helps.
// This is used to set the default properties in the factory before calling the receive method
private void SetConnectionProperties(IConnectionFactory cf)
{
cf.SetStringProperty(XMSC.WMQ_HOST_NAME, _mqConfiguration.Host);
cf.SetIntProperty(XMSC.WMQ_PORT, _mqConfiguration.Port);
cf.SetStringProperty(XMSC.WMQ_CHANNEL, _mqConfiguration.Channel);
cf.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, _mqConfiguration.QueueManager);
cf.SetStringProperty(XMSC.WMQ_SSL_CLIENT_CERT_LABEL, _mqConfiguration.CertificateLabel);
cf.SetStringProperty(XMSC.WMQ_SSL_KEY_REPOSITORY, _mqConfiguration.KeyRepository);
cf.SetStringProperty(XMSC.WMQ_SSL_CIPHER_SPEC, _mqConfiguration.CipherSuite);
cf.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT);
cf.SetIntProperty(XMSC.WMQ_CLIENT_RECONNECT_OPTIONS, XMSC.WMQ_CLIENT_RECONNECT);
cf.SetIntProperty(XMSC.WMQ_CLIENT_RECONNECT_TIMEOUT, XMSC.WMQ_CLIENT_RECONNECT_TIMEOUT_DEFAULT);
}
public IEnumerable<IMessage> ReceiveMessage()
{
using var connection = _connectionFactory.CreateConnection();
using var session = connection.CreateSession(false, AcknowledgeMode.AutoAcknowledge);
using var destination = session.CreateQueue(_mqConfiguration.ReceiveQueue);
using var consumer = session.CreateConsumer(destination);
connection.Start();
var result = new List<IMessage>();
var keepRunning = true;
while (keepRunning)
{
try
{
var sw = new Stopwatch();
sw.Start();
var message = _mqConfiguration.ConsumerTimeoutMs == 0 ? consumer.Receive()
: consumer.Receive(_mqConfiguration.ConsumerTimeoutMs);
if (message != null)
{
result.Add(message);
_messageLogger.LogInMessage(message);
var ellapsedMillis = sw.ElapsedMilliseconds;
if (_mqConfiguration.ConsumerTimeoutMs == 0)
{
keepRunning = false;
}
}
else
{
keepRunning = false;
}
}
catch (Exception e)
{
// We log the exception
keepRunning = false;
}
}
consumer.Close();
destination.Dispose();
session.Dispose();
connection.Close();
return result;
}

The symptoms look like a match for APAR IJ20591: Managed .NET SSL application making MQGET calls unexpectedly receives MQRC_CONNECTION_BROKEN when running in .NET Core. This impacts messages larger than 15kb and IBM MQ .net standard (core) libraries using TLS channels. See also this thread. This will be fixed in 9.2.0.5, no CDS release is listed.
It states:
Setting the heartbeat interval to lower values may reduce the frequency of occurrence.
If your .NET application is not using a CCDT you can lower the heartbeat by having the SVRCONN channel's HBINT lowered and reconnecting your application.

Issues with connections to database under low load (dotnet core)

I'm trying to load test a basic API, I've started getting some strange issues coming out of the database connection.
I've now narrowed it down to the SQL connection itself. (I'm using SELECT 1 to test connection only)
under very low load (15 calls per second) everything works exactly as expected.
under low load (25 calls per second) the first 4-5 calls come back at an okay speed, then slowing down rapidly. A lot of calls timing out due to no connection in the pool.
under medium load (50 calls per second) everything locks up entirely, nothing comes back. And I start to get strange things like A network-related or instance-specific error occurred while establishing a connection to SQL Server. coming up. Cannot get a connection from the pool again.
exec sp_who2 on the server shows no connections from dotnet either.
To make it worse the only way to recover from this is to bounce the entire service.
I have ruled out the server itself because this is happening on a powerful SQL server on-prem, an azureSql database, and a local service running on docker.
int selected = 0;
var timer = Stopwatch.StartNew();
using (SqlConnection connection = CreateNewConnection())
{
try
{
connection.Open();
selected = connection.QueryFirst<int>("SELECT 1");
timer.Stop();
}
catch (Exception e)
{
Console.WriteLine("Failed connection");
Console.WriteLine("fatal " + e.Message);
responseBuilder.AddErrors(e);
}
finally
{
connection.Close();
}
}
responseBuilder.WithResult(new {selected, ms = timer.ElapsedMilliseconds});
I've even tried disposing, and forcing the connection close manually to understand what is going on.
This is running dotnet core, and dapper (I get the same issues even without dapper)
I've also tried upping the max connection pool limit to absurd numbers like 1000, and there was no effect.
edit
After trying a bit more, I decided to try with Postgres. Which works perfectly at over 1k calls per second.
Am I missing something on in sql server itself? or on the connection?
Something to point out, These are shotgun calls. So a batch gets fired off as fast as possible, then wait for each request to return after.
Also this is using linux (and environments are docker k8s)
Someone wanted to know how connections got created
private IDbConnection CreateNewConnection()
{
var builder = new SqlConnectionStringBuilder()
{
UserID = "sa",
Password = "012Password!",
InitialCatalog = "test",
DataSource = "localhost",
MultipleActiveResultSets = true,
MaxPoolSize = 1000
};
return new SqlConnection(builder.ConnectionString);
}
Another note
Not shotgunning (waiting for the previous call to complete, before sending another) seems to have a decent enough throughput. It appears to be something with handling too many requests at the same time
Version Information
dotnet 2.1.401
SqlClient 4.5.1

I can verify something fishy is going on but it's probably not pooling. I created a console application and run it both from a Windows console and a WSL console on the same box. This way I was able to run the same code, from the same client but different OS/runtime.
On Windows, each connection took a less than a millisecond even with an absurd 500 DOP :
985 : 00:00:00.0002307
969 : 00:00:00.0002107
987 : 00:00:00.0002270
989 : 00:00:00.0002392
The same code inside WSL would take 8 seconds or more, even with a DOP of 20! Larger DOP values resulted in timeouts. 10 would produce results similar to Windows.
Once I disabled MARS though performance went back to normal :
983 : 00:00:00.0083687
985 : 00:00:00.0083759
987 : 00:00:00.0083971
989 : 00:00:00.0083938
992 : 00:00:00.0084922
991 : 00:00:00.0045206
994 : 00:00:00.0044566
That's still 20 times slower than running on Windows directly but hardly noticable until you check the numbers side by side.
This is the code I used in both cases :
static void Main(string[] args)
{
Console.WriteLine("Starting");
var options=new ParallelOptions { MaxDegreeOfParallelism = 500 };
var watch=Stopwatch.StartNew();
Parallel.For(0,1000,options,Call);
Console.WriteLine($"Finished in {watch.Elapsed}");
}
public static void Call(int i)
{
var watch = Stopwatch.StartNew();
using (SqlConnection connection = CreateNewConnection())
{
try
{
connection.Open();
var cmd=new SqlCommand($"SELECT {i}",connection);
var selected =cmd.ExecuteScalar();
Console.WriteLine($"{selected} : {watch.Elapsed}");
}
catch (Exception e)
{
Console.WriteLine($"Ooops!: {e}");
}
}
}
private static SqlConnection CreateNewConnection()
{
var builder = new SqlConnectionStringBuilder()
{
UserID = "someUser",
Password = "somPassword",
InitialCatalog = "tempdb",
DataSource = #"localhost",
MultipleActiveResultSets = true,
Pooling=true //true by default
//MaxPoolSize is 100 by default
};
return new SqlConnection(builder.ConnectionString);
}
}

Websocket server on Azure

I'm trying to host a websockets server on Azure. I'm a bit confused and was hoping you could help me.
I've followed many articles but my code is close to the one from this article : https://azure.microsoft.com/en-us/blog/introduction-to-websockets-on-windows-azure-web-sites/
public void ProcessRequest(HttpContext context)
{
if (context.IsWebSocketRequest)
{
context.AcceptWebSocketRequest(ProcessWS);
}
}
public bool IsReusable { get { return false; } }
private async Task ProcessWS(AspNetWebSocketContext context)
{
try
{
WebSocket socket = context.WebSocket;
while (true)
{
var url = context.RequestUri;
ArraySegment<byte> buffer = new ArraySegment<byte>(new byte[1024]);
WebSocketReceiveResult result = await socket.ReceiveAsync(buffer, CancellationToken.None);
if (socket.State == WebSocketState.Open)
{
string userMessage = Encoding.UTF8.GetString(buffer.Array, 0, result.Count)
.Trim(new char[] { ' ' }); // We remove the spaces before and after
// DO SOMETHING
}
else
{
break;
}
}
}
catch (Exception e)
{
Log.Info("Exception" + e.Message + " >>>" + e.StackTrace);
}
}
This works fine, I'm able to get the messages from my devices and answer to them.
But in some cases I need to send a message to another device, example :
DEVICE A sends "Tell Device B to blink"
Since it's a websockets server and Device B has already talked with the server I should have somewhere a connection opened with Device B. And when Device A asks me for it I can send a message to Device B.
But how can I achieve that with my code ? How can I find the connection to device B ? If not possible how should I do it ?
I hope my problem is described enough to be understood.
Thank you,

But how can I achieve that with my code ? How can I find the connection to device B ? If not possible how should I do it ?
According to your scenario, I followed this tutorial about webapi-and-websockets and implement my Web API project that could establish connection between clients. Here is the screenshoot for test, you could refer to it:
Additionally, you could leverage SignalR and map your client (user) to signalR connections. For more details, you could refer to Mapping SignalR Users to Connections. Also, you could refer to the git sample Microsoft.AspNet.SignalR.Samples.
UPDATE:
How would you use signalR in that case ?
For a simple way, we could retain connection and user info stored in memory, For more details, you could refer to In-memory storage. Based on this scenario, I wrote a sample project AspDotNet-SignalR-Chat, you could refer to it, and here is the screenshot for test:

TcpClient dispose order

I am writing a network layer on top of TCP and I am facing some troubles during my UnitTest phase.
Here is what I'm doing (My library is composed of multiple classes but I only show you the native instructions causing my problems, to limit the size of the post):
private const int SERVER_PORT = 15000;
private const int CLIENT_PORT = 16000;
private const string LOCALHOST = "127.0.0.1";
private TcpClient Client { get; set; }
private TcpListener ServerListener { get; set; }
private TcpClient Server { get; set; }
[TestInitialize]
public void MyTestInitialize()
{
this.ServerListener = new TcpListener(new IPEndPoint(IPAddress.Parse(LOCALHOST), SERVER_PORT));
this.Client = new TcpClient(new IPEndPoint(IPAddress.Parse(LOCALHOST), CLIENT_PORT));
this.ServerListener.Start();
}
// In this method, I just try to connect to the server
[TestMethod]
public void TestConnect1()
{
var connectionRequest = this.ServerListener.AcceptTcpClientAsync();
this.Client.Connect(LOCALHOST, SERVER_PORT);
connectionRequest.Wait();
this.Server = connectionRequest.Result;
}
// In this method, I assume there is an applicative error within the client and it is disposed
[TestMethod]
public void TestConnect2()
{
var connectionRequest = this.ServerListener.AcceptTcpClientAsync();
this.Client.Connect(LOCALHOST, SERVER_PORT);
connectionRequest.Wait();
this.Server = connectionRequest.Result;
this.Client.Dispose();
}
[TestCleanup]
public void MyTestCleanup()
{
this.ServerListener?.Stop();
this.Server?.Dispose();
this.Client?.Dispose();
}
First of all, I HAVE TO dispose the server first if I want to connect earlier to the server on the same port from the same endpoint:
If you run my tests like this, it will run successfully the first time.
The second time, it will throw an exception, in both tests, on the Connect method, arguing the port is already in use.
The only way I found to avoid this exception (and to be able to connect on the same listener from the same endpoint) is to provoke a SocketException within the Server by sending bytes to the disposed client twice (on the first sending, there is no problem, the exception is thrown only on the second sending).
I don't even need to Dispose the Server if I provoke an Exception ...
Why is the Server.Dispose() not closing the connection and freeing the port ??? Is there a better way to freeing the port than by provoking an Exception ?
Thanks in advance.
(Sorry for my English, I am not a native speaker)
Here is an example within a main fonction, to be checkout more easily:
private const int SERVER_PORT = 15000;
private const int CLIENT_PORT = 16000;
private const string LOCALHOST = "127.0.0.1";
static void Main(string[] args)
{
var serverListener = new TcpListener(new IPEndPoint(IPAddress.Parse(LOCALHOST), SERVER_PORT));
var client = new TcpClient(new IPEndPoint(IPAddress.Parse(LOCALHOST), CLIENT_PORT));
serverListener.Start();
var connectionRequest = client.ConnectAsync(LOCALHOST, SERVER_PORT);
var server = serverListener.AcceptTcpClient();
connectionRequest.Wait();
// Oops, something wrong append (wrong password for exemple), the client has to be disposed (I really want this behavior)
client.Dispose();
// Uncomment this to see the magic happens
//try
//{
//server.Client.Send(Encoding.ASCII.GetBytes("no problem"));
//server.Client.Send(Encoding.ASCII.GetBytes("oops looks like the client is disconnected"));
//}
//catch (Exception)
//{ }
// Lets try again, with a new password for example (as I said, I really want to close the connection in the first place, and I need to keep the same client EndPoint !)
client = new TcpClient(new IPEndPoint(IPAddress.Parse(LOCALHOST), CLIENT_PORT));
connectionRequest = client.ConnectAsync(LOCALHOST, SERVER_PORT);
// If the previous try/catch is commented, you will stay stuck here,
// because the ConnectAsync has thrown an exception that will be raised only during the Wait() instruction
server = serverListener.AcceptTcpClient();
connectionRequest.Wait();
Console.WriteLine("press a key");
Console.ReadKey();
}
You may need to restart Visual Studio (or wait some time) if you trigger the bug and the program refuse to let you connect.

Your port is already in use. Run netstat and see. You'll find ports still open in the TIME_WAIT state.
Because you have not gracefully closed the sockets, the network layer must keep these ports open, in case the remote endpoint sends more data. Were it to do otherwise, the sockets could receive spurious data meant for something else, corrupting the data stream.
The right way to fix this is to close the connections gracefully (i.e. use the Socket.Shutdown() method). If you want to include a test involving the remote endpoint crashing, then you'll need to handle that scenario correctly as well. For one, you should set up an independent remote process that you can actually crash. For another, your server should correctly accommodate the situation by not trying to use the port again until an appropriate time has passed (i.e. the port is actually closed and is no longer in TIME_WAIT).
On that latter point, you may want to consider actually using the work-around you've discovered: TIME_WAIT involves the scenario where the status of the remote endpoint is unknown. If you send data, the network layer can detect the failed connection and effect the socket cleanup earlier.
For additional insights, see e.g.:
Port Stuck in Time_Wait
Reconnect to the server
How can I forcibly close a TcpListener
How do I prevent Socket/Port Exhaustion?
(But do not use the recommendation found among the answers to use SO_REUSEADDR/SocketOptionName.ReuseAddress…all that does is hide the problem, and can result in corrupted data in real-world code.)

C# server - is this good way for timeouting and disconnecting?

On my multithreaded server I am experiencincg troubles with connections that are not coming from the proper Client and so hang unathorized. I did not want to create new thread only for checking if clients are connected for some time without authorization. Instead of this, I have add this checking to RecieveData thread, shown on the code below. Do you see some performance issue or this is acceptable? The main point is that everytime client is connected (and Class client is instantionized) it starts stopwatch. And so I add to this thread condition - if the time is greater than 1 and the client is still not authorized, its added on the list of clients determinated for disconnection. Thanks
EDIT: This While(true) is RecieveData thread. I am using async. operations - from tcplistener.BeginAccept to threadpooling. I have updated the code to let you see more.
protected void ReceiveData()
{
List<Client> ClientsToDisconnect = new List<Client>();
List<System.Net.Sockets.Socket> sockets = new List<System.Net.Sockets.Socket>();
bool noClients = false;
while (true)
{
sockets.Clear();
this.mClientsSynchronization.TryEnterReadLock(-1);
try
{
for (int i = 0; i < this.mClientsValues.Count; i++)
{
Client c = this.mClientsValues[i];
if (!c.IsDisconnected && !c.ReadingInProgress)
{
sockets.Add(c.Socket);
}
//clients connected more than 1 second without recieved name are suspect and should be disconnected
if (c.State == ClientState.NameNotReceived && c.watch.Elapsed.TotalSeconds > 1)
ClientsToDisconnect.Add(c);
}
if (sockets.Count == 0)
{
continue;
}
}
finally
{
this.mClientsSynchronization.ExitReadLock();
}
try
{
System.Net.Sockets.Socket.Select(sockets, null, null, RECEIVE_DATA_TIMEOUT);
foreach (System.Net.Sockets.Socket s in sockets)
{
Client c = this.mClients[s];
if (!c.SetReadingInProgress())
{
System.Threading.ThreadPool.QueueUserWorkItem(c.ReadData);
}
}
//remove clients in ClientsToDisconnect
foreach (Client c in ClientsToDisconnect)
{
this.RemoveClient(c,true);
}
}
catch (Exception e)
{
//this.OnExceptionCaught(this, new ExceptionCaughtEventArgs(e, "Exception when reading data."));
}
}
}

I think I see what you are trying to do and I think a better way would be to store new connections in a holding area until they have properly connected.
I'm not positive but it looks like your code could drop a valid connection. If a new connection is made after the checking section and the second section takes more than a second all the timers would time out before you could verify the connections. This would put the new connections in both the socket pool AND the ClientsToDisconnect pool. Not good. You would drop a currently active connection and chaos would ensue.
To avoid this, I would make the verification of a connection a seperate thread from the using of the connection. That way you won't get bogged down in timing issues (well...you still will but that is what happens when we work with threads and sockets) and you are sure that all the sockets you are using won't get closed by you.

My gut reaction is that while (true) plus if (sockets.Count == 0) continue will lead to heartache for your CPU. Why don't you put this on a Timer or something so that this function is only called every ~.5s? Is the 1s barrier really that important?