I have a .NET windows service running as a consumer/subscriber which is listening to a queue for messages.
The windows service is running on the same machine as where rabbit mq server s/w is installed.
The queue if idle for 60 minutes results in the connection for it being dropped (i know this as i monitor the UI dashboard) and puts the windows service into a bad state.
This is proving to be frustrating to resolve.
I have applied heart beat setting on the rabbit mq client but this has had no effect.
The following error is what i get in the log file when connection drops
=ERROR REPORT==== 22-Aug-2017::12:20:29 ===
closing AMQP connection <0.1186.0> ([FE80::C00E:F801:A2A7:8530]:61481 ->
[FE80::C00E:F801:A2A7:8530]:5672):
missed heartbeats from client, timeout: 30s
Rbbit mq server log file settings:
[{rabbit,[ {heartbeat, 60}]}].
Client code:
var connectionFactory = new ConnectionFactory
{
HostName = hostName,
UserName = userName,
Password = password,
RequestedHeartbeat = heartBeat,
AutomaticRecoveryEnabled = true,
NetworkRecoveryInterval = TimeSpan.FromSeconds(numberOfSecondsInterval),
RequestedConnectionTimeout = RequestedConnectionTimeoutInMiliseconds
};
if (port > 0)
connectionFactory.Port = port;
var connection = connectionFactory.CreateConnection();
var model = connection.CreateModel();
model.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);
return new Tuple<IConnection, IModel>(connection, model);
heartbeat value above is set to 30 seconds,
network recovery value is set to 10 seconds &
request connection time out is set to 2 seconds
I don't know what else I'm missing here in terms of configuration??
The server where above is running from is Windows 2012 R2
Basically I'm expecting that to see the connections remain in place always regardless of idle time.
Is there a Windows OS level TCP keep-alive setting I need to make sure is in place as well?
Rabbit MQ version is 3.6.8
I managed to successfully stop the idle connections from dropping (after 60 mins) on RabbitMQ server by applying the re-connect logic that was referenced in this SO post.
To note: The answer was updated to state that the latest version of RabbitMQ client has auto connection recovery enabled so manual re-connection logic should not be needed. This was not true in my case as i had applied these settings already but i still saw the connections dropping after 60 minutes idle time. The client and the server in my scenario are on the same machine.
If anyone by any chance knows where the 60 minutes idle time setting is coming from i would be grateful, i scanned all the rabbitmq config settings and could not find anything related to it.
Related
We are using the IBM.XMS 8.0.0.5 .NET library to connect to the IBM MQ server and create our listeners.
But sometimes the VPN tunnel goes in sleep mode (this happens if one of our servers restarts for example). To prevent this, it is necessary to keep the VPN tunnel 'awake' by sending a network packet through the tunnel.
I looked around but IBM MQ does not have any implementation to test the connection to the server. I need some kind of 'ping' which will keep the tunnel up. But pinging is not allowed, I think they reject ICMP echos.
I am planning to create an async Task which will test the connection regularly depending on a configured interval.
Any advice on this one please?
PS Sometimes the connection falls asleep as I mentioned above. And without knowing anything the system engineer restarts the service which sends a dispose to the IBM server, even a dispose message leads to the connection being back up. And all items on the queue start being consumed by us while the service is being stopped... I need to solve this but I don't have a clue how I can do this.
EDIT
Connection is made and consumers are set up as follows:
var factoryFactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
// Create WMQ Connection Factory.
var cf = factoryFactory.CreateConnectionFactory();
// Set the properties
cf.SetStringProperty(XMSC.WMQ_HOST_NAME, _parameters.IbmMqHost);
cf.SetIntProperty(XMSC.WMQ_PORT, _parameters.IbmMqPort);
cf.SetStringProperty(XMSC.WMQ_CHANNEL, _parameters.IbmMqChannel);
cf.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT);
cf.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, _parameters.IbmMqQueueManager);
cf.SetStringProperty(XMSC.USERID, username);
cf.SetStringProperty(XMSC.PASSWORD, pw);
cf.SetIntProperty(XMSC.WMQ_CLIENT_RECONNECT_OPTIONS, XMSC.WMQ_CLIENT_RECONNECT);
// Create connection.
connection = cf.CreateConnection();
connection.ExceptionListener = ExceptionCallback;
session = connection.CreateSession(false, AcknowledgeMode.AutoAcknowledge);
destination = session.CreateQueue(queuename);
consumer = session.CreateConsumer(destination);
consumer.MessageListener = listener;
connection.Start();
I created an architecture formed by a client and a server those communicates on a WCF Channel in localhost, all works fine, but if there is no activity (requests from client) between the two ones for more than 10 minutes the server doesn't respond anymore. The connection is still alive but simply server is not responding to client request, so the client must disconnect and reconnect for being able to send request to the server. Maybe I let some parameters slip.
The address I used is: net.tcp://localhost:8080/ICS;
Channel type: duplex;
The problem here is in receiveTimeout. The service host uses this timeout to determine when to drop idle connections. If no message is received within the configured time span the connection is closed. By default it is 10 minutes.
Update, ReliableMessaging is not enabled therefore edit InactivityTimeout makes no sense
Whereas changing ReceiveTimeout parameter of my binding settings solves the problem.
My code:
var bind = new NetTcpBinding(); // my binding instance
var relSessionEnabled = bind.ReliableSession.Enabled; // this is false
var inactivityTimeout = bind.ReliableSession.InactivityTimeout; // this is 10 minutes
bind.ReceiveTimeout = TimeSpan.MaxValue; // this was 10 minutes before this instructuion
I have a .NET windows service that uses an open source library (Asterisk.net - C#) to listen to TCP connections on a specific port.
This service is deployed on a number of VM instances of Windows 7 (all from the same source image). The connections all come from the VM host (centos). On one (and only one) of these instances the ACK response from windows to the connecting client is delayed by three seconds on the occasional incoming connection. Other times, the ACK is sent immediately.
This delay causes the client to time out:
I'm no expert at TCP sockets, but it seems from a debug that these ACKs are sent before the connection is handed to the app (or in this case library), and even if the accepting thread is blocked, so it is a problem at the windows or .net library level?
The code that handles the inbound connections quickly hands it off to a thread and returns.
public IO.SocketConnection Accept()
{
if (tcpListener != null)
{
TcpClient tcpClient = tcpListener.AcceptTcpClient();
if (tcpClient != null)
return new IO.SocketConnection(tcpClient, encoding);
}
return null;
}
So, what could be causing this infuriating delay? What am I missing?
These 3 packets you recorded are the establishing of the connection. See https://en.m.wikipedia.org/wiki/Transmission_Control_Protocol#Connection_establishment.
However, if it happens only in 1 VM, then it's no programming issue. Try loading the VMs in different order, check firewall settings etc.
Also, test with a network terminal software like HWGroup Hercules and monitor the connections using Sysinternals TCPView.
I tried creating a poison message scenario in the following manner.
1- Created a message queue on a server (transactional queue).
2- Created a receiver app that handles incoming messages on that server.
3- Created a client app located on a client machine which sends messages to that server with the specific name for the queue.
4- I used the sender client app with the following code (C# 4.0 framework):
System.Messaging.Message mm = new System.Messaging.Message("Some msg");
mm.TimeToBeReceived = new TimeSpan(0, 0, 50);
mm.TimeToReachQueue = new TimeSpan(0, 0, 30);
mm.UseDeadLetterQueue = true;
mq.Send(mm);
So this is setting the timeout to reach queue to 30 seconds.
First test worked fine. Message went through and was received by the server app.
My second test, I disconnected my ethernet cable, then did another send from the client machine.
I can see in the message queue on the client machine that the message is waiting to be sent ("Waiting for connection"). My problem is that when it goes beyond the 30 sec (or 50sec too), the message never goes in the Dead-letter queue on the client machine.
Why is it so ? ... I was expecting it to go there some it timed-out.
Tested on Windows 7 (client) / Windows server 2008 r2 (server)
Your question is a few days old already. Did you find out anything?
My interpretation of your scenario would be that the unplugged cable is the key.
In the scenario John describes, there is an existing connection and the receiver could not process the message correctly within the set time limit.
In you scenario, however, the receiving endpoint never gets the chance to process the message, so the timeout can never occur. As you said, the state of the message is Waiting for connection. A message that was never sent cannot logically have a timeout to reach its destination.
Just ask yourself, how many resources Windows/ MSMQ would unneccessaryly sacrifice - and how often - to check MessageQueues for how-many conditions if the queues is essentially inactive? There might be a lot of queues with a lot of messages on a system.
The behavior I would expect is that if you plug the network cable back in and the connection is re-established that then, only when it is needed, your poison message wil be checked for the timeout and eventually moved to the DeadLetter queue.
You might want to check this scenario out - or did you already check it out the meantime?
I'm using C# on Windows Server 2008, and I want to receive a message from a public transactional queue on another machine in the same domain. The error looks like this:
System.Messaging.MessageQueueException: Cannot import the transaction.
at System.Messaging.MessageQueue.ReceiveCurrent(TimeSpan timeout, Int32 action, CursorHandle cursor, MessagePropertyFilter filter, MessageQueueTransaction internalTransaction, MessageQueueTransactionType transactionType)
at System.Messaging.MessageQueue.Receive(TimeSpan timeout, MessageQueueTransactionType transactionType)
at JobManagerLib.JobProcessor.Process(Action waitForNewMessageCallback) in C:\Dev\OtherProjects\POC\WindowsService\JobManagerSample\JobManagerLib\JobProcessor.cs:line 132
I've tried DTCPing, which succeeds in one direction but fails in the other. Here is the pertinent part of the log:
++++++++++++hosts ++++++++++++
127.0.0.1 localhost
::1 localhost
08-20, 15:47:22.739-->Error(0x424) at clutil.cpp #256
08-20, 15:47:22.739-->-->OpenCluster
08-20, 15:47:22.739-->-->1060(The specified service does not exist as an installed service.)
++++++++++++++++++++++++++++++++++++++++++++++
DTCping 1.9 Report for DEV-MSMQ2
++++++++++++++++++++++++++++++++++++++++++++++
RPC server is ready
++++++++++++Validating Remote Computer Name++++++++++++
08-20, 15:47:26.207-->Start DTC connection test
Name Resolution:
dev-msmq1-->192.168.22.11-->Dev-msmq1
08-20, 15:47:26.222-->Start RPC test (DEV-MSMQ2-->dev-msmq1)
RPC test failed
Does anybody have any idea why this might be failing? The Windows Firewall has been opened for MSDTC. It's hard to find much info about Windows 2008 and MSMQ.
EDIT: The queue names are FormatName:DIRECT=OS:dev-msmq1\getmap, and FormatName:DIRECT=OS:dev-msmq1\logevent. They are public, transactional queues, and Everyone has peek/receive permission on them. The pertinent part of my code is as follows:
using (TransactionScope tx = new TransactionScope(TransactionScopeOption.RequiresNew))
{
using (var queue = new MessageQueue(QueueName))
{
queue.Formatter = new XmlMessageFormatter(new string[] { _targetParameterType });
var message = queue.Receive(TimeOut, MessageQueueTransactionType.Automatic);
string messageId = message.Label;
...
}
}
Thanks
So I did find a solution: abandon the whole thing altogether and switch to using WCF and the net.Msmq binding. Now the queue communication is working fine.
Make sure, both machine's clocks are in sync. I seen this before where authentication will fail because the server and the client are off time by a minute. This will happen even if the queues are public and have permissions on everyone.
Just for completeness, allowing only MSDTC and MSMQ through the firewall is not enough when running IPv4:
You need to let ICMP traffic through the firewall as well (IPv6 can resolve host names regardless of the firewall, but your DTCPIng log indicates you are running IPv4).
I have struggled with the same error you see in DTCPing, and in my case it turned out to be triggered by the firewall blocking the ICMP traffic.