Do you have any pointers how to determine when a subscription problem has occurred so I can reconnect?
My service uses RabbitMQ.Client.MessagePatterns.Subscription for it's subscription. After some time, my client silently stops receiving messages. I suspect network issues as I our VPN connection is not the most reliable.
I've read through the docs for awhile looking for a key to find out when this subscription might be broken due to a network issue without much luck. I've tried checking that the connection and channel are still open, but it always seems to report that it is still open.
The messages it does process work quite well and are acknowledged back to the queue so I don't think it's an issue with the "ack".
I'm sure I must be just missing something simple, but I haven't yet found it.
public void Run(string brokerUri, Action<byte[]> handler)
{
log.Debug("Connecting to broker: {0}".Fill(brokerUri));
ConnectionFactory factory = new ConnectionFactory { Uri = brokerUri };
using (IConnection connection = factory.CreateConnection())
{
using (IModel channel = connection.CreateModel())
{
channel.QueueDeclare(queueName, true, false, false, null);
using (Subscription subscription = new Subscription(channel, queueName, false))
{
while (!Cancelled)
{
BasicDeliverEventArgs args;
if (!channel.IsOpen)
{
log.Error("The channel is no longer open, but we are still trying to process messages.");
throw new InvalidOperationException("Channel is closed.");
}
else if (!connection.IsOpen)
{
log.Error("The connection is no longer open, but we are still trying to process message.");
throw new InvalidOperationException("Connection is closed.");
}
bool gotMessage = subscription.Next(250, out args);
if (gotMessage)
{
log.Debug("Received message");
try
{
handler(args.Body);
}
catch (Exception e)
{
log.Debug("Exception caught while processing message. Will be bubbled up.", e);
throw;
}
log.Debug("Acknowledging message completion");
subscription.Ack(args);
}
}
}
}
}
}
UPDATE:
I simulated a network failure by running the server in a virtual machine and I do get an exception (RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted) when I break the connection for long enough so perhaps it isn't a network issue. Now I don't know what it would be but it fails after just a couple hours of running.
EDIT: Since I'm sill getting upvotes on this, I should point out that the .NET RabbitMQ client now has this functionality built in: https://www.rabbitmq.com/dotnet-api-guide.html#connection-recovery
Ideally, you should be able to use this and avoid manually implementing reconnection logic.
I recently had to implement nearly the same thing. From what I can tell, most of the available information on RabbitMQ assumes that either your network is very reliable or that you run a RabbitMQ broker on the same machine as any client sending or receiving messages, allowing Rabbit to deal with any connection issues.
It's really not that hard to set up the Rabbit client to be robust against dropped connections, but there are a few idiosyncrasies that you need to deal with.
The first thing you need to do turn on the heartbeat:
ConnectionFactory factory = new ConnectionFactory()
{
Uri = brokerUri,
RequestedHeartbeat = 30,
};
Setting the "RequestedHeartbeat" to 30 will make the client check every 30 seconds if the connection is still alive. Without this turned on, the message subscriber will sit there happily waiting for another message to come in without a clue that its connection has gone bad.
Turning the heartbeat on also makes the server check to see if the connection is still up, which can be very important. If a connection goes bad after a message has been picked up by the subscriber but before it's been acknowledged, the server just assumes that the client is taking a long time, and the message gets "stuck" on the dead connection until it gets closed. With the heartbeat turned on, the server will recognize when the connection goes bad and close it, putting the message back in the queue so another subscriber can handle it. Without the heartbeat, I've had to go in manually and close the connection in the Rabbit management UI so that the stuck message can get passed to a subscriber.
Second, you will need to handle OperationInterruptedException. As you noticed, this is usually the exception the Rabbit client will throw when it notices the connection has been interrupted. If IModel.QueueDeclare() is called when the connection has been interrupted, this is the exception you will get. Handle this exception by disposing of your subscription, channel, and connection and creating new ones.
Finally, you will have to handle what your consumer does when trying to consume messages from a closed connection. Unfortunately, each different way of consuming messages from a queue in the Rabbit client seems to react differently. QueueingBasicConsumer throws EndOfStreamException if you call QueueingBasicConsumer.Queue.Dequeue on a closed connection. EventingBasicConsumer does nothing, since it's just waiting for a message. From what I can tell from trying it, the Subscription class you're using seems to return true from a call to Subscription.Next, but the value of args is null. Once again, handle this by disposing of your connection, channel, and subscription and recreating them.
The value of connection.IsOpen will be updated to False when the connection fails with the heartbeat on, so you can check that if you would like. However, since the heartbeat runs on a separate thread, you will still need to handle the case where the connection is open when you check it, but closes before subscription.Next() is called.
One final thing to watch out for is IConnection.Dispose(). This call will throw a EndOfStreamException if you call dispose after the connection has been closed. This seems like a bug to me, and I don't like not calling dispose on an IDisposable object, so I call it and swallow the exception.
Putting that all together in a quick and dirty example:
public bool Cancelled { get; set; }
IConnection _connection = null;
IModel _channel = null;
Subscription _subscription = null;
public void Run(string brokerUri, string queueName, Action<byte[]> handler)
{
ConnectionFactory factory = new ConnectionFactory()
{
Uri = brokerUri,
RequestedHeartbeat = 30,
};
while (!Cancelled)
{
try
{
if(_subscription == null)
{
try
{
_connection = factory.CreateConnection();
}
catch(BrokerUnreachableException)
{
//You probably want to log the error and cancel after N tries,
//otherwise start the loop over to try to connect again after a second or so.
continue;
}
_channel = _connection.CreateModel();
_channel.QueueDeclare(queueName, true, false, false, null);
_subscription = new Subscription(_channel, queueName, false);
}
BasicDeliverEventArgs args;
bool gotMessage = _subscription.Next(250, out args);
if (gotMessage)
{
if(args == null)
{
//This means the connection is closed.
DisposeAllConnectionObjects();
continue;
}
handler(args.Body);
_subscription.Ack(args);
}
}
catch(OperationInterruptedException ex)
{
DisposeAllConnectionObjects();
}
}
DisposeAllConnectionObjects();
}
private void DisposeAllConnectionObjects()
{
if(_subscription != null)
{
//IDisposable is implemented explicitly for some reason.
((IDisposable)_subscription).Dispose();
_subscription = null;
}
if(_channel != null)
{
_channel.Dispose();
_channel = null;
}
if(_connection != null)
{
try
{
_connection.Dispose();
}
catch(EndOfStreamException)
{
}
_connection = null;
}
}
Related
I am very new to .NET remoting. We have an MVC website making call to a Windows application using TCP connection (.NET remoting). There is a timer running every 30 seconds which makes a call via TCP, but after it finishes, the connection still remains established. As a result, after a few days, the server is throwing an exception because all ports are used up. Then we have to restart the app pool to resolve the issue. I am not sure how can we close the port after use, so that we can use it again. We have a lot of users for the site.
Below is the client side code to register a channel
bool Registered = false;
foreach (IChannel ic in ChannelServices.RegisteredChannels)
{
if (ic.ChannelName == ChannelNameRemoting)
{
return ic;
}
}
// Channel not found yet
IDictionary channelConfig = new Hashtable();
channelConfig["name"] = ChannelNameRemoting;
channelConfig["secure"] = false;
BinaryClientFormatterSinkProvider defaultClientSink = new BinaryClientFormatterSinkProvider();
if (remotingSinkProvider == null)
{
remotingSinkProvider = new CustomClientChannelSinkProvider();
remotingSinkProvider.EncodingDecodingProviderEvent
+= new CustomClientChannelSinkProvider.EncodingDecodingProviderDelegate(remotingSinkProvider_EncodingDecodingProviderEvent);
}
defaultClientSink.Next = remotingSinkProvider;
IChannel channel = new TcpChannel(channelConfig, defaultClientSink, null);
if (!Registered)
{
ChannelServices.RegisterChannel(channel, false);
}
return channel;
Below call will create a connection to connect to the server in timer.
var connection = (testConnect)Activator.GetObject(
typeof(testConnect),
"tcp://" + _remotingUrl + ":" + _remotingPort + "/test/test4"
);
connection.FunctionCall();
I guess you need to UnRegister you channel.
Please check the link
So ultimately we create the instance of TcpChannel and register it to channel service, and then finally return it to parent method, so that can use and can make the call. in the file/class where you have method that return the channel instance should also have another method to unregister the tcp channel, and once you are done with the call, you need to call this new methd to unregister the channel, and that will close the connection.
public void UnregisterMyTcpChannel(TcpChannel yourChannelInstance)
{
ChannelServices.UnregisterChannel(yourChannelInstance);
}
I have a service running as local SYSTEM that launches another application with the user credentials.
That second app is only a tray icon that shows balloon tips to the user with the string received using the callback method. This second application connects to the WCF in duplex mode.
My problem is that for some reason the connection to the WCF is finalized at the end of the method Main. So I cannot send a callback message to the app right after the execution, included in the last line "kiosk.MyStart(args);". there the callback is still pointing to null.
Any idea how could I solve this issue?
static void Main(string []args)
{
if (Environment.UserInteractive)
{
// Start the WCf service
var host = new ServiceHost(typeof(WcfService));
host.Open();
//Launch the Kiosk Agent which connects to the WCF
bool ret = ProcessAsUser.Launch("C:\\Program Files (x86)\\KIOSK\\KioskAgent.exe");
WinService kiosk = new WinService(args);
// some checks and a welcome message is sent to the user.
kiosk.MyStart(args);
//...
//...
}
}
Edit: to clarify a bit more, inside kiosk.MyStart method is where I try to execute the callback to show a welcome message, but the callback is still NULL.
As a result I assume that the client was not properly started for any reason and I launch it once again...
if (WcfService.Callback != null)
WcfService.Callback.UIMessageOnCallback(UIMessage);
else
ProcessAsUser.Launch("C:\\Program Files (x86)\\KIOSK\\KioskAgent.exe");
Add a try catch block over the callback method, if the client not reachable it falls in the catch you can unsubscribe it. Is also good practice send a keepalive message to your client, to check if it available.
private void InformClient(ClientInfo clientInfo)
{
var subscribers = this._subscriberRepository.GetAll();
foreach (var subscriber in subscribers)
{
try
{
if (subscriber.Callback.FireInformClient(clientInfo));
{
//If subscriber not reachable, unsubscribe it
this._subscriberRepository.Unsubscribe(subscriber.ClientId);
}
}
catch (Exception exception)
{
//If subscriber not reachable, unsubscribe it
this._subscriberRepository.Unsubscribe(subscriber.ClientId);
Log.Error(nameof(InformClient), exception);
}
}
}
IClientCallback
public interface IClientCallback
{
[OperationContract]
bool FireInformClient(ClientInfo clientInfo);
}
If you have more subscribers for example a terminal, server create a subscriberRepository to manage all subscribers.
var callback = OperationContext.Current.GetCallbackChannel<IClientCallback>();
if (this._subscriberRepository.Subscribe(clientId, callback))
{
return true;
}
I am making a connection to a service i created on another server via:
using (var clientSocket = new TcpClient())
{
...
//Connect async
var result = clientSocket.BeginConnect(hostIP, portNumber, null, null);
//Wait for connection up to our timeout
if (!result.AsyncWaitHandle.WaitOne(TimeSpan.FromSeconds(5)))
{
//This is NEVER run
throw new Exception("Connection timed out.");
}
//It makes it here but shouldn't!
}
If the other server is up but the service that listens on the port is down, this still returns true! (And if the server is down, it does properly throw the exception)
Why?
How do I make it fail if the service is down (and thus nothing's listening on that port)?
Perhaps you could use the newer ConnectAsync method instead which even allows you to supply a CancellationToken in case you require your client-connecting task to abort prematurely.
using (var clientSocket = new TcpClient())
{
//Connect async and wait for connection up to our timeout
if (!clientSocket.ConnectAsync(hostIP, portNumber).Wait(TimeSpan.FromSeconds(5)))
{
throw new Exception("Connection timed out.");
}
}
It would appear that there's no way to have it fail if there's nothing listening. Instead, you can use a ReadTimeout to handle the error of nothing listening on the other end.
I am writing a network layer on top of TCP and I am facing some troubles during my UnitTest phase.
Here is what I'm doing (My library is composed of multiple classes but I only show you the native instructions causing my problems, to limit the size of the post):
private const int SERVER_PORT = 15000;
private const int CLIENT_PORT = 16000;
private const string LOCALHOST = "127.0.0.1";
private TcpClient Client { get; set; }
private TcpListener ServerListener { get; set; }
private TcpClient Server { get; set; }
[TestInitialize]
public void MyTestInitialize()
{
this.ServerListener = new TcpListener(new IPEndPoint(IPAddress.Parse(LOCALHOST), SERVER_PORT));
this.Client = new TcpClient(new IPEndPoint(IPAddress.Parse(LOCALHOST), CLIENT_PORT));
this.ServerListener.Start();
}
// In this method, I just try to connect to the server
[TestMethod]
public void TestConnect1()
{
var connectionRequest = this.ServerListener.AcceptTcpClientAsync();
this.Client.Connect(LOCALHOST, SERVER_PORT);
connectionRequest.Wait();
this.Server = connectionRequest.Result;
}
// In this method, I assume there is an applicative error within the client and it is disposed
[TestMethod]
public void TestConnect2()
{
var connectionRequest = this.ServerListener.AcceptTcpClientAsync();
this.Client.Connect(LOCALHOST, SERVER_PORT);
connectionRequest.Wait();
this.Server = connectionRequest.Result;
this.Client.Dispose();
}
[TestCleanup]
public void MyTestCleanup()
{
this.ServerListener?.Stop();
this.Server?.Dispose();
this.Client?.Dispose();
}
First of all, I HAVE TO dispose the server first if I want to connect earlier to the server on the same port from the same endpoint:
If you run my tests like this, it will run successfully the first time.
The second time, it will throw an exception, in both tests, on the Connect method, arguing the port is already in use.
The only way I found to avoid this exception (and to be able to connect on the same listener from the same endpoint) is to provoke a SocketException within the Server by sending bytes to the disposed client twice (on the first sending, there is no problem, the exception is thrown only on the second sending).
I don't even need to Dispose the Server if I provoke an Exception ...
Why is the Server.Dispose() not closing the connection and freeing the port ??? Is there a better way to freeing the port than by provoking an Exception ?
Thanks in advance.
(Sorry for my English, I am not a native speaker)
Here is an example within a main fonction, to be checkout more easily:
private const int SERVER_PORT = 15000;
private const int CLIENT_PORT = 16000;
private const string LOCALHOST = "127.0.0.1";
static void Main(string[] args)
{
var serverListener = new TcpListener(new IPEndPoint(IPAddress.Parse(LOCALHOST), SERVER_PORT));
var client = new TcpClient(new IPEndPoint(IPAddress.Parse(LOCALHOST), CLIENT_PORT));
serverListener.Start();
var connectionRequest = client.ConnectAsync(LOCALHOST, SERVER_PORT);
var server = serverListener.AcceptTcpClient();
connectionRequest.Wait();
// Oops, something wrong append (wrong password for exemple), the client has to be disposed (I really want this behavior)
client.Dispose();
// Uncomment this to see the magic happens
//try
//{
//server.Client.Send(Encoding.ASCII.GetBytes("no problem"));
//server.Client.Send(Encoding.ASCII.GetBytes("oops looks like the client is disconnected"));
//}
//catch (Exception)
//{ }
// Lets try again, with a new password for example (as I said, I really want to close the connection in the first place, and I need to keep the same client EndPoint !)
client = new TcpClient(new IPEndPoint(IPAddress.Parse(LOCALHOST), CLIENT_PORT));
connectionRequest = client.ConnectAsync(LOCALHOST, SERVER_PORT);
// If the previous try/catch is commented, you will stay stuck here,
// because the ConnectAsync has thrown an exception that will be raised only during the Wait() instruction
server = serverListener.AcceptTcpClient();
connectionRequest.Wait();
Console.WriteLine("press a key");
Console.ReadKey();
}
You may need to restart Visual Studio (or wait some time) if you trigger the bug and the program refuse to let you connect.
Your port is already in use. Run netstat and see. You'll find ports still open in the TIME_WAIT state.
Because you have not gracefully closed the sockets, the network layer must keep these ports open, in case the remote endpoint sends more data. Were it to do otherwise, the sockets could receive spurious data meant for something else, corrupting the data stream.
The right way to fix this is to close the connections gracefully (i.e. use the Socket.Shutdown() method). If you want to include a test involving the remote endpoint crashing, then you'll need to handle that scenario correctly as well. For one, you should set up an independent remote process that you can actually crash. For another, your server should correctly accommodate the situation by not trying to use the port again until an appropriate time has passed (i.e. the port is actually closed and is no longer in TIME_WAIT).
On that latter point, you may want to consider actually using the work-around you've discovered: TIME_WAIT involves the scenario where the status of the remote endpoint is unknown. If you send data, the network layer can detect the failed connection and effect the socket cleanup earlier.
For additional insights, see e.g.:
Port Stuck in Time_Wait
Reconnect to the server
How can I forcibly close a TcpListener
How do I prevent Socket/Port Exhaustion?
(But do not use the recommendation found among the answers to use SO_REUSEADDR/SocketOptionName.ReuseAddress…all that does is hide the problem, and can result in corrupted data in real-world code.)
I have written Windows service, which perform Modbus WriteMultipleRegisters function call over TCP using NModbus library to 3-party devices every 10 minutes (ticks of System.Threading.Timer).
Occasionally this connection hang up open usually during network problems. As the device accepts only one Modbus connection at time and others are refused, connection during all next ticks fail with SocketException - ConnectionRefused.
But the device automatically closes connections which don't respond after short time. Something must keep connection open at my side even for two days. What's more when my Service is restarted, everything is fine again. So there is definitely some forgotten open connection. But I didn't manage to reproduce this bug in dev, so I don't where/when.. connection hang up. I only know that next connection is refused.
I do the modbus function call with this part of code:
using (TcpClient client = new TcpClient(device.ip, 502))
{
using (Modbus.Device.ModbusIpMaster master = Modbus.Device.ModbusIpMaster.CreateIp(client))
{
master.WriteMultipleRegisters(500, new ushort[] { 0xFF80 });
}
}
device.ip is string containing IP address of device - it's correct, confirmed from SocketException details.
As I'm using using statement dispose is called on both objects.
I have looked trough NModbus source code and everything is disposed correctly.
Any idea how its possible that with this code connection is not closed?
I agree with nemec. If you review the documentation for TcpClient.Dispose if does not specifically mention closing the connection. It frees managed and unmanaged resources by default, but it may not correctly tear down the connection.
Try changing your code to:
using (TcpClient client = new TcpClient(device.ip, 502))
{
try
{
using (Modbus.Device.ModbusIpMaster master = Modbus.Device.ModbusIpMaster.CreateIp(client))
{
master.WriteMultipleRegisters(500, new ushort[] { 0xFF80 });
}
}
catch(Exception e)
{
// Log exception
}
finally
{
client.Close();
}
}
That way you are doing a clean close before dispose and it should clean up even if the Modbus protocol throws some kind of exception.
did you play with TcpClient.LingerState Property
defualt setting could cause problems with resetting winsock
check it out
http://msdn.microsoft.com/pl-pl/library/system.net.sockets.tcpclient.lingerstate%28v=vs.110%29.aspx
This is not an answer, but a comment with code. We have this same issue on some of our installed computers, but not all of them. The issue itself is also very intermittent, sometimes going months without happening. I am hoping someone can find an answer. Here is our brute force destroy / reconnect code that does not work:
try
{
try
{
try
{
// Close the stream
var stream = _tcpClient.GetStream();
if (stream != null)
stream.Close();
}
catch { }
try
{
// Close the socket
if (_tcpClient.Client != null)
_tcpClient.Client.Close();
}
catch { }
// Close the client
_tcpClient.Close();
_tcpClient = null;
}
catch { }
if (_device != null)
{
_device.Dispose();
_device = null;
}
}
catch { }
System.Threading.Thread.Sleep(1000);