For some reason this started happening. Didn't yesterday. My client hangs after channel.BasicPublish returns in channel.Dispose. My connection is not bad, i.e. it's in open state and app is not being shutdown as suggested in https://groups.google.com/forum/?fromgroups=#!topic/rabbitmq-discuss/5nzeEqI5qxw. Both ways behave the same:
using (var channel = _connection.CreateModel()) {
//use channel here
}
and
var channel = _connection.CreateModel()
//use channel here
channel.Dispose();
They have bug 25255 on this issue - link.
For now try using a timeout around this call.
private void DisconnectWithTimeout(IConnection connection, int timeoutMillis)
{
var task = Task.Run(() => connection.Dispose());
if (!task.Wait(timeoutMillis))
{
//timeout
throw new TimeoutException("Timeout on connection.Dispose()");
}
}
Related
I am testing .NET version of gRPC to understand how to handle network failures. I put the server to one external machine and debugging the client. The server ticks with a message onnce a second and the client just shows it on the console. So when I stop my local Wi-Fi connection for seconds, then gRPC engine automatically recovers and I even get remaining values. However, if I disable Wi-Fi for longer time like a minute, then it just gets stuck. I don't even get any exceptions so that I can just handle this case and recover manually. This scenario works fine when I close the server app manually, then an exception will occur on the client. This is what I have on the client:
static async Task Main(string[] args)
{
try
{
await Subscribe();
}
catch (Exception)
{
Console.WriteLine("Fail");
Thread.Sleep(1000);
await Main(args);
}
Console.ReadLine();
}
private static async Task Subscribe()
{
using var channel = GrpcChannel.ForAddress("http://x.x.x.x:5555");
var client = new Greeter.GreeterClient(channel);
var replies = client.GerReplies(new HelloRequest { Message = "Test" });
while (await replies.ResponseStream.MoveNext(CancellationToken.None))
{
Console.WriteLine(replies.ResponseStream.Current.Message);
}
Console.WriteLine("Completed");
}
This works when the server app stopped but it doesn't work if I just disable loca Wi-Fi connection on the client side. How can I handle such a case and similar ones?
I've managed to solve it by KeepAlivePingDelay setting:
var handler = new SocketsHttpHandler
{
KeepAlivePingDelay = TimeSpan.FromSeconds(5),
KeepAlivePingTimeout = TimeSpan.FromSeconds(5),
};
using var channel = GrpcChannel.ForAddress("http://x.x.x.x:5555", new GrpcChannelOptions
{
HttpHandler = handler
});
This configuration force gRPC fail after 10 seconds in case of no connection.
I got an xUnit test which does some checks one a propietary TCP protocol implementation. The test launches a mocked server which is instructed to send the consumer in the unit test some data it should process:
[Fact]
public async void EmptyList()
{
string rawMessage = ...
MockServer server = new MockServer();
server.Start();
ManualResetEvent messageReceived = new ManualResetEvent(false);
CancellationTokenSource tokenSource = new CancellationTokenSource();
try
{
using (var cn = new Connection())
{
cn.MessageReceived += (s, e) =>
{
var m = e.Message;
Assert.Equal<CommandId>(CommandId.Command1, m.CommandId);
var l = m.GetList<int>("Users");
Assert.Equal(0, l.Count);
var emptyDict = m.GetComplexData<string>("EmptyComplex");
Assert.Equal(0, emptyDict.Count);
messageReceived.Set();
};
await cn.ConnectAsync(server.Hostname, server.Port, tokenSource.Token);
var receiveTask = cn.Receive(tokenSource.Token);
server.SendMessageToAll(rawMessage); // mock server send the connected client some sample data
messageReceived.WaitOne(TimeSpan.FromSeconds(5));
Assert.False(receiveTask.IsFaulted); // [[A]]
}
}
finally
{
tokenSource.Cancel();
}
}
When one of the asserts in the message handler MessageReceived fails, it does not give the proper error message. The console shows it caused a ThreadAbortException. In fact before inserting the line marked with A the test even completed as a success ignoring the error from the message handler.
So how can I change this that I get proper error message like Does not match expected...? Is there a way the errors in the other thread (here in receiveTask) are handled a proper unit test failures?
Would
if (receiveTask.Exception != null) throw receiveTask.Exception;
be the proper way?
Also I noted that when I use try-finally inside MessageReceived so that the test must not wait for the full timeout (messageReceived.Set()) the task itself does not get to faulted and does not have an exception. So I am not sure this is a reliable way to do this.
I am making a connection to a service i created on another server via:
using (var clientSocket = new TcpClient())
{
...
//Connect async
var result = clientSocket.BeginConnect(hostIP, portNumber, null, null);
//Wait for connection up to our timeout
if (!result.AsyncWaitHandle.WaitOne(TimeSpan.FromSeconds(5)))
{
//This is NEVER run
throw new Exception("Connection timed out.");
}
//It makes it here but shouldn't!
}
If the other server is up but the service that listens on the port is down, this still returns true! (And if the server is down, it does properly throw the exception)
Why?
How do I make it fail if the service is down (and thus nothing's listening on that port)?
Perhaps you could use the newer ConnectAsync method instead which even allows you to supply a CancellationToken in case you require your client-connecting task to abort prematurely.
using (var clientSocket = new TcpClient())
{
//Connect async and wait for connection up to our timeout
if (!clientSocket.ConnectAsync(hostIP, portNumber).Wait(TimeSpan.FromSeconds(5)))
{
throw new Exception("Connection timed out.");
}
}
It would appear that there's no way to have it fail if there's nothing listening. Instead, you can use a ReadTimeout to handle the error of nothing listening on the other end.
Do you have any pointers how to determine when a subscription problem has occurred so I can reconnect?
My service uses RabbitMQ.Client.MessagePatterns.Subscription for it's subscription. After some time, my client silently stops receiving messages. I suspect network issues as I our VPN connection is not the most reliable.
I've read through the docs for awhile looking for a key to find out when this subscription might be broken due to a network issue without much luck. I've tried checking that the connection and channel are still open, but it always seems to report that it is still open.
The messages it does process work quite well and are acknowledged back to the queue so I don't think it's an issue with the "ack".
I'm sure I must be just missing something simple, but I haven't yet found it.
public void Run(string brokerUri, Action<byte[]> handler)
{
log.Debug("Connecting to broker: {0}".Fill(brokerUri));
ConnectionFactory factory = new ConnectionFactory { Uri = brokerUri };
using (IConnection connection = factory.CreateConnection())
{
using (IModel channel = connection.CreateModel())
{
channel.QueueDeclare(queueName, true, false, false, null);
using (Subscription subscription = new Subscription(channel, queueName, false))
{
while (!Cancelled)
{
BasicDeliverEventArgs args;
if (!channel.IsOpen)
{
log.Error("The channel is no longer open, but we are still trying to process messages.");
throw new InvalidOperationException("Channel is closed.");
}
else if (!connection.IsOpen)
{
log.Error("The connection is no longer open, but we are still trying to process message.");
throw new InvalidOperationException("Connection is closed.");
}
bool gotMessage = subscription.Next(250, out args);
if (gotMessage)
{
log.Debug("Received message");
try
{
handler(args.Body);
}
catch (Exception e)
{
log.Debug("Exception caught while processing message. Will be bubbled up.", e);
throw;
}
log.Debug("Acknowledging message completion");
subscription.Ack(args);
}
}
}
}
}
}
UPDATE:
I simulated a network failure by running the server in a virtual machine and I do get an exception (RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted) when I break the connection for long enough so perhaps it isn't a network issue. Now I don't know what it would be but it fails after just a couple hours of running.
EDIT: Since I'm sill getting upvotes on this, I should point out that the .NET RabbitMQ client now has this functionality built in: https://www.rabbitmq.com/dotnet-api-guide.html#connection-recovery
Ideally, you should be able to use this and avoid manually implementing reconnection logic.
I recently had to implement nearly the same thing. From what I can tell, most of the available information on RabbitMQ assumes that either your network is very reliable or that you run a RabbitMQ broker on the same machine as any client sending or receiving messages, allowing Rabbit to deal with any connection issues.
It's really not that hard to set up the Rabbit client to be robust against dropped connections, but there are a few idiosyncrasies that you need to deal with.
The first thing you need to do turn on the heartbeat:
ConnectionFactory factory = new ConnectionFactory()
{
Uri = brokerUri,
RequestedHeartbeat = 30,
};
Setting the "RequestedHeartbeat" to 30 will make the client check every 30 seconds if the connection is still alive. Without this turned on, the message subscriber will sit there happily waiting for another message to come in without a clue that its connection has gone bad.
Turning the heartbeat on also makes the server check to see if the connection is still up, which can be very important. If a connection goes bad after a message has been picked up by the subscriber but before it's been acknowledged, the server just assumes that the client is taking a long time, and the message gets "stuck" on the dead connection until it gets closed. With the heartbeat turned on, the server will recognize when the connection goes bad and close it, putting the message back in the queue so another subscriber can handle it. Without the heartbeat, I've had to go in manually and close the connection in the Rabbit management UI so that the stuck message can get passed to a subscriber.
Second, you will need to handle OperationInterruptedException. As you noticed, this is usually the exception the Rabbit client will throw when it notices the connection has been interrupted. If IModel.QueueDeclare() is called when the connection has been interrupted, this is the exception you will get. Handle this exception by disposing of your subscription, channel, and connection and creating new ones.
Finally, you will have to handle what your consumer does when trying to consume messages from a closed connection. Unfortunately, each different way of consuming messages from a queue in the Rabbit client seems to react differently. QueueingBasicConsumer throws EndOfStreamException if you call QueueingBasicConsumer.Queue.Dequeue on a closed connection. EventingBasicConsumer does nothing, since it's just waiting for a message. From what I can tell from trying it, the Subscription class you're using seems to return true from a call to Subscription.Next, but the value of args is null. Once again, handle this by disposing of your connection, channel, and subscription and recreating them.
The value of connection.IsOpen will be updated to False when the connection fails with the heartbeat on, so you can check that if you would like. However, since the heartbeat runs on a separate thread, you will still need to handle the case where the connection is open when you check it, but closes before subscription.Next() is called.
One final thing to watch out for is IConnection.Dispose(). This call will throw a EndOfStreamException if you call dispose after the connection has been closed. This seems like a bug to me, and I don't like not calling dispose on an IDisposable object, so I call it and swallow the exception.
Putting that all together in a quick and dirty example:
public bool Cancelled { get; set; }
IConnection _connection = null;
IModel _channel = null;
Subscription _subscription = null;
public void Run(string brokerUri, string queueName, Action<byte[]> handler)
{
ConnectionFactory factory = new ConnectionFactory()
{
Uri = brokerUri,
RequestedHeartbeat = 30,
};
while (!Cancelled)
{
try
{
if(_subscription == null)
{
try
{
_connection = factory.CreateConnection();
}
catch(BrokerUnreachableException)
{
//You probably want to log the error and cancel after N tries,
//otherwise start the loop over to try to connect again after a second or so.
continue;
}
_channel = _connection.CreateModel();
_channel.QueueDeclare(queueName, true, false, false, null);
_subscription = new Subscription(_channel, queueName, false);
}
BasicDeliverEventArgs args;
bool gotMessage = _subscription.Next(250, out args);
if (gotMessage)
{
if(args == null)
{
//This means the connection is closed.
DisposeAllConnectionObjects();
continue;
}
handler(args.Body);
_subscription.Ack(args);
}
}
catch(OperationInterruptedException ex)
{
DisposeAllConnectionObjects();
}
}
DisposeAllConnectionObjects();
}
private void DisposeAllConnectionObjects()
{
if(_subscription != null)
{
//IDisposable is implemented explicitly for some reason.
((IDisposable)_subscription).Dispose();
_subscription = null;
}
if(_channel != null)
{
_channel.Dispose();
_channel = null;
}
if(_connection != null)
{
try
{
_connection.Dispose();
}
catch(EndOfStreamException)
{
}
_connection = null;
}
}
I have two .NET applications that talk to each other over a named pipe. Everything is great the first time through, but after the first message is sent, and the server is going to listen again, the WaitForConnection() method throws a System.IO.Exception with message Pipe is broken.
Why am I getting this exception here? This is my first time working with pipes, but a similar pattern has worked for me in the past with sockets.
Code ahoy!
Server:
using System.IO.Pipes;
static void main()
{
var pipe = new NamedPipeServerStream("pipename", PipeDirection.In);
while (true)
{
pipe.Listen();
string str = new StreamReader(pipe).ReadToEnd();
Console.Write("{0}", str);
}
}
Client:
public void sendDownPipe(string str)
{
using (var pipe = new NamedPipeClientStream(".", "pipename", PipeDirection.Out))
{
using (var stream = new StreamWriter(pipe))
{
stream.Write(str);
}
}
}
The first call to sendDownPipe gets the server to print the message I send just fine, but when it loops back up to listen again, it poops.
I'll post my code that seems to work - I was curious since I never did anything with pipes. I didn't find the class you name for the server-side in the relevant namespace, so here's the code based on the NamedPipeServerStream. The callback stuff is just because I couldn't be bothered with two projects.
NamedPipeServerStream s = new NamedPipeServerStream("p", PipeDirection.In);
Action<NamedPipeServerStream> a = callBack;
a.BeginInvoke(s, ar => { }, null);
...
private void callBack(NamedPipeServerStream pipe)
{
while (true)
{
pipe.WaitForConnection();
StreamReader sr = new StreamReader(pipe);
Console.WriteLine(sr.ReadToEnd());
pipe.Disconnect();
}
}
And the client does this:
using (var pipe = new NamedPipeClientStream(".", "p", PipeDirection.Out))
using (var stream = new StreamWriter(pipe))
{
pipe.Connect();
stream.Write("Hello");
}
I can repeat above block multiple times with the server running, no prob.
The problem for me has occurred when I would call pipe.WaitForConnection() from the server, after the client disconnected. The solution is to catch the IOException and call pipe.Disconnect(), and then call pipe.WaitForConnection() again:
while (true)
{
try
{
_pipeServer.WaitForConnection();
break;
}
catch (IOException)
{
_pipeServer.Disconnect();
continue;
}
}
I ran into a similar issue when I put Environment.Exit(0) at the end of my Main method, which apparently killed the entire process even though I thought the code was unreachable (because it was after a while loop waiting for a different thread to stop).
I had the same problem - it is caused by disposing server's StreamReader by Using...End Using, which also take down NamedPipeServerStream. Solution is simply don't Using...End Using it and trust in garbage collector.