Develop Reference

Azure Event Hub timeouts

I occasionally see timeout errors in event hub client when trying to send messages to Azure event hub. It looks like resource limit is reached for client but I'm not sure...
Here the code:
MessagingFactory messagingFactory = null;
EventHubClient hubClient = null;
try
{
messagingFactory = MessagingFactory.CreateFromConnectionString(this.connectionString);
hubClient = messagingFactory.CreateEventHubClient(this.configurationManager.EventHubName);
var batchIterator = new EventBatchIterator<T>(events);
foreach (var batch in batchIterator)
{
await hubClient.SendBatchAsync(batch);
}
}
catch (Exception e)
{
this.logger.Error("An error occurred during sent messages to event hub", e);
}
finally
{
if (hubClient != null)
{
await hubClient.CloseAsync();
}
if (messagingFactory != null)
{
await messagingFactory.CloseAsync();
}
}
Exception is:
An error occurred during communication with 'N/A'. Check the
connection information, then retry.
Microsoft.ServiceBus.Messaging.MessagingCommunicationException
A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond
System.Net.Sockets.SocketException

According to you mentioned execption indicates that a user-initiated operation is taking longer than the operation timeout. My workaround is that you could increase OperationTimeout or retry count.
Demo code for increasing timeout
var builder = new ServiceBusConnectionStringBuilder("connection string")
{
TransportType = TransportType.Amqp,
OperationTimeout = TimeSpan.FromSeconds(90)
};
messagingFactory = MessagingFactory.CreateFromConnectionString(builder.ToString());
More info about Timeout exception you refer to this document.
Common causes
There are two common causes for this error: incorrect configuration, or a transient service error.
Incorrect configuration The operation timeout might be too small for the operational condition. The default value for the operation timeout in the client SDK is 60 seconds. Check to see if your code has the value set to something too small. Note that the condition of the network and CPU usage can affect the time it takes for a particular operation to complete, so the operation timeout should not be set to a very small value.
Transient service error Sometimes the Event Hubs service can experience delays in processing requests; for example, during periods of high traffic. In such cases, you can retry your operation after a delay, until the operation is successful. If the same operation still fails after multiple attempts, please visit the Azure service status site to see if there are any known service outages.

RabbitMQ c# System.IO.EndOfStreamException

I get the following exception when a consumer is blocking to receive a message from the SharedQueue:
Unhandled Exception: System.IO.EndOfStreamException: SharedQueue closed
at RabbitMQ.Util.SharedQueue.EnsureIsOpen()
at RabbitMQ.Util.SharedQueue.Dequeue()
at Consumer.Program.Main(String[] args) in c:\Users\pdecker\Documents\Visual
Studio 2012\Projects\RabbitMQTest1\Consumer\Program.cs:line 33
Here is the line of code that is being executed when the exception is thrown:
BasicDeliverEventArgs e = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
So far I have seen the exception occuring when rabbitMQ is inactive. Our application needs to have the consumer always connected and listening for keystrokes. Does anyone know the cause of this problem? Does anyone know how to recover from this problem?
Thanks in advance.

The consumer is tied to the channel:
var consumer = new QueueingBasicConsumer(channel);
So if the channel has closed, then the consumer will not be able to fetch any additional events once the local Queue has been cleared.
Check for the channel to be open with
channel.IsOpen == true
and that the Queue has available events with
if( consumer.Queue.Count() > 0 )
before calling:
BasicDeliverEventArgs e = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
To be more specific, I would check the following before calling Dequeue()
if( !channel.IsOpen || !connection.IsOpen )
{
Your_Connection_Channel_Init_Function();
consumer = new QueueingBasicConsumer(channel); // consumer is tied to channel
}
if( consumer.Queue.Any() )
BasicDeliverEventArgs e = (BasicDeliverEventArgs)consumer.Queue.Dequeue();

Don't worry this is just expected behavior, it means there is no message left in queue to process. Don't even try it is not gonna work...
consumer.Queue.Any()
Just catch the EndOfStreamException:
private void ConsumeMessages(string queueName)
{
using (IConnection conn = factory.CreateConnection())
{
using (IModel channel = conn.CreateModel())
{
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume(queueName, false, consumer);
Trace.WriteLine(string.Format("Waiting for messages from: {0}", queueName));
while (true)
{
BasicDeliverEventArgs ea = null;
try
{
ea = consumer.Queue.Dequeue();
}
catch (EndOfStreamException endOfStreamException)
{
Trace.WriteLine(endOfStreamException);
// If you want to end listening end of queue call break;
break;
}
if (ea == null) break;
var body = ea.Body;
// Consume message how you want
Thread.Sleep(300);
channel.BasicAck(ea.DeliveryTag, false);
}
}
}
}

There is another possible source of trouble: your corporate firewall.
Thats because such firewall can drop your connection to RabbitMQ when the connection is idle for a certain amount of time.
Although RabbitMQ connection has a heartbeat feature to prevent this, if the heartbeat pulse happens after the firewall connection timeout, it is useless.
This is the default heartbeat interval configuration in seconds:
Default: 60 (580 prior to release 3.5.5)
From RabbitMQ:
Detecting Dead TCP Connections with Heartbeats
Introduction
Network can fail in many ways, sometimes pretty subtle (e.g. high
ratio packet loss). Disrupted TCP connections take a moderately long
time (about 11 minutes with default configuration on Linux, for
example) to be detected by the operating system. AMQP 0-9-1 offers a
heartbeat feature to ensure that the application layer promptly finds
out about disrupted connections (and also completely unresponsive
peers).
Heartbeats also defend against certain network equipment which
may terminate "idle" TCP connections.
That happened to us and we solved the problem by decreasing the Heartbeat Timeout Interval in the global configuration:
In your rabbitmq.config, find the heartbeat and set it to a value smaller than that of your firewall rule.
You can change the interval in your client, too:
Enabling Heartbeats with Java Client To configure the heartbeat
timeout in the Java client, set it with
ConnectionFactory#setRequestedHeartbeat before creating a connection:
ConnectionFactory cf = new ConnectionFactory();
// set the heartbeat timeout to 60 seconds
cf.setRequestedHeartbeat(60);
Enabling Heartbeats with the .NET Client To configure the heartbeat
timeout in the .NET client, set it with
ConnectionFactory.RequestedHeartbeat before creating a connection:
var cf = new ConnectionFactory();
//set the heartbeat timeout to 60 seconds
cf.RequestedHeartbeat = 60;

The answers here that say that this is the expected behavior are correct, however I would argue that it's bad to have it throw an exception by design like this.
from the documentation: "Callers of Dequeue() will block if no items are available until some other thread calls Enqueue() or the queue is closed. In the latter case this method will throw EndOfStreamException."
So, like GlenH7 said, you have to check that channel is open before calling Dequeue() (IModel.IsOpen).
However, what if the channel closes while Dequeue() is blocking? I think it's best to call Queue.DequeueNoWait(null), and block the thread yourself by waiting for it to return something that isn't null. So, something like:
while(channel.IsOpen)
{
var args = consumer.Queue.DequeueNoWait(null);
if(args == null) continue;
//...
}
This way, it won't throw that exception.

Quick check if WCF endpoint is available (and how to handle if not)

I have a very simple WCF service running.
An ASP.NET web application is consuming this service from code behind during pageload like this:
try
{
using (var myWCFClient = new MyWCFClient())
{
int myInt = myWCFClient.GetValue();
}
}
catch (Exception ex)
{
}
Where "MyWCFClient" is the proxy object which is autogenerated when adding a service reference to my project.
Everything works fine, but the problem is when the Service Endpoint is down for some reason, it takes more than 30 seconds for the EndpointNotFoundException is catched. Of course this is unacceptable because it delays pageload since everything is synchronous. And because i have no simple mechanism to push the data to the page async after pageload, an async call to the WCF service is not preferred.
I also tried to set the sentTimeout in web.config to 5 seconds, but this doen't solve the problem.
When is set the OperationTimeout of the InnerChannel to 5 seconds like this...
((IContextChannel)myWCFClient.InnerChannel).OperationTimeout = TimeSpan.FromSeconds(5);
...i do get a TimeOutException, but this exception is not throwed within 5 seconds, it also takes more than 30 seconds...
Does anybody knows how to handle this situation, or is there a way to quick check if the service is running before doing the actual call???

We use a ping function on the host that simply returns True in the response payload, and if you response takes longer than 5 seconds, we throw our own exception and exit the process. All the logic is on the client side, minus the Ping() function. We don't wait for the timeout exception.

WCF: Timeout error

I have a piece of code that calls a WCF service that is hosted on a server.
The code keeps looping around and around calling this method over and over again. (It's asking for a 'status', so it's not doing any work at all).
That's fine except that after a short period of time I get an error:
This request operation sent to net.tcp://serverName:9001/service1 did not receive a reply within the configured timeout (00:00:09.9843754)
And suddenly i cannot get to the server at all EVER. I increased the timeout to 1min but still the same problem. Note that the program that hosts the service is doing nothing else, just offering it's 'status'. So it's not an issue with the WCF service app being busy.
I think it's a problem with the code calling the service because when i re-start the app it can connect to the service just fine ... until after another short time i get the timeout error again. For this reason i don't thnk it's a network error either, as when I restart the app it's ok for a period of time.
Here is the code i use to call the service. Do i need to dispose of the ChannelFactory after each call to clean it up or what am i doing worng?
NetTcpBinding binding = new NetTcpBinding(SecurityMode.Message);
binding.Security.Message.ClientCredentialType = MessageCredentialType.Windows;
EndpointAddress endPoint = new EndpointAddress(new Uri(clientPath));
ChannelFactory<IClient> channel = new ChannelFactory<IClient>(binding, endPoint);
channel.Faulted += new EventHandler(channel_Faulted);
IClient client = channel.CreateChannel();
((IContextChannel)client).OperationTimeout = new TimeSpan(0, 0, 10);
ClientStatus clientStatus = client.GetStatus();

You do have to close client connections after you finish calling GetStatus. The best way to do this is to use a using block. But you can also do something like this after your call client.GetStatus()
ClientStatus clientStatus = client.GetStatus();
try
{
if (client.State != System.ServiceModel.CommunicationState.Faulted)
{
client.Close();
}
}
catch (Exception ex)
{
client.Abort();
}

WCF: Self-host a NetTcp service, doesn't resolve

I am trying to host a WCF service, using NetTcpBinding in a Windows service. (I'm going to use it as an API for various clients both Web and Windows32) Obviously, I am doing this within a test host before putting it in a Windows service.
I have the following contract:
namespace yyy.xxx.Server.API.WCF
{
[ServiceContract]
public interface ISecureSessionBroker
{
[OperationContract]
string GetSessionToken(string username, string encryptedPassword, string clientApiKey, string clientAddress);
}
}
with the following implementation:
namespace yyy.xxx.Server.API.WCF
{
public class SecureSessionBroker : ISecureSessionBroker
{
#region ~ from ISecureSessionBroker ~
public string GetSessionToken(string username, string encryptedPassword, string clientApiKey, string clientAddress)
{
return Guid.NewGuid().ToString();
}
#endregion
}
}
I am hosting the WCF service using the code below (within a class/method):
try
{
_secureSessionBrokerHost = new ServiceHost(typeof(SecureSessionBroker));
NetTcpBinding netTcpBinding = new NetTcpBinding();
_secureSessionBrokerHost.AddServiceEndpoint(typeof(ISecureSessionBroker), netTcpBinding, "net.tcp://localhost:8080/secureSessionBrokerTcp");
int newLimit = _secureSessionBrokerHost.IncrementManualFlowControlLimit(100);
// Open the ServiceHost to start listening for messages.
_secureSessionBrokerHost.Open();
}
catch (Exception ex)
{
throw;
}
The key thing here is that I do not want to have to rely on an App.config file. Everything must be configured programmatically. When I run this code, the service appears to come "up" and listen. (ie. I have no exceptions)
BUT when I use the client code below:
string secureSessionBrokerUrl = string.Format("{0}/secureSessionBrokerTcp","net.tcp://localhost/8080",url);
EndpointAddress endpointAddress=new EndpointAddress(secureSessionBrokerUrl);
System.ServiceModel.Channels.Binding binding = new NetTcpBinding();
yyy.xxx.Windows.AdminTool.API.WCF.SecureSessions.SecureSessionBrokerClient
client = new yyy.xxx.Windows.AdminTool.API.WCF.SecureSessions.SecureSessionBrokerClient(binding,endpointAddress);
string sessionToken=client.GetSessionToken("", "", ""); // exception here
MessageBox.Show(sessionToken);
... I always get an exception. At the moment, I am getting:
This request operation sent to
net.tcp://localhost:8080/secureSessionBrokerTcp
did not receive a reply within the
configured timeout (00:01:00). The
time allotted to this operation may
have been a portion of a longer
timeout. This may be because the
service is still processing the
operation or because the service was
unable to send a reply message.
Please consider increasing the
operation timeout (by casting the
channel/proxy to IContextChannel and
setting the OperationTimeout property)
and ensure that the service is able to
connect to the client.
So I guess it cannot resolve the service.
Where am I going wrong? How do I test for the existence of the service over TCP? I have used the SvcTraceViewer and I just get the same message, so no news there.
I would prefer to ask the user for a URL of the service, so they would enter "net.tcp://localhost:8080" or something, which would then be used as a BaseAddress for the various calls to the SecureSessionBroker (and other) WCF services ... without resorting to App.config.
Unfortunately, all the examples I can find all use the App.config.
Interestingly, I can host the service using the VS Host and the client connects fine. (Using:
D:\dev2008\xxx\yyy.xxx.Server>WcfSvcHost.exe /service:bin/debug/yyy.
xxx.Server.dll /config:App.config)

Ok, it came to me in a flash of inspiration.
I was using a Windows Form (alarm bells) to "host" the service. Clicking out of the form, I used a bit of code to call the service (included) on a button click. Of course, the service was not in its own thread, so the service could not respond.
I've fixed it by putting the Service container (which contains the host) within its own thread:
Thread thread = new Thread(new ThreadStart(_serviceWrapper.Start));
thread.Start();
The Start() method sets up the ServiceHost.
I incorrectly thought that while a WCF Service Host will create threads for incoming requests, it will only do this if it is in its own non-blocking thread (ie. not a UI thread).
Hope it helps someone else.