Azure Event Hub timeouts - c#

I occasionally see timeout errors in event hub client when trying to send messages to Azure event hub. It looks like resource limit is reached for client but I'm not sure...
Here the code:
MessagingFactory messagingFactory = null;
EventHubClient hubClient = null;
try
{
messagingFactory = MessagingFactory.CreateFromConnectionString(this.connectionString);
hubClient = messagingFactory.CreateEventHubClient(this.configurationManager.EventHubName);
var batchIterator = new EventBatchIterator<T>(events);
foreach (var batch in batchIterator)
{
await hubClient.SendBatchAsync(batch);
}
}
catch (Exception e)
{
this.logger.Error("An error occurred during sent messages to event hub", e);
}
finally
{
if (hubClient != null)
{
await hubClient.CloseAsync();
}
if (messagingFactory != null)
{
await messagingFactory.CloseAsync();
}
}
Exception is:
An error occurred during communication with 'N/A'. Check the
connection information, then retry.
Microsoft.ServiceBus.Messaging.MessagingCommunicationException
A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond
System.Net.Sockets.SocketException

According to you mentioned execption indicates that a user-initiated operation is taking longer than the operation timeout. My workaround is that you could increase OperationTimeout or retry count.
Demo code for increasing timeout
var builder = new ServiceBusConnectionStringBuilder("connection string")
{
TransportType = TransportType.Amqp,
OperationTimeout = TimeSpan.FromSeconds(90)
};
messagingFactory = MessagingFactory.CreateFromConnectionString(builder.ToString());
More info about Timeout exception you refer to this document.
Common causes
There are two common causes for this error: incorrect configuration, or a transient service error.
Incorrect configuration The operation timeout might be too small for the operational condition. The default value for the operation timeout in the client SDK is 60 seconds. Check to see if your code has the value set to something too small. Note that the condition of the network and CPU usage can affect the time it takes for a particular operation to complete, so the operation timeout should not be set to a very small value.
Transient service error Sometimes the Event Hubs service can experience delays in processing requests; for example, during periods of high traffic. In such cases, you can retry your operation after a delay, until the operation is successful. If the same operation still fails after multiple attempts, please visit the Azure service status site to see if there are any known service outages.

Related

MQTTClient Listener disconnects randomly

I have an MQTT listener written in c#.
The program is running in Azure and
for some reason after a period of time, it gets disonnected with an exception:
"The operation has timed out." or
"Exception of type 'MQTTnet.Exceptions.MqttCommunicationTimedOutException' was thrown."
On production, the listener must always be online so on disconnect event i'm reconnecting, but it happens randomly, it can get disconnected 4 times a day and sometimes it can stay online without disconnect for a few days.
question is, why is it happening? the device that it listens to is sending a timestamp request every few minutes, but it should be very fast and shouldn't cause a timeout.
Here is the code:
private static IMqttClient _client;
private static IMqttClientOptions _options;
static async Task Main(string[] args)
{
//create subscriber client
var factory = new MqttFactory();
_client = factory.CreateMqttClient();
//configure options
_options = new MqttClientOptionsBuilder()
.WithClientId("ListenerClient")
.WithTcpServer(Utility.brokerIp, Utility.brokerPort).WithCredentials(Utility.brokerUser, Utility.brokerPassword)
.WithCleanSession()
.Build();
//Handlers
_client.UseConnectedHandler(e =>
{
Console.WriteLine("Connected successfully with MQTT Brokers Topic.");
WriteToLog("***Connected To MQTT Listener.***");
//Subscribe to topics******************
});
_client.UseDisconnectedHandler(e =>
{
WriteToLog("***DisConnected From MQTT Listener.***");
WriteToLog(e.Exception.Message);
_client.ConnectAsync(_options).Wait();
return;
});
_client.UseApplicationMessageReceivedHandler(async e =>
{
//manage messages
});
//Connect
_client.ConnectAsync(_options).Wait();
Task.Run(() => Thread.Sleep(Timeout.Infinite)).Wait();
_client.DisconnectAsync().Wait();
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
We had a similar issue at one time, I believe the queue you are trying to connect to has very intermittent traffic. And then whatever service, or server you have hosting the queue it self, is setup to hibernate the queue when no traffic hits the queue for some predetermined period of time.
When you then try to use the queue, the "timeout" happens because the queue can't wake up from hibernation quick enough for you to get a processed message through.
if the queue is Azure hosted, try to get Azure support to confirm this is the case, if you are on premise hosting, try to verify your own configuration is set this way, and reduce the "wait" period to something deliberately small like 30 seconds, and verify that a hibernated queue, causes a time-out.

Azure Service Bus MessageReceiver Receive() web sockets error

I am working with a c# program within my network and am able to post messages to an Azure Service Bus queue. When receiving them, I get an exception on MessageReceiver.Receive(). The code and error is below;
MessagingFactory factory = MessagingFactory.CreateFromConnectionString(QueueConnectionString);
//Receiving a message
MessageReceiver testQueueReceiver = factory.CreateMessageReceiver(QueueName);
using (BrokeredMessage retrievedMessage = testQueueReceiver.Receive(new TimeSpan(0, 0, 20)))
{
try
{
var message = new StreamReader(retrievedMessage.GetBody<Stream>(), Encoding.UTF8).ReadToEnd();
retrievedMessage.Complete();
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
retrievedMessage.Abandon();
}
}
The error gets thrown on the 'using' line at
testQueueReceiver.Receive(...);
The server rejected the upgrade request. 400 This endpoint is only for web-socket requests
I can't find anything on the web with the exception of one post which seems to suggest it is a firewall / ports issue. I have all the azure service bus ports outbound open (9350-9354, 80, 443) locally but there is a chance the 9000's are blocked at the firewall. Should it require these? Any pointers would be greatly appreciated !
Service MessagingCommunication Exception - The End point is only for web socket requests
I'm just wondering why don't you use OnMessage instead of polling the queue?
var connectionString = "";
var queueName = "samplequeue";
var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
client.OnMessage(message =>
{
Console.WriteLine(String.Format("Message body: {0}", message.GetBody<String>()));
Console.WriteLine(String.Format("Message id: {0}", message.MessageId));
message.Complete()
});
This was fixed due to some proxy issues.
The account that the code was running under was an async service. I needed to log in as that account, open IE and go to connections (LAN) and remove the proxy checkboxes (detect settings automatically, etc). Once this was done, the code bypassed the proxy and worked fine.

The operation did not complete within the allotted timeout of 00:01:00

I am using code snippet to send message into the service bus topic.
try
{
// sb is instance of ServiceBusConfig.GetServiceBusForChannel
await sb.SendAsync(message);
}
catch (Exception ex)
{
this.logger.LogError(
"chanel",
"An error occurred while sending a notification: " + ex.Message,
ex);
throw;
}
and implementation is
public async Task SendAsync(BrokeredMessage message)
{
if (this.topicClient == null)
{
this.topicClient = TopicClient.CreateFromConnectionString(this.primaryConnectionString, this.topicPath);
this.topicClient.RetryPolicy = this.retryPolicy;
}
await this.topicClient.SendAsync(message);
}
Error:-
"ErrorCode,12005,Message,""An error occurred while sending a
notification: The operation did not complete within the allotted
timeout of 00:01:00. The time allotted to this operation may have
been a portion of a longer timeout. For more information on exception
types and proper exception handling,
Exception,""Microsoft.ServiceBus.Messaging.MessagingException: The
operation did not complete within the allotted timeout of 00:01:00.
The time allotted to this operation may have been a portion of a
longer timeout.
For more information on exception types and proper exception handling
This happens occasionally. Azure can change and shift their underlying hardware willy nilly. These sort of errors pop up every now and then. As long you have some appropriate retry logic where appropriate, the message will eventually get through ...

WPF and WCF with Named Pipes

The following code works just fine in WinForms:
if (System.Configuration.ConfigurationManager.AppSettings["Data.UseNamedPipes"] == "true")
{
AES.Cloud.DataAccessLayer.TheEDGEContext.UseLocalDb = true;
string address = "net.pipe://localhost/" + Guid.NewGuid() + "/DataManager";
var svc = AES.InProcFactory.CreateInstance<AES.Cloud.DataService.EstimatingDataService, AES.Cloud.DataService.IDataManager>(2147483647, 2147483647, address);
DataService<AES.TheEdge.Framework.Services.Contracts.IDataManager>._namedPipeAddress = address;
var ss = svc.GetNextKey(); //<-- Calling service here.
}
but when I use WPF it times out (exception at the bottom but it's useless) when I call the service, however if I invoke this code on a different thread it works just fine in WPF:
System.Threading.ThreadPool.QueueUserWorkItem( new System.Threading.WaitCallback( (o) =>{
//same code as above... no problem, no timeout, no blocking
}
Is there some issue using the startup/main thread to do this sort of thing? Confused in Jupiter...
Exception:
System.TimeoutException: This request operation sent to net.pipe://localhost/c1a59720-11fc-49c3-9d82-8185203a6f5d/DataManager did not receive a reply within the configured timeout (00:01:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.

RabbitMQ c# System.IO.EndOfStreamException

I get the following exception when a consumer is blocking to receive a message from the SharedQueue:
Unhandled Exception: System.IO.EndOfStreamException: SharedQueue closed
at RabbitMQ.Util.SharedQueue.EnsureIsOpen()
at RabbitMQ.Util.SharedQueue.Dequeue()
at Consumer.Program.Main(String[] args) in c:\Users\pdecker\Documents\Visual
Studio 2012\Projects\RabbitMQTest1\Consumer\Program.cs:line 33
Here is the line of code that is being executed when the exception is thrown:
BasicDeliverEventArgs e = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
So far I have seen the exception occuring when rabbitMQ is inactive. Our application needs to have the consumer always connected and listening for keystrokes. Does anyone know the cause of this problem? Does anyone know how to recover from this problem?
Thanks in advance.
The consumer is tied to the channel:
var consumer = new QueueingBasicConsumer(channel);
So if the channel has closed, then the consumer will not be able to fetch any additional events once the local Queue has been cleared.
Check for the channel to be open with
channel.IsOpen == true
and that the Queue has available events with
if( consumer.Queue.Count() > 0 )
before calling:
BasicDeliverEventArgs e = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
To be more specific, I would check the following before calling Dequeue()
if( !channel.IsOpen || !connection.IsOpen )
{
Your_Connection_Channel_Init_Function();
consumer = new QueueingBasicConsumer(channel); // consumer is tied to channel
}
if( consumer.Queue.Any() )
BasicDeliverEventArgs e = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
Don't worry this is just expected behavior, it means there is no message left in queue to process. Don't even try it is not gonna work...
consumer.Queue.Any()
Just catch the EndOfStreamException:
private void ConsumeMessages(string queueName)
{
using (IConnection conn = factory.CreateConnection())
{
using (IModel channel = conn.CreateModel())
{
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume(queueName, false, consumer);
Trace.WriteLine(string.Format("Waiting for messages from: {0}", queueName));
while (true)
{
BasicDeliverEventArgs ea = null;
try
{
ea = consumer.Queue.Dequeue();
}
catch (EndOfStreamException endOfStreamException)
{
Trace.WriteLine(endOfStreamException);
// If you want to end listening end of queue call break;
break;
}
if (ea == null) break;
var body = ea.Body;
// Consume message how you want
Thread.Sleep(300);
channel.BasicAck(ea.DeliveryTag, false);
}
}
}
}
There is another possible source of trouble: your corporate firewall.
Thats because such firewall can drop your connection to RabbitMQ when the connection is idle for a certain amount of time.
Although RabbitMQ connection has a heartbeat feature to prevent this, if the heartbeat pulse happens after the firewall connection timeout, it is useless.
This is the default heartbeat interval configuration in seconds:
Default: 60 (580 prior to release 3.5.5)
From RabbitMQ:
Detecting Dead TCP Connections with Heartbeats
Introduction
Network can fail in many ways, sometimes pretty subtle (e.g. high
ratio packet loss). Disrupted TCP connections take a moderately long
time (about 11 minutes with default configuration on Linux, for
example) to be detected by the operating system. AMQP 0-9-1 offers a
heartbeat feature to ensure that the application layer promptly finds
out about disrupted connections (and also completely unresponsive
peers).
Heartbeats also defend against certain network equipment which
may terminate "idle" TCP connections.
That happened to us and we solved the problem by decreasing the Heartbeat Timeout Interval in the global configuration:
In your rabbitmq.config, find the heartbeat and set it to a value smaller than that of your firewall rule.
You can change the interval in your client, too:
Enabling Heartbeats with Java Client To configure the heartbeat
timeout in the Java client, set it with
ConnectionFactory#setRequestedHeartbeat before creating a connection:
ConnectionFactory cf = new ConnectionFactory();
// set the heartbeat timeout to 60 seconds
cf.setRequestedHeartbeat(60);
Enabling Heartbeats with the .NET Client To configure the heartbeat
timeout in the .NET client, set it with
ConnectionFactory.RequestedHeartbeat before creating a connection:
var cf = new ConnectionFactory();
//set the heartbeat timeout to 60 seconds
cf.RequestedHeartbeat = 60;
The answers here that say that this is the expected behavior are correct, however I would argue that it's bad to have it throw an exception by design like this.
from the documentation: "Callers of Dequeue() will block if no items are available until some other thread calls Enqueue() or the queue is closed. In the latter case this method will throw EndOfStreamException."
So, like GlenH7 said, you have to check that channel is open before calling Dequeue() (IModel.IsOpen).
However, what if the channel closes while Dequeue() is blocking? I think it's best to call Queue.DequeueNoWait(null), and block the thread yourself by waiting for it to return something that isn't null. So, something like:
while(channel.IsOpen)
{
var args = consumer.Queue.DequeueNoWait(null);
if(args == null) continue;
//...
}
This way, it won't throw that exception.

Categories

Resources