Develop Reference

Deferring and re-receiving a deferred message in an IHostBuilder hosted service

If the processing of an Azure Service Bus message depends on another resource, e.g. an API or a database service, and this resource is not available, not calling CompleteMessageAsync() is not an option, because the message will be immediately received again until the Max Delivery Count is reached, and then put into the DLQ. If an API is down for maintenance, we want to wait a bit before retrying.
One of the answers to this question has the general steps for deferring and receiving deferred messages. This is a little better than Microsoft's documentation, but not enough for me to understand the intent of the API, and how it is to be implemented in a hosted service that basically sits in ServiceBusProcessor.StartProcessingAsync all day long.
This is the basic structure of my service:
public class ServiceBusWatcher : IHostedService, IDisposable
{
public Task StartAsync(CancellationToken stoppingToken)
{
ReceiveMessagesAsync();
return Task.CompletedTask;
}
private async void ReceiveMessagesAsync()
{
ServiceBusClient client = new ServiceBusClient(connectionString);
processor = client.CreateProcessor(queueName, new ServiceBusProcessorOptions());
processor.ProcessMessageAsync += MessageHandler;
await processor.StartProcessingAsync();
}
async Task MessageHandler(ProcessMessageEventArgs args)
{
// a dependency is not available that allows me to process a message. so:
await args.DeferMessageAsync(args.Message);
Once the message is deferred, it is my understanding that the processor will not get to it anymore (or will it?). Instead, I have to use ReceiveDeferredMessageAsync() to receive it, along with the sequence number of the originally received message.
In my case, it will make sense to wait minutes or hours before trying again.
This could be done with a separate service that uses a timer and an explicit call to ReceiveDeferredMessageAsync(), as opposed to using a ServiceBusProcessor. I also suppose that the deferred message sequence numbers will have to be persisted in non-volatile storage so that they don't get lost.
Does this sound like a viable approach? I don't like having to remember its sequence numbers so that I can get to a message later. It goes against everything that using a message queue brings to the table in the first place.
Or, instead of deferring, I could just post a new "internal" message with the sequence number and use the ScheduledEnqueueTimeUtc property to delay receiving it. Once I receive this message, I could call ReceiveDeferredMessageAsync() with that sequence number to get to the original message. This seems elegant at the surface, but messages could quickly multiply if there is a longer outage of a dependency.
Another idea that could work without another service: I could complete and repost the payload of the message and set ScheduledEnqueueTimeUtc to a time in the future, as described in another answer to the question I mentioned earlier. Assuming that this works (Microsoft's documentation does not mention what this property is for), it seems simple and clean, and I like simple.
How have you solved this? Is there a better/preferred way that balances low complexity with high robustness without requiring a large amount of code?

Deferring a message works when you know what message you want to retrieve later and your receiver will have the message sequence number saved to retrieve the deferred message. If the receiver has no ability to save message sequence number, the delaying the message is a better option. Delaying a message will mean to copy the original message data into a newly scheduled one and completing the original message. That way the consumer doesn't have to neither hold on to the message sequence number nor initiate the retrieval of a specific message.

Azure EventHub: Send Async performance

I have pretty naive code :
public async Task Produce(string topic, object message, MessageHeader messageHeaders)
{
try
{
var producerClient = _EventHubProducerClientFactory.Get(topic);
var eventData = CreateEventData(message, messageHeaders);
messageHeaders.Times?.Add(DateTime.Now);
await producerClient.SendAsync(new EventData[] { eventData });
messageHeaders.Times?.Add(DateTime.Now);
//.....
Log.Info($"Milliseconds spent: {(messageHeaders.Times[1]- messageHeaders.Times[0]).TotalMilliseconds});
}
}
private EventData CreateEventData(object message, MessageHeader messageHeaders)
{
var eventData = new EventData(Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(message)));
eventData.Properties.Add("CorrelationId", messageHeaders.CorrelationId);
if (messageHeaders.DateTime != null)
eventData.Properties.Add("DateTime", messageHeaders.DateTime?.ToString("s"));
if (messageHeaders.Version != null)
eventData.Properties.Add("Version", messageHeaders.Version);
return eventData;
}
in logs I had values for almost 1 second (~ 800 milliseconds)
What could be a reason for such long execution time?

The EventHubProducerClient opens connections to the Event Hubs service lazily, waiting until the first time an operation requires it. In your snippet, the call to SendAsync triggers an AMQP connection to be created, an AMQP link to be created, and authentication to be performed.
Unless the client is closed, most future calls won't incur that overhead as the connection and link are persistent. Most being an important distinction in that statement, as the client may need to reconnect in the face of a network error, when activity is low and the connection idles out, or if the Event Hubs service terminates the connection/link.
As Serkant mentions, if you're looking to understand timings, you'd probably be best served using a library like Benchmark.NET that works ove a large number of iterations to derive statistically meaningful results.

You are measuring the first 'Send'. That will incur some overhead that other Sends won't. So, always do warm up first like send single event and then measure the next one.
Another important thing. It is not right to measure just single 'Send' call. Measure bunch of calls instead and calculate latency percentile. That should provide a better figure for your tests.

NServiceBus: Timeout gets handled by multiple Sagas

We currently have a NServiceBus 5 system, which contains two recurring Sagas. Since they act as dispatcher to periodically pull multiple sorts of data from an external system, we're using the Timeouts to trigger this: We created a generic and empty class called ExecuteTask, which is used by the Saga to handle the timeout.
public class ScheduleSaga1 : Saga<SchedulerSagaData>,
IAmStartedByMessages<StartScheduleSaga1>,
IHandleMessages<StopSchedulingSaga>,
IHandleTimeouts<ExecuteTask>
And the other Saga is almost identically defined:
public class ScheduleSaga2: Saga<SchedulerSagaData>,
IAmStartedByMessages<StartScheduleSaga2>,
IHandleMessages<StopSchedulingSaga>,
IHandleTimeouts<ExecuteTask>
The timeout is handled equally in both Sagas:
public void Handle(StartScheduleSaga1 message)
{
if (_schedulingService.IsDisabled())
{
_logger.Info($"Task '{message.TaskName}' is disabled!");
}
else
{
Debugger.DoDebug($"Scheduling '{message.TaskName}' started!");
Data.TaskName = message.TaskName;
// Check to avoid that if the saga is already started, don't initiate any more tasks
// as those timeout messages will arrive when the specified time is up.
if (!Data.IsTaskAlreadyScheduled)
{
// Setup a timeout for the specified interval for the task to be executed.
Data.IsTaskAlreadyScheduled = true;
// Send the first Message Immediately!
SendMessage();
// Set the timeout
var timeout = _schedulingService.GetTimeout();
RequestTimeout<ExecuteTask>(timeout);
}
}
}
public void Timeout(ExecuteTask state)
{
if (_schedulingService.IsDisabled())
{
_logger.Info($"Task '{Data.TaskName}' is disabled!");
}
else
{
SendMessage();
// Action that gets executed when the specified time is up
var timeout = _schedulingService.GetTimeout();
Debugger.DoDebug($"Request timeout for Task '{Data.TaskName}' set to {timeout}!");
RequestTimeout<ExecuteTask>(timeout);
}
}
private void SendMessage()
{
// Send the Message to the bus so that the handler can handle it
Bus.Send(EndpointConfig.EndpointName, Activator.CreateInstance(typeof(PullData1Request)));
}
Now the problem: Since both Sagas are requesting Timeouts for ExecuteTask, it gets dispatched to both Sagas!
Even worse, it seems like the stateful Data in the Sagas gets messed up, since both Sagas are sending both message.
Therefore, it seems like the Timeouts are getting sent to all the Saga Instances which are requesting it.
But looking at the example https://docs.particular.net/samples/saga/simple/ there is no special logic regarding multiple Saga instances and their state.
Is my assumption correct? If this is the case, what are the best practices to have multiple Sagas requesting and receiving Timeouts?

The only reason I can think of when this is happening is that they share the same identifier to uniquely identify the saga instance.
Both ScheduleSaga1 and ScheduleSaga2 are using the same SchedulerSagaData for storing state. NServiceBus sees an incoming message and tries to retrieve the state, based on the unique identifier in the incoming message. If both StartScheduleSaga1 and StartScheduleSaga2 come in with identifier 1 for example, NServiceBus will search for saga state in the table SchedulerSagaData with unique identifier 1.
Both ScheduleSaga1 and ScheduleSaga2 will then share the same row!!!
Timeouts are based on SagaId in the TimeoutEntity table. Because both sagas share the same SagaId, it's logical they are both executed once the timeout arrives.
At the minimum you should not reuse the identifier to schedule tasks. It's probably better to not share the same class for storing saga state. Also easier to debug.

Speeding up http web request in WCF invoked method

Hi I am writing a WCF service not hosted on IIS. It runs on my server as console application.
I have a static method that is invoked by the mentioned service.
In this method I have async web request that sends sms.
Sometimes it happens that sms is never recieved. After some debuging I found out that when I remove async call the web request sometimes throws an exception with message: 'The operation has timed out'. This happens when i try to send many smss in short period of time.
But when i type the address of the web request in browser everything works fine. (the times i press refresh, no matter how fast, that number of times i receive sms) how can this be achived with what i've got.
So far i have
public static bool DoTheRequest(string number, string message)
{
try
{
HttpWebRequest myReq = (HttpWebRequest)WebRequest.Create(string.Format("http://SomeURL?With={0}&something={1}", number, message));
myReq.BeginGetResponse(FinishWebRequest, myReq);
}
catch (Exception ex)
{
throw ex;
}
return true;
}
static void FinishWebRequest(IAsyncResult result)
{
HttpWebResponse response = (result.AsyncState as HttpWebRequest).EndGetResponse(result) as HttpWebResponse;
}
EDIT:
And the service definition:
[OperationContract]
void TestSms(string number);
and implementation:
public void TestSms(string number)
{
Utilities.DoTheRequest(number, "THIS IS A TEST");
}
Please help

Depending on how many times you are calling the send-sms function in a short period of time, I surmise that it's doing it a lot faster than you can refresh in your browser - and you're managing to flood whatever service it is you are using with your SMS calls.
In this scenario, I would suggest that your web-service method actually puts the SMS in to a queue, and some kind of a background worker thread that is designed not to throttle the API that actually sends the SMS, has the job of batching up these SMSs and sending them.
Better yet, to facilitate the issue that machines and software are not perfect, and lightning does indeed strike, I would suggest that you push SMSs in to some kind of back-end data store and leave it at that. A background worker job/thread then has the job of finding all SMSs that are unsent and trying to send them (but sending no more than "x per minute"), each one being marked as "Sent" upon completion.
This has the added advantage that you can throw as many SMSs as you want at your backing data storage, if the machine dies it can pick up where it left off, and if you get some failures, they can be retried.

How do I prevent Socket/Port Exhaustion?

I am attempting to performance test a website by hitting it with requests across multiple threads. Each thread executes n times. (in a for loop)
However, I am running into problems. Specifically the WebException ("Unable to connect to remote server") with the inner exception:
An operation on a socket could not be performed because the system
lacked sufficient buffer space or because a queue was full
127.0.0.1:52395
I am attempting to run 100 threads at 500 iterations per thread.
Initially I was using HttpWebRequest in System.Net to make the GET request to the server. Currently I am using WebClient as I assumed that each iteration was using a new socket (so 100 * 500 sockets in a short period of time). I assumed WebClient (which is instantiated once per thread) would only use one socket.
I don't need 50 000 sockets open at once, as I would like to send the GET request, receive the response, and close the socket, freeing it for use in the next loop iteration. I understand that it would be a problem to
However, even with WebClient, a bunch of sockets are being requested resulting in a bunch of sockets in TIME_WAIT mode (checked using netstat). This causes other applications (like internet browsers) to hang and stop functioning.
I can operate my test with less iterations and/or less threads, as it appears the sockets do eventually exit this TIME_WAIT state. However, this is not a solution as it doesn't adequately test the abilities of the web server.
Question:
How do I explicitly close a socket (from the client side) after each thread iteration in order to prevent TIME_WAIT states and socket exhaustion?
Code:
Class that wraps the HttpRequest
Edit: Wrapped WebClient in a using, so a new one is instantiated,used and disposed for every iteration. The problem still persists.
public sealed class HttpGetTest : ITest {
private readonly string m_url;
public HttpGetTest( string url ) {
m_url = url;
}
void ITest.Execute() {
using (WebClient webClient = new WebClient()){
using( Stream stream = webClient.OpenRead( m_url ) ) {
}
}
}
}
The part of my ThreadWrapperClass that creates a new thread:
public void Execute() {
Action Hammer = () => {
for( int i = 1; i <= m_iterations; i++ ) {
//Where m_test is an ITest injected through constructor
m_test.Execute();
}
};
ThreadStart work = delegate {
Hammer();
};
Thread thread = new Thread( work );
thread.Start();
}

Do you understand the purpose of TIME_WAIT? It's a period during which it would be unsafe to reuse the port because lost packets (that have been successfully retransmitted) from the previous transaction might yet be delivered within that time period.
You could probably tweak it down in the registry somewhere, but I question if this is a sensible next step.
My experience of creating realistic load in a test environment have proved very frustrating. Certainly running your load-tester from localhost is by no means realistic, and most network tests I have made using the .net http apis seem to require more grunt in the client than the server itself.
As such, it's better to move to a second machine for generating load on your server... however domestic routing equipment is rarely up to the job of supporting anywhere near the number of connections that would cause any sort of load on a well written server app, so now you need to upgrade your routing/switching equipment as well!
Lastly, I've had some really strange and unexpected performance issues around the .net Http client API. At the end of the day, they all use HttpWebRequest to do the heavy lifting. IMO it's nowhere near as performant as it could be. DNS is sychronous, even when calling the APIs asynchronously (although if you're only requesting from a single host, this isn't an issue), and after sustained usage CPU usage creeps up until the client becomes CPU constrained rather than IO constrained. If you're looking to generate sustained and heavy load, any request-heavy app reliant on HttpWebRequest is IMO a bogus investment.
All in all, a pretty tricky job, and ultimately, something that can only be proved in the wild, unless you've got plently of cash to spend on an armada of better equipment.
[Hint: I got much better perfomance from my own client written using async Socket apis and a 3rd party DNS client library]

Q: How do I explicitly close a socket ... in order to prevent
TIME_WAIT states?
A: Dude, TIME_WAIT is an integral - and important! - part of TCP/IP itself!
You can tune the OS to reduce TIME_WAIT (which can have negative repercussions).
And you can tune the OS to increase #/ephemeral ports:
http://msdn.microsoft.com/en-us/library/aa560610%28v=bts.20%29.aspx
Here's a link on why TIME_WAIT exists ... and why it's a Good Thing:
http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html

It's not an issue of closing sockets or releasing resources in your app. The TIME _WAIT is a TCP stack timeot on released sockets to prevent their re-use until such time as it is virtually impossible for any packets 'left over' from a previous connection to that socket to not have expired.
For test purposes, you can reduce the wait time from the default, (some minutes, AFAIK), to a smaller value. When load-testing servers, I set it at six seconds.
It's in the registry somewhere - you'll find it if you Google.
Found it:
Change TIME_WAIT delay

It looks like you are not forcing your WebClient to get rid of the resources that it has allocated. You are performing a Using on the stream that is returned, but your WebClient still has resources.
Either wrap your WebClient instantiation in a using block, or manually call dispose on it once you are done reading from the URL.
Try this:
public sealed class HttpGetTest : ITest {
private readonly string m_url;
public HttpGetTest( string url ) {
m_url = url;
}
public void ITest.Execute() {
using( var m_webClient = new WebClient())
{
using( Stream stream = m_webClient.OpenRead( m_url ) )
{
}
}
}
}

You don't need to mess around with TIME_WAIT to accomplish what you want.
The problem is that you are disposing the WebClient every time you call Execute(). When you do that, you close the socket connection with the server and the TCP port keeps busy for the TIME_WAIT period.
A better approach is to create the WebClient in the constructor of your HttpGetTest class and reuse the same object throughout the test.
WebClient uses keep alive by default and will reuse the same connection for all its requests so in your case there will be only 100 opened connections for this.