I'm using the Twilio C# API to send batches of messages to the Twilio API (typically small amounts, but can grow to several thousand at a time), and in an effort to make the process faster, I started switching everything over to asynchronous calls (as well as some parallel foreaches).
The Twilio API works most of the time when it's completely synchronous, but when I change the request to be asynchronous, I start to get heaps of errors. They all essentially say:
Connection Error: POST and the internal error is A task was canceled.
After doing some digging, I've determined that the Twilio API is built on top of the HttpClient, and it uses the client to send all its requests. However, I can't figure out how to make it all work properly. On average, when sending batches with over 3,000 messages, roughly 20% of them fail.
My sending code is pretty basic
// async SendTextNotificationsAsync
try
{
var result = MessageResource.Create(
to,
from: new PhoneNumber(PhoneNumber),
body: message);
return result;
}
catch (ApiException ex) // API exception, probably an invalid number.
{
LogTwilioBadNumber(ex, phone);
return null;
}
catch (ApiConnectionException ex) // Connection error, let's reattempt
{
if (tryAttempt < 3)
{
System.Threading.Thread.Sleep(500);
// failed call again
return await SendTextNotificationsAsync(phone, message, tryAttempt + 1);
}
if (tryAttempt >= 3)
{
LogTwilioConnectionException(ex, phone, patientId, tryAttempt);
}
return null;
}
The above is called from another method that looks something like:
// async void CompleteCampaignsAsync
await Task.Run(() => Parallel.ForEach(campaign.Details, async n =>
{
await com.SendTextNotificationsAsync(to, message);
}));
And finally, the controller simply calls something like so:
// Don't wait for this to finish, internal processes handle logging
// Will get a time-out error if we wait for this.
await Task.Run(() => cl.CompleteCampaignsAsync()).ConfigureAwait(false);
There are several other steps and layers, so the above is massively simplified. Everything is asynchronous and everything is awaited. The only real exception is the main controller due to it being a long running process that needs to just get kicked off.
In the first method, I've tried both MessageResource.Create and MessageResource.CreateAsync. In the latter, I get through about 80 messages before its an unending stream of task cancelled errors. In the former, the process runs to completion, but roughly 20%-50% of the tasks fail.
Is there anything immediately obviously wrong that would allow me to correct this?
Related
I have a web app that connects to an external API.
That API has a limit of 3 connections per second.
I have a method that gets employee data for a whole factory.
It works fine, but I've found that if a particular factory has a lot of employees, I hit the API connection limit and get an error.
(429) API calls exceeded...maximum 3 per Second
So I decided to use await Task.Delay(1000) to set a 1 second delay, every time this method is used.
Now it seems to have reduced the number of errors I get, but I am still getting a few limit errors.
Is there another method I could use to ensure my limit is not reached?
Here is my code:
public async Task<YourSessionResponder> GetAll(Guid factoryId)
{
UserSession.AuthData sessionManager = new UserSession.AuthData
{
UserName = "xxxx",
Password = "xxxx"
};
ISessionHandler sessionMgr = new APIclient();
YourSessionResponder response;
response = await sessionMgr.GetDataAsync(sessionManager, new ListerRequest
{
FactoryId = factoryId;
await Task.Delay(1000);
return response;
}
I call it like this:
var yourEmployees = GetAll(factoryId);
I have a web app that connects to an external API.
Your current code limits the number of outgoing requests made by a single incoming request to your API. What you need to do is limit all of your outgoing requests, app-wide.
It's possible to do this using a SemaphoreSlim:
private static readonly SemaphoreSlim Mutex = new(1);
public async Task<YourSessionResponder> GetAll(Guid factoryId)
{
...
YourSessionResponder response;
await Mutex.WaitAsync();
try
{
response = await sessionMgr.GetDataAsync(...);
await Task.Delay(1000);
}
finally
{
Mutex.Release();
}
return response;
}
But I would take a different approach...
Is there another method I could use to ensure my limit is not reached?
Generally, I recommend just retrying on 429 errors, using de-correlated jittered exponential backoff (see Polly for an easy implementation). That way, when you're "under budget" for the time period, your requests go through immediately, and they only slow down when you hit your API limit.
From a comment on the question:
I am calling it like this: var yourEmployees = GetAll(factoryId);
Then you're not awaiting the task. While there's a 1-second delay after each network operation, you're still firing off all of the network operations in rapid succession. You need to await the task before moving on to the next one:
var yourEmployees = await GetAll(factoryId);
Assuming that this is happening in some kind of loop or repeated operation, of course. Otherwise, where would all of these different network tasks be coming from? Whatever high-level logic is invoking the multiple network operations, that logic needs to await one before moving on to the next.
I have a long running process which performs matches between millions of records I call this code using a Service Bus, However when my process passes the 5 minute limit Azure starts processing the already processed records from the start again.
How can I avoid this
Here is my code:
private static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
long receivedMessageTrasactionId = 0;
try
{
IQueueClient queueClient = new QueueClient(serviceBusConnectionString, serviceBusQueueName, ReceiveMode.PeekLock);
// Process the message
receivedMessageTrasactionId = Convert.ToInt64(Encoding.UTF8.GetString(message.Body));
// My Very Long Running Method
await DataCleanse.PerformDataCleanse(receivedMessageTrasactionId);
//Get Transaction and Metric details
await queueClient.CompleteAsync(message.SystemProperties.LockToken);
}
catch (Exception ex)
{
Log4NetErrorLogger(ex);
throw ex;
}
}
Messages are intended for notifications and not long running processing.
You've got a fewoptions:
Receive the message and rely on receiver's RenewLock() operation to extend the lock.
Use user-callback API and specify maximum processing time, if known, via MessageHandlerOptions.MaxAutoRenewDuration setting to auto-renew message's lock.
Record the processing started but do not complete the incoming message. Rather leverage message deferral feature, sending yourself a new delayed message with the reference to the deferred message SequenceNumber. This will allow you to periodically receive a "reminder" message to see if the work is finished. If it is, complete the deferred message by its SequenceNumber. Otherise, complete the "reminder" message along with sending a new one. This approach would require some level of your architecture redesign.
Similar to option 3, but offload processing to an external process that will report the status later. There are frameworks that can help you with that. MassTransit or NServiceBus. The latter has a sample you can download and play with.
Note that option 1 and 2 are not guaranteed as those are client-side initiated operations.
I have adapted the following code to attempt to load from a web api from the asp.net code by Wasson.
static public async Task<IEnumerable<T>> ExecuteAsync(HttpClient client, String endPoint)
{
IEnumerable<T> result = null;
HttpResponseMessage response = client.GetAsync(endPoint).Result;
response.EnsureSuccessStatusCode();
if (response.IsSuccessStatusCode)
{
var tResult = await response.Content.ReadAsStringAsync();
result = JsonConvert.DeserializeObject<IEnumerable<T>>(tResult);
}
return result;
}
As I have it designed, I think this procedure will run asynchronously, but I am not sure how to test it to make sure it does.
I am new to using asynchronous coding, so any suggestions would be helpful.
The calling procedure is as follows:
public virtual IEnumerable<T> Fill()
{
IEnumerable<T> result = null;
try
{
using (var client = CreateClient("", new MediaTypeWithQualityHeaderValue("application/json")))
{
String _endPoint = "api/" + typeof(T).Name + "/Get";
result = (ExecuteAsync(client, _endPoint)).Result;
}
}
catch (Exception ex)
{
LogError(ex.Message);
}
return result;
}
Is this combination likely to run asynchronously as I want it to?
The reason I want to be able to run it asynchronously is that the Fill routines fetch large amounts of data from a database. It is more a case of a large number of individual records rather than a large size for each record, although the size of some of the records may also be an issue.
What I don't want is the user sitting there waiting for the entire page to load while the communication with the database is occurring.
It doesn't really matter in this case whether I am directly connecting to the database or going through the web api I am using: It takes a long time in some cases either way.
I am also concerned about deadlocks: One large fetch coming in while another is pending.
I initially patterned the code after Mike Wasson's sample code, but found that the code hanged indefinitely even though it compiled.
Does this help frame the situation a little better?
Not sure what you're after here mate.
The code you posted will still block the thread it's on since you're calling Result which will block.
result = (ExecuteAsync(client, _endPoint)).Result;
You could make it asynchronous by letting the async and await keywords spread back to whatever method initially created the request.
public virtual async Task<IEnumerable<T>> Fill()
{
...
result = await ExecuteAsync(client, _endPoint);
...
}
However I'm not sure this really is what you're after since you mention
What I don't want is the user sitting there waiting for the entire page to load while the communication with the database is occurring.
By default, making things asynchronous will not solve this for you. An operation that takes 15 seconds to complete will take 15 seconds to complete regardless of it being synchronous or asynchronous. The difference is that if it's asynchronous then you're being more resource effective (meaning, you'll free up the thread to preform other duties while waiting for something to complete).
However, you can use tasks to achive parallelity. Take for example:
public void DoWork()
{
Thread.Sleep(10000);
}
public void DoMoreWork()
{
Thread.Sleep(10000);
}
If we called these two like this
DoWork();
DoMoreWork();
Then that would take 20000 ms or 20 seconds
And if we make it asynchronous
public async Task DoWork()
{
await Task.Delay(10000);
}
public async Task DoMoreWork()
{
await Task.Delay(10000);
}
and call it like this
await DoWork();
await DoMoreWork();
Then it will STILL take 20 seconds to complete. However, since DoWork and DoMoreWork can operate independent from eachothers we could run them both at the same time by doing
await Task.WhenAll(DoWork(), DoMoreWork());
This would (in a perfect world) result in both tasks being done in 10 seconds halving the time the user has to wait.
Hope I shed some light on the issue :)
I have an MVC controller method that does a number of things, but the last thing it does is this:
public void PerformCheckout(int salesAddressId)
{
try
{
...
...
// We just made a sale. Send emails to all market owners.
SendSalesEmailsAsync(master);
}
catch (System.Exception exception)
{
// Log the error
}
And then SendSalesEmailesAsynch() looks like this:
private async Task SendSalesEmailsAsync(SalesMaster salesMaster)
{
...
...
// Send an email to the owner of each marker
foreach(string market in markets)
await SendSalesEmailAsync(salesMaster, salesDetailList, market).ConfigureAwait(false);
}
SendSalesEmailAsynch() looks like this:
// Send an email to a market owner
private async Task SendSalesEmailAsync(SalesMaster salesMaster, List<SalesDetail> salesDetail, string marketName)
{
...
...
Email email = new Email(new List<string> { sellerEmailAddress }, emailSubject, emailHtmlBody);
await email.SendAsync().ConfigureAwait(false);
And finally, the method that actually sends the email:
public async Task SendAsync()
{
// Create a network credentials object
var credentials = new NetworkCredential(azureUserName, azurePassword);
// Create an Web transport for sending the email
var transportWeb = new Web(credentials);
// Send the email. We don't care about the current context so let's run this in a thread pool context.
// For more information: http://blog.stephencleary.com/2012/07/dont-block-on-async-code.html
await transportWeb.DeliverAsync(this._email).ConfigureAwait(false);
}
I am pretty new with Async and Await. Does all this look proper? Also, I'm not entirely sure that the catch block in the controller action method will get called if an error occurs while sending an email.
As you suspected, the catch block won't get any exceptions since you didn't synchronize with current context. In order to do that you have to mark your action method as async and use await for calling SendSalesEmailsAsync. So the action method will look like this:
public async Task PerformCheckout(int salesAddressId)
{
try
{
...
...
// We just made a sale. Send emails to all market owners.
await SendSalesEmailsAsync(master);
}
catch (System.Exception exception)
{
// Log the error
}
This is not the typical approach used for async await. I would not feel comfortable saying "incorrect" because design goals are so implementation dependent.
There are several issues that show up in the code shown. Having a method which takes an async action and returns void is probably a bad idea for a few reasons.
Controllers shouldn't really be exposing functionality that doesn't include returning information. If it is an action to perform, it should be called as some sort of service or business logic. Furthermore, since the method is public and part of the controller anyone can invoke it from the url. As it is sending a group of emails, this may be undesirable.
Async/await should be used to free up processing power. For the most part, it should only be used when either the processing time or waiting time exceeds 40 milliseconds. As emails are fire and forget, this may not be the case in your situation in which case the entire method could be converted back to synchronous.
If it is taking longer than 40ms, then async is going to be okay to use. However, in the shown code the first call is not properly used. When an async call is made without await it is basically "fire and forget". Especially when there is no return value, the return is never noticed. As a result, if an exception is thrown it is also forgotten, and the catch block will never executed.
Using await causes a new task continuation to be created. The rest of the method is essentially forked while the call takes place. Once the called execution has finished, then the rest of the method which was forked executes. Without await, the rest of the method simply executes, and the execution of the side call probably finishes far later.
All in all, this design should probably be refactored to be called from a more controlled environment, and possibly even from a synchronous method.
Background: We have import-functions that can take anywhere from a few seconds to 1-2 hours to run depending on the file being imported. We want to expose a new way of triggering imports, via a REST request.
Ideally the REST service would be called, trigger the import and reply with a result when done. My question is: since it can take up to two hours to run, is it possible to reply or will the request timeout for the caller? Is there a better way for this kind of operation?
What I use in these cases is an asynchronous operation that returns no result (void function result in case of c# Web API), then send the result asynchronously using a message queue.
E.g.
[HttpPut]
[Route("update")]
public void Update()
{
var task = Task.Run(() => this.engine.Update());
task.ContinueWith(t => publish(t, "Update()"));
}