Develop Reference

Azure Service Bus Queue Trigger function is called more than once when deployed

I have two Azure Functions. One is HTTP triggered, let's call it the API and the other one ServiceBusQueue triggered, and let's call this one the Listener.
The first one (the API) puts an HTTP request into a queue and the second one (the Listener) picks that up and processes that. The functions SDK version is: 3.0.7.
I have two projects in my solution for this. One which contains the Azure Functions and the other one which has the services. The API once triggered, calls a service from the other project that puts the message into the queue. And the Listener once received a message, calls a service from the service project to process the message.
Any long-running process?
The Listener actually performs a lightweight workflow and it all happens very quickly considering the amount of work it executes. The average time of execution is 90 seconds.
What's the queue specs?
The queue that the Listener listens to and is hosted in an Azure ServiceBus namespace has the following properties set:
Max Delivery Count: 1
Message time to live: 1 day
Auto-delete: Never
Duplicate detection window: 10 min
Message lock duration: 5 min
And here a screenshot for it:
The API puts the HTTP request into the queue using the following method:
public async Task ProduceAsync(string queueName, string jsonMessage)
{
jsonMessage.NotNull();
queueName.NotNull();
IQueueClient client = new QueueClient(Environment.GetEnvironmentVariable("ServiceBusConnectionString"), queueName, ReceiveMode.PeekLock)
{
OperationTimeout = TimeSpan.FromMinutes(5)
};
await client.SendAsync(new Message(Encoding.UTF8.GetBytes(jsonMessage)));
if (!client.IsClosedOrClosing)
{
await client.CloseAsync();
}
}
And the Listener (the service bus queue triggered azure function), has the following code to process the message:
[FunctionName(nameof(UpdateBookingCalendarListenerFunction))]
public async Task Run([ServiceBusTrigger(ServiceBusConstants.UpdateBookingQueue, Connection = ServiceBusConstants.ConnectionStringKey)] string message)
{
var data = JsonConvert.DeserializeObject<UpdateBookingCalendarRequest>(message);
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarListenerFunction)} picked up a message at {DateTime.Now}. Data: {data}");
await _workflowHandler.HandleAsync(data);
}
The Problem
The Listener function processes the same message 3 times! And I have no idea why! I've Googled and read through a few of StackOverFlow threads such as this one. And it looks like that everybody advising to ensure lock duration is long enough for the process to get executed completely. Although, I've put in 5 minutes for the lock, yet, the problem keeps coming. I'd really appreciate any help on this.

Just adding this in here so might be helpful for some others.
After some more investigations I've realized that in my particular case, the issue was regardless of the Azure Functions and Service Bus. In my workflow handler that the UpdateBookingCalendarListenerFunction sends messages to, I was trying to call some external APIs in a parallel approach, but, for some unknown reasons (to me) the handler code was calling off the external APIs one additional time, regardless of how many records it iterates over. The below code shows how I had implemented the parallel API calls and the other code shows how I've done it one by one that eventually led to a resolution for the issue I had.
My original code - calling APIs in parallel
public async Task<IEnumerable<StaffMemberGraphApiResponse>> AddAdminsAsync(IEnumerable<UpdateStaffMember> admins, string bookingId)
{
var apiResults = new List<StaffMemberGraphApiResponse>();
var adminsToAdd = admins.Where(ad => ad.Action == "add");
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarWorkflowDetailHandler)} Recognized {adminsToAdd.Count()} admins to add to booking with id: {bookingId}");
var addAdminsTasks = adminsToAdd.Select(admin => _addStaffGraphApiHandler.HandleAsync(new AddStaffToBookingGraphApiRequest
{
BookingId = bookingId,
DisplayName = admin.DisplayName,
EmailAddress = admin.EmailAddress,
Role = StaffMemberAllowedRoles.Admin
}));
if (addAdminsTasks.Any())
{
var addAdminsTasksResults = await Task.WhenAll(addAdminsTasks);
apiResults = _populateUpdateStaffMemberResponse.Populate(addAdminsTasksResults, StaffMemberAllowedRoles.Admin).ToList();
}
return apiResults;
}
And my new code without aggregating the API calls into the addAdminsTasks object and hence with no await Task.WhenAll(addAdminsTasks):
public async Task<IEnumerable<StaffMemberGraphApiResponse>> AddStaffMembersAsync(IEnumerable<UpdateStaffMember> members, string bookingId, string targetRole)
{
var apiResults = new List<StaffMemberGraphApiResponse>();
foreach (var item in members.Where(v => v.Action == "add"))
{
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarWorkflowDetailHandler)} Adding {targetRole} to booking: {bookingId}. data: {JsonConvert.SerializeObject(item)}");
apiResults.Add(_populateUpdateStaffMemberResponse.PopulateAsSingleItem(await _addStaffGraphApiHandler.HandleAsync(new AddStaffToBookingGraphApiRequest
{
BookingId = bookingId,
DisplayName = item.DisplayName,
EmailAddress = item.EmailAddress,
Role = targetRole
}), targetRole));
}
return apiResults;
}
I've investigated the first approach and the numbers of tasks were exact match of the number of the IEnumerable input, yet, the API was called one additional time. And within the _addStaffGraphApiHandler.HandleAsync, there is literally nothing than an HttpClient object that raises a POSTrequest. Anyway, using the second code has resolved the issue.

Task stops working on different environment

I'm having hard time with this one.
So in my asp.net application there is such a method:
public CopyResponse thirdStage(CopyRequest request)
{
CopyCCResponse response = new CopyCCResponse();
Task.Run(() =>
{
performCopying(request);
});
return response;
}
private void performCopying(CopyCCRequest request)
{
using (Repository = new myDbContext())
{
// do some initial action
try
{
// in general it looks like below
foreach(var children in father)
{
var newChildren = chldren.Copy();
Repository.Childrens.Add(newChildren);
foreach (var grandchldren in children.grandchildrens)
{
var newGrandchildren = grandchldren.Copy();
newGrandchildren.Parent = newChildren;
Repository.Grandchildrens.Add(newGrandchildren);
}
Repository.SaveChanges();
}
}
catch (Exception ex)
{
// log that action failed
throw ex;
}
}
}
This method and all other (there are some similar) works as designed on my local computer without any problems.
Unfortunately, on another environment those methods fail:
Copying smaller parts of data works fine. But when there is over 3000 objects to operate on, method fails.
Main application is responding correctly nevertheless.
Most of the operation is done well (most data is copied and saved in database)
Application doesn't enter catch block. Instructions for failed copying are not executed. Exception isn't caught by the error handler (BTW, I know by default the app can't catch exceptions from independent task, I wrote my handler so it will manage to do so).
IIS worker process seems to take over 300MB and 0% of processor power after copying stopped. More than half of RAM on server is still free.
I looked into windows event log, but haven't found anything.
Do you have any suggestions how I can handle this issue?

You can't do reliable "Fire and forget" tasks from inside IIS, if the site is not being served the application pool will get its AppDomain shut down after a while.
Two options to use are:
HostingEnvironment.QueueBackgroundWorkItem to tell IIS you are doing background work. This will let the server know of the work and it will delay the shutdown as long as it can (default up to 90 seconds max) before it kills your process.
public CopyResponse thirdStage(CopyRequest request)
{
CopyCCResponse response = new CopyCCResponse();
HostingEnvironment.QueueBackgroundWorkItem(() =>
{
performCopying(request);
});
return response;
}
Another option is to use a 3rd party library that is designed for doing background work in IIS like Hangfire.io, this will run a service inside of IIS that does the work and attempts to keep the instance alive till the work is done. You can also configure Hangfire to run as a separate process so you don't need to rely on the lifetime of the IIS instance.
public CopyResponse thirdStage(CopyRequest request)
{
CopyCCResponse response = new CopyCCResponse();
BackgroundJob.Enqueue(() =>
{
performCopying(request);
});
return response;
}
Note, using hangfire with a seperate process may require you to do a little redesign of performCopying(CopyCCRequest request) to support being run from a separate process, using it from inside the IIS instance should not require any changes.

Web API Service - How to make the requests at the server to be executed concurrently

I am using a WebApi rest service controller, hosted by IIS 7.5,
as i understood from this post:
Are all the web requests executed in parallel and handled asynchronously?
A webApi service, by default, executes all its incoming requests in parallel, but only if the current multiple requests (at a certain time) came from different sessions.
That is to say, if single client will send some simultaneously requests to server, all of them will be executed sequentially and won't be executed concurrently.
This behavior is a real problem for us, because in some cases, our client sends bunch of requests from different client's listeners, asynchronously (by browser), and all of them will actually be queued instead of being executed concurrently at the server. Therefore, in some cases, we experiencing a serious performance issues which are really noticeable at the client's web page.
How can we solve this problem?
I understand we can maybe disable session state but that isn't a normal thing to do.

Actually, disabling session state is the normal solution for web APIs. If you need it for some/all of your calls, you can call HttpContext.SetSessionStateBehavior (e.g., from Application_BeginRequest). Multiple read-only session state requests can run concurrently.

Do you try async Task ? Here is sample Controller:
public class SendJobController : ApiController
{
public async Task<ResponseEntity<SendJobResponse>> Post([FromBody] SendJobRequest request)
{
return await PostAsync(request);
}
private async Task<ResponseEntity<SendJobResponse>> PostAsync(SendJobRequest request)
{
Task<ResponseEntity<SendJobResponse>> t = new Task<ResponseEntity<SendJobResponse>>(() =>
{
ResponseEntity<SendJobResponse> _response = new ResponseEntity<SendJobResponse>();
try
{
//
// some long process
//
_response.responseStatus = "OK";
_response.responseMessage = "Success";
_response.responseObject = new SendJobResponse() { JobId = 1 };
}
catch (Exception ex)
{
_response.responseStatus = "ERROR";
_response.responseMessage = ex.Message;
}
return _response;
});
t.Start();
return await t;
}
}

HttpContent.ReadAsStringAsync causes request to hang (or other strange behaviours)

We are building a highly concurrent web application, and recently we have started using asynchronous programming extensively (using TPL and async/await).
We have a distributed environment, in which apps communicate with each other through REST APIs (built on top of ASP.NET Web API). In one specific app, we have a DelegatingHandler that after calling base.SendAsync (i.e., after calculating the response) logs the response to a file. We include the response's basic information in the log (status code, headers and content):
public static string SerializeResponse(HttpResponseMessage response)
{
var builder = new StringBuilder();
var content = ReadContentAsString(response.Content);
builder.AppendFormat("HTTP/{0} {1:d} {1}", response.Version.ToString(2), response.StatusCode);
builder.AppendLine();
builder.Append(response.Headers);
if (!string.IsNullOrWhiteSpace(content))
{
builder.Append(response.Content.Headers);
builder.AppendLine();
builder.AppendLine(Beautified(content));
}
return builder.ToString();
}
private static string ReadContentAsString(HttpContent content)
{
return content == null ? null : content.ReadAsStringAsync().Result;
}
The problem is this: when the code reaches content.ReadAsStringAsync().Result under heavy server load, the request sometimes hangs on IIS. When it does, it sometimes returns a response -- but hangs on IIS as if it didn't -- or in other times it never returns.
I have also tried reading the content using ReadAsByteArrayAsync and then converting it to String, with no luck.
When I convert the code to use async throughout I get even weirder results:
public static async Task<string> SerializeResponseAsync(HttpResponseMessage response)
{
var builder = new StringBuilder();
var content = await ReadContentAsStringAsync(response.Content);
builder.AppendFormat("HTTP/{0} {1:d} {1}", response.Version.ToString(2), response.StatusCode);
builder.AppendLine();
builder.Append(response.Headers);
if (!string.IsNullOrWhiteSpace(content))
{
builder.Append(response.Content.Headers);
builder.AppendLine();
builder.AppendLine(Beautified(content));
}
return builder.ToString();
}
private static Task<string> ReadContentAsStringAsync(HttpContent content)
{
return content == null ? Task.FromResult<string>(null) : content.ReadAsStringAsync();
}
Now HttpContext.Current is null after the call to content.ReadAsStringAsync(), and it keeps being null for all the subsequent requests! I know this sounds unbelievable -- and it took me some time and the presence of three coworkers to accept that this was really happening.
Is this some kind of expected behavior? Am I doing something wrong here?

I had this problem. Although, I haven't fully tested yet, using CopyToAsync instead of ReadAsStringAsync seems to fix the problem:
var ms = new MemoryStream();
await response.Content.CopyToAsync(ms);
ms.Seek(0, SeekOrigin.Begin);
var sr = new StreamReader(ms);
responseContent = sr.ReadToEnd();

With regards to your second issue, the async/await is syntactic sugar for the compiler building a state machine where the call to to a function preceded by "await" returns immediately on the current thread...one that contains HttpContext.Current in its thread local storage. The completion of that async call can occur on a different thread...one that does NOT have HttpContext.Current in its thread local storage.
If you want the completion to execute on the same thread (thus having the same objects in thread local storage like HttpContext.Current), then you need to be aware of this behavior. This is especially important on calls from the main UI thread (if you're building a Windows application) or in ASP.NET, calls from an ASP.NET request thread where you are dependent on HttpContext.Current.
See reference docs on ConfigureAwait(false). Also, view some Channel 9 tutorials on TPL. Once the "easy" stuff is grokked, the presenter will invariably talk about this issue as it causes subtle problems that are not easily understood unless you know what the TPL is doing underneath the covers.
Good luck.
With regards to your first problem, if the caller gets a result, I'm not convinced that IIS has not completed the request. How are you determining that the ASP.NET request thread initiated by this caller is hung in IIS?

Using MVC3 with Async tasks to update the UI

What I have is an AJAX form on a View that makes a call to the server. This call perform n number of tasks where n is a number decided by records in a database (typically no more than 10 records). Each record corresponds to a Build Definition in TFS so what I am trying to do is get all of these Build Definitions, queue them in TFS, and as each build completes update the UI so that user knows which builds have completed.
Unfortunately I am not sure about how best to do this. I was thinking something along these lines:
foreach (var dep in builds)
{
TFS tfsServer = new TFS(TFS_SERVER_ADDRESS);
IBuildServer buildServer;
int id = tfsServer.QueuBuild(dep.TeamProject, dep.BuildDefinition);
string teamProject = dep.TeamProject;
Task.Factory.StartNew(() => GetBuildStatus(teamProject, id, tfsServer));
}
The task that is called is:
private void GetBuildStatus(string TeamProject, int BuildID, TFS Server)
{
Server.GetBuildStatus(TeamProject, BuildID);
AsyncManager.OutstandingOperations.Decrement();
}
The problem here is that my Completed method isn't going to get called until all of the builds have completed. How would I go about feeding data back up to the UI a piece at a time?
It is also worth mentioning that the GetBuildStatus method looks like this:
do
{
var build = buildsView.QueuedBuilds.FirstOrDefault(x => x.Id == BuildID);
if(build != null)
{
status = build.Status;
detail = build.Build;
}
} while (status != QueueStatus.Completed);
return detail.Status.ToString();

Given that the duration of a build will be longer than the timeout for an HTTP request you cannot leave the browser waiting while this happens. You need to return a page and then poll for updates from that page using AJAX. Typically you'd have a timer in javascript that triggers a regular call back to the server to get the updated status information.
But, since you are using .NET you could also consider trying SignalR which lets you use long polling, server sent events or web sockets to wait for updates from the server and it wraps it all up in some easy to implement .NET classes and Javascript.