What I have is an AJAX form on a View that makes a call to the server. This call perform n number of tasks where n is a number decided by records in a database (typically no more than 10 records). Each record corresponds to a Build Definition in TFS so what I am trying to do is get all of these Build Definitions, queue them in TFS, and as each build completes update the UI so that user knows which builds have completed.
Unfortunately I am not sure about how best to do this. I was thinking something along these lines:
foreach (var dep in builds)
{
TFS tfsServer = new TFS(TFS_SERVER_ADDRESS);
IBuildServer buildServer;
int id = tfsServer.QueuBuild(dep.TeamProject, dep.BuildDefinition);
string teamProject = dep.TeamProject;
Task.Factory.StartNew(() => GetBuildStatus(teamProject, id, tfsServer));
}
The task that is called is:
private void GetBuildStatus(string TeamProject, int BuildID, TFS Server)
{
Server.GetBuildStatus(TeamProject, BuildID);
AsyncManager.OutstandingOperations.Decrement();
}
The problem here is that my Completed method isn't going to get called until all of the builds have completed. How would I go about feeding data back up to the UI a piece at a time?
It is also worth mentioning that the GetBuildStatus method looks like this:
do
{
var build = buildsView.QueuedBuilds.FirstOrDefault(x => x.Id == BuildID);
if(build != null)
{
status = build.Status;
detail = build.Build;
}
} while (status != QueueStatus.Completed);
return detail.Status.ToString();
Given that the duration of a build will be longer than the timeout for an HTTP request you cannot leave the browser waiting while this happens. You need to return a page and then poll for updates from that page using AJAX. Typically you'd have a timer in javascript that triggers a regular call back to the server to get the updated status information.
But, since you are using .NET you could also consider trying SignalR which lets you use long polling, server sent events or web sockets to wait for updates from the server and it wraps it all up in some easy to implement .NET classes and Javascript.
Related
I have two Azure Functions. One is HTTP triggered, let's call it the API and the other one ServiceBusQueue triggered, and let's call this one the Listener.
The first one (the API) puts an HTTP request into a queue and the second one (the Listener) picks that up and processes that. The functions SDK version is: 3.0.7.
I have two projects in my solution for this. One which contains the Azure Functions and the other one which has the services. The API once triggered, calls a service from the other project that puts the message into the queue. And the Listener once received a message, calls a service from the service project to process the message.
Any long-running process?
The Listener actually performs a lightweight workflow and it all happens very quickly considering the amount of work it executes. The average time of execution is 90 seconds.
What's the queue specs?
The queue that the Listener listens to and is hosted in an Azure ServiceBus namespace has the following properties set:
Max Delivery Count: 1
Message time to live: 1 day
Auto-delete: Never
Duplicate detection window: 10 min
Message lock duration: 5 min
And here a screenshot for it:
The API puts the HTTP request into the queue using the following method:
public async Task ProduceAsync(string queueName, string jsonMessage)
{
jsonMessage.NotNull();
queueName.NotNull();
IQueueClient client = new QueueClient(Environment.GetEnvironmentVariable("ServiceBusConnectionString"), queueName, ReceiveMode.PeekLock)
{
OperationTimeout = TimeSpan.FromMinutes(5)
};
await client.SendAsync(new Message(Encoding.UTF8.GetBytes(jsonMessage)));
if (!client.IsClosedOrClosing)
{
await client.CloseAsync();
}
}
And the Listener (the service bus queue triggered azure function), has the following code to process the message:
[FunctionName(nameof(UpdateBookingCalendarListenerFunction))]
public async Task Run([ServiceBusTrigger(ServiceBusConstants.UpdateBookingQueue, Connection = ServiceBusConstants.ConnectionStringKey)] string message)
{
var data = JsonConvert.DeserializeObject<UpdateBookingCalendarRequest>(message);
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarListenerFunction)} picked up a message at {DateTime.Now}. Data: {data}");
await _workflowHandler.HandleAsync(data);
}
The Problem
The Listener function processes the same message 3 times! And I have no idea why! I've Googled and read through a few of StackOverFlow threads such as this one. And it looks like that everybody advising to ensure lock duration is long enough for the process to get executed completely. Although, I've put in 5 minutes for the lock, yet, the problem keeps coming. I'd really appreciate any help on this.
Just adding this in here so might be helpful for some others.
After some more investigations I've realized that in my particular case, the issue was regardless of the Azure Functions and Service Bus. In my workflow handler that the UpdateBookingCalendarListenerFunction sends messages to, I was trying to call some external APIs in a parallel approach, but, for some unknown reasons (to me) the handler code was calling off the external APIs one additional time, regardless of how many records it iterates over. The below code shows how I had implemented the parallel API calls and the other code shows how I've done it one by one that eventually led to a resolution for the issue I had.
My original code - calling APIs in parallel
public async Task<IEnumerable<StaffMemberGraphApiResponse>> AddAdminsAsync(IEnumerable<UpdateStaffMember> admins, string bookingId)
{
var apiResults = new List<StaffMemberGraphApiResponse>();
var adminsToAdd = admins.Where(ad => ad.Action == "add");
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarWorkflowDetailHandler)} Recognized {adminsToAdd.Count()} admins to add to booking with id: {bookingId}");
var addAdminsTasks = adminsToAdd.Select(admin => _addStaffGraphApiHandler.HandleAsync(new AddStaffToBookingGraphApiRequest
{
BookingId = bookingId,
DisplayName = admin.DisplayName,
EmailAddress = admin.EmailAddress,
Role = StaffMemberAllowedRoles.Admin
}));
if (addAdminsTasks.Any())
{
var addAdminsTasksResults = await Task.WhenAll(addAdminsTasks);
apiResults = _populateUpdateStaffMemberResponse.Populate(addAdminsTasksResults, StaffMemberAllowedRoles.Admin).ToList();
}
return apiResults;
}
And my new code without aggregating the API calls into the addAdminsTasks object and hence with no await Task.WhenAll(addAdminsTasks):
public async Task<IEnumerable<StaffMemberGraphApiResponse>> AddStaffMembersAsync(IEnumerable<UpdateStaffMember> members, string bookingId, string targetRole)
{
var apiResults = new List<StaffMemberGraphApiResponse>();
foreach (var item in members.Where(v => v.Action == "add"))
{
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarWorkflowDetailHandler)} Adding {targetRole} to booking: {bookingId}. data: {JsonConvert.SerializeObject(item)}");
apiResults.Add(_populateUpdateStaffMemberResponse.PopulateAsSingleItem(await _addStaffGraphApiHandler.HandleAsync(new AddStaffToBookingGraphApiRequest
{
BookingId = bookingId,
DisplayName = item.DisplayName,
EmailAddress = item.EmailAddress,
Role = targetRole
}), targetRole));
}
return apiResults;
}
I've investigated the first approach and the numbers of tasks were exact match of the number of the IEnumerable input, yet, the API was called one additional time. And within the _addStaffGraphApiHandler.HandleAsync, there is literally nothing than an HttpClient object that raises a POSTrequest. Anyway, using the second code has resolved the issue.
This description at the moment is all theory, I don't have any code yet. I was hoping to bounce ideas off people.
I have a VueJS app, let's say a To Do app. It lists all of the things I need to do today. When I complete a To Do, I check a box in my Vue app and Axios fires of an Http Post to a .NET API end point. Let's say that API method has to do several things, like update several databases, execute a few stored procedures, etc. as in it can take a few seconds to complete. My Vue app gets a success response and I can then go on to check the next completed item.
If I have several things I've completed it could take several minutes of my day to check, wait, check and wait. Now I want to check several items or maybe even select all. I want to submit a list of items to the API, let them queue up and process in the background while I go about other business in the app. All the while, a panel in the app displays the items still being processed in the background. As each one completes in the API, a push notification occurs and the UI updates, removing the item from the list.
Does this sound doable? Would I have Vue listening for updates from the API? or would Vue periodically have to poll the API to see what it still has outstanding? What is the preferred way? The goal is to free up the user to keep working rather than watch paint dry.
#Connie, I can tell you from experience that it's really possible, with a few tweaks.
The first thing I'd do, is to add all the logic inside Vuex.
Making it really simple, the steps would be:
1. Create a vuex state called toDos, and I'd assume that each toDo would be an object containing a format such:
toDoModel = {
id: 1,
completed: false
}
API receives only 1 ID for updating
Create a vuex mutation for updating this toDos state:
updateToDo(state, toDoObject) {
const toDoObjectIndexOnState = state.toDos.findIndex(toDo =>
toDo.id == toDoObject.id)
//ToDo not found on state list
if (toDoObjectIndexOnState == -1) {
state.toDos.push(toDoObject)
return
}
state.toDos[toDoObjectIndexOnState] = toDoObject
}
Create a vuex action called updateToDoState, to perform the Axios call and update state:
updateToDoState({commit}, toDoId){
// Call API on Axios, assuming `data` as the key for returning the toDo with updated info
response = axios.post(ENDPOINT, toDoId)
.then({data: toDo} => {
if (!toDo) return
// Call mutation
commit('updateToDo', toDo)
})
Make the call on your main Vue Component to call the updateToDoState action on each checklist click to update toDo state
API can receive multiple IDs for updating
(you have two approaches:
- Have a mutation to change each toDo per time and the action would loop through them all
- Have the action to pass thewhole list and the mutation would take care of updating the store object for each returned Id
Here my examples assume the fist option
Create a vuex mutation for updating this toDos state:
updateToDo(state, toDoObject) {
const toDoObjectIndexOnState = state.toDos.findIndex(toDo =>
toDo.id == toDoObject.id)
//ToDo not found on state list
if (toDoObjectIndexOnState == -1) {
state.toDos.push(toDoObject)
return
}
state.toDos[toDoObjectIndexOnState] = toDoObject
}
Create a vuex action called updateToDosState, to perform the Axios call and update state:
updateToDosState({commit}, toDosIdsList){
// Call API on Axios, assuming `data` as the key for returning the toDo with updated info
response = axios.post(ENDPOINT, toDosIdsList)
.then({data: toDos} => {
if (!toDos) return
// Call mutation
toDos.forEach(toDo => commit('updateToDo', toDo))
})
Make the call on your main Vue Component to call the updateToDoState action for batch updating the toDos state
In case any part of this logic / code is not 100% clear, just let me know and I can update here!
I'm trying to find a solution to a send email action that may take a long time and time out our load balancer which is on Rackspace. The only question I could find that relates to this specific issue is this:
keep load balancer from timing out during long operation
As I understand it I need to run another action whilst the main slow action is completing to constantly poll and return in order to keep things alive. My email action contains the following:
var sendto = db.Users
.Where(b => b.Id == model.SentTo |
((model.SelectedRoles.Any(s => b.Roles.Select(h => h.RoleId).Contains(s)))
&& ((b.enrollment.Any(h => h.cohort.CourseID == model.CourseID) | model.CourseID == null))
&& (b.OrgID == model.OrgID | model.OrgID == null))).ToList();
foreach (var address in sendto)
{
string Body = "message goes here";
EmailConfig.SendMessageViaMailGun(filestoattach, address.Email, null, email.Subject, Body);
}
So a list is created and then looped through with emails being sent to each person on the list. The Async method answer in the question above seems like it would do the trick but in the comments I can see this is considered a bad idea. It's also out of date in terms of how async works in the latest MVC version.
My question is what is the best way to keep this action from timing out the load balancer whilst it is completing?
This has nothing to do with async really, it is an infrastructure issue.
There are two ways to perform long operations:
The proper way: have a backend server and a process running there + communicate to this backend process via queuing (or database polling), then the client updates based on the progress (stored in some database) and update the UI on the web server. You also need to track the progress on the backend to continue on case of unexpected shutdown.
The cheap way: Spin a different thread (or task) on the web server, and have it perform the operation, and poll from javascript the progress of this thread. This could however get shut down any minute (webserver recycle) and you lose the operation (if you are ok with this), then you need to pick up the operation and continue. A crude way would be to just wrap the whole thing you have with Task.Run, and return right away, then query the progress from Javascript, but as I said above this is prone to interruptions.
I'm newbie with SignalR and want to learn so much. i already read beginner documents. But in this case i've stucked. what i want to do is when a user got new message i want to fire a script, like alert or showing div like "you have new mail" for notify the recieved user. And my question is how can i do that ? is there anyone know how to achieve this ? or good "step-by-step" document? i really want to work with SignalR.
ps: i'm using Visual Studio 2012 and MsSQL server
edit: i forgot to write, notification must be fired when message created to DB
Thank you
In your Scripts use the following, naturally this is not all the code, but enough based off tutorials to get you going. Your userId will be generated server side, and somehow your script can get it off an element of the page, or whatever method you want. It runs when the connection is started and then every 10 seconds. Pinging our server side method of CheckMessage() .
This js would need refactoring but should give you the general idea.
...
var messageHub = $.connection.messageHub;
var userId = 4;
$.connection.hub.start().done(function () {
StartCheck();
}
//Runs every 10 seconds..
function StartCheck()
{
setInterval(messageHub.server.checkMessage(userId,$.connection.hub.id), 10000);
}
This method takes in a userId, assuming your db is set up that way, and grabs them all from your database; naturally the method used is probably not appropriate for your system, however change it as you need to. It also checks if the user has any messages, and if so sends down another message to our SignalR scripts.
public void CheckMessage(int userId,int connectionId)
{
var user = userRepo.RetrieveAllUsers.FirstOrDefault(u=>u.id == userId);
if(user.HasMessages)
{
Clients.Group(connectionId).DisplayMailPopUp();
}
}
Finally this message, upon being called would run your code to do the 'You have Mail alert' - be it a popup, a div being faded in or whatever.
...
messageHub.client.displayMailPopUp = function () {
alert("You have Mail!");
};
...
Hopefully this helps - I recommend the following links for reading up and building your first SignalR app:
http://www.asp.net/signalr/overview/signalr-20/getting-started-with-signalr-20/tutorial-getting-started-with-signalr-20-and-mvc-5
And a smaller sample: http://code.msdn.microsoft.com/SignalR-Getting-Started-b9d18aa9
I've been trying to create a controller in my project for delivering what could turn out to be quite complex reports. As a result they can take a relatively long time and a progress bar would certainly help users to know that things are progressing. The report will be kicked off via an AJAX request, with the idea being that periodic JSON requests will get the status and update the progress bar.
I've been experimenting with the AsyncController as that seems to be a nice way of running long processes without tying up resources, but it doesn't appear to give me any way of checking on the progress (and seems to block further JSON requests and I haven't discovered why yet). After that I've tried resorting to storing progress in a static variable on the controller and reading the status from that - but to be honest that all seems a bit hacky!
All suggestions gratefully accepted!
Here's a sample I wrote that you could try:
Controller:
public class HomeController : AsyncController
{
public ActionResult Index()
{
return View();
}
public void SomeTaskAsync(int id)
{
AsyncManager.OutstandingOperations.Increment();
Task.Factory.StartNew(taskId =>
{
for (int i = 0; i < 100; i++)
{
Thread.Sleep(200);
HttpContext.Application["task" + taskId] = i;
}
var result = "result";
AsyncManager.OutstandingOperations.Decrement();
AsyncManager.Parameters["result"] = result;
return result;
}, id);
}
public ActionResult SomeTaskCompleted(string result)
{
return Content(result, "text/plain");
}
public ActionResult SomeTaskProgress(int id)
{
return Json(new
{
Progress = HttpContext.Application["task" + id]
}, JsonRequestBehavior.AllowGet);
}
}
Index() View:
<script type="text/javascript">
$(function () {
var taskId = 543;
$.get('/home/sometask', { id: taskId }, function (result) {
window.clearInterval(intervalId);
$('#result').html(result);
});
var intervalId = window.setInterval(function () {
$.getJSON('/home/sometaskprogress', { id: taskId }, function (json) {
$('#progress').html(json.Progress + '%');
});
}, 5000);
});
</script>
<div id="progress"></div>
<div id="result"></div>
The idea is to start an asynchronous operation that will report the progress using HttpContext.Application meaning that each task must have an unique id. Then on the client side we start the task and then send multiple AJAX requests (every 5s) to update the progress. You may tweak the parameters to adjust to your scenario. Further improvement would be to add exception handling.
4.5 years after this question has been answered, and we have a library that can make this task much easier: SignalR. No need to use shared state (which is bad because it can lead to unexpected results), just use the HubContext class to connect to a Hub that sends messages to the client.
First, we set up a SignalR connection like usual (see e.g. here), except that we don't need any server-side method on our Hub. Then we make an AJAX call to our Endpoint/Controller/whatever and pass the connection ID, which we get as usual:var connectionId = $.connection.hub.id;. On the server side of things, you can start your process on a different thread and retutn 200 OK to the client. The process will know the connectionId so that it can send messages back to the client, like this:
GlobalHost.ConnectionManager.GetHubContext<LogHub>()
.Clients.Client(connectionId)
.log(message);
Here, log is a client-side method that you want to call from the server, hence it should be defined like you usually do with SignalR:
$.connection.logHub.client.log = function(message){...};
More details in my blog post here
If you have an access to your web server machine as a choice you can create windows service or simple console application to perform long running work. Your web app should add a record to db to indicate that operation should start, and your windows service which periodicaly checks for new records in db should start performing task and report progress to db.
Your web app can use ajax request to check for progress and show to users.
I used this method to implement excel reports for asp.net mvc application.
This report was created by windows service which runs on machine and constantly checks for new records in reports table and when it finds a new record it starts to create a report and indicate a progress by updating record field. Asp.net mvc application just has been adding new report record and tracking progress in database untill it finished and then provided a link to ready file to be downloaded.