I have a web application where users can enqueue tasks to export some data. The process can take up to 2-3 hours but can complete in seconds depending what was requested.
The export process will be handled like that:
public class ExportCommandHandler: IAsyncRequestHandler<ExportCommand, bool>
{
...
public async Task<bool> Handle(ExportCommand message)
{
return await _externalScheduler.EnqueueJobAsync(new ExportJob(...));
}
}
The _externalScheduler.EnqueueJobAsync currently just calls a self-hosted webapi method exposed in the external process to start a job there.
The problem here I want to get a notification when the job was completed to notify user it's ready if he's still browsing our web-site (planning to use SignalR here). If it wasn't hosted in the external process it would be obvious.
But what's the best approach to get notified when it's hosted in the external process? Is exposing a web-method in the web application for external process to call after completion is a proper way?
We're currently planning using Quartz.NET for processing external jobs but don't mind trying some other things like Hangfire.
Sorry it wasn't clear to me if the external process is owned by a third party system or it's yours.
In case it is a third party a web hook is a good and simple solution. You can define an endpoint which is called when the external process finishes.
Otherwise I would suggest using a messaging system to communicate both services. Just send a message to the external process to initiate the export, and send another message from the external process when the job is finished. This way you ensure that if one service is down at the moment of the notification the message will be sent and processed as soon as the service goes up again.
Related
I am working on a Service Fabric Application, in which I am running my Application that contains a bunch of ASP.NET Core Web APIs. Now when I run my application on my local service fabric cluster that is configured with 5 nodes, the application runs successfully and I am able to send post requests the exposed Web APIs. Actually I want to hit the code running on a same cluster node with different post requests to the exposed APIs on that particular node.
For further explanation, for example there is an API exposed on Node '0' that accept a post request and execute a Job, and also there is an API that abort the running job. Now when I request to execute a Job, it starts to execute on Node '0' but when I try to abort the Job, the service fabric cluster forward the request to a different node for example say node '1'. In resulting I could not able to abort the running Job because there is no running Job available on Node '1'. I don't know how to handle this situation.
For states, I am using a Stateless service of type ASP.Net Core Web API and running the app on 5 nodes of my local service fabric cluster.
Please suggest what should be the best approach.
Your problem is because you are running your APIs to do a Worker task.
You should use your API to schedule the work in the Background(Process\Worker) and return to the user a token or operation id. The user will use this token to request the status or cancel the task.
The first step: When you call your API the first time, you could generate a GUID(Or insert in DB) and put this message in a queue(i.e: Service Bus), and then return the GUID to the caller.
The second step: A worker process will be running in your cluster listening for messages from this queue and process these messages whenever a message arrives. You can make this a single thread service that process message by message in a loop, or a multi-threaded service that process multiple messages using one Thread for each message. It will depend how complex you want to be:
In a single threaded listener, to scale you application, you have to span multiple instances so that multiple tasks will run in parallel, you can do that in SF with a simple scale command and SF will distribute the service instances across your available nodes.
In a multi-threaded version you will have to manage the concurrency for better performance, you might have to consider memory, cpu, disk and so on, otherwise you risk having too much load in a single node.
The third step, the cancellation: The cancellation process is easy and there are many approaches:
Using a similar approach and enqueue a cancellation message
Your service will listen for the cancellation in a separate thread and cancel the running task(if running).
Using a different queue to send the cancellation messages is better
If running multiple listener instances you might consider a topic instead of a queue.
Using a cache key to store the job status and check on every iteration if the cancellation has been requested.
Table with job status, where you check on every iteration as you would do with the cache key.
Creating a Remote endpoint to make a direct call to the service and trigger a cancellation token.
There are many approaches, these are simple, and you might make use of multiple in combination to have a better control of your tasks.
You'll need some storage to do that.
Create a table (e.g JobQueue). Before starting to process the job, you store in a database, store the status (e.g Running, it could be an enum), and then return the ID to the caller. Once you need to abort/cancel the job, you call the abort method from the API sending the ID you want to abort. In the abort method, you just update the status of the job to Aborting. Inside the first method (which runs the job), you'll need to check this table onde in a while, if it's aborting, then you stop the job (and update the status to Aborted). Or you could just delete from the database once the job has been aborted or finished.
Alternatively, if you want the data to be temporary, you could use a sixth server as a cache server and store data there. This cache server could be a clustered server as well, but then you would need to use something like Redis.
I need to create a ASP.NET API service that when called doesnt wait for a response from the webserver. Basically I have a long sql task that I want to run then when its completed send an email to the user to tell them the job is done. It needs to avoid server response timeout, so something that just lets the user carry on without waiting round. I cant seem to find a way in MVC to do this, is it possible?
IMHO, I would queue this job and process it using another process outside IIS.
For example, this would be the flow:
User performs a request to your API to start the long task, but what API does in the server-side is queueing the whole task.
API returns a 200 OK response specifying that the job was queued successfully. You may use Azure Service Bus, Queues, MSMQ, RabbitMQ, Redis or even SQL Server using a table to maintain job state.
Some Windows Service, Azure Worker Role or a periodic scheduled task dequeues the task, processes it and as soon as it ends, it sends an email to the user to notify that the operation was done.
Queue the task and return the response immediately.
Basically, your server-side handler (controller action, Web API method, whatever) shouldn't invoke the long-running back-end task directly. It should do something relatively fast to just queue the task and then immediately return some indication that the task has been successfully queued. Another process entirely should actually execute the long-running task.
What I would recommend is two server-side applications. One is the web application, the other is either a Windows Service or a periodically scheduled Console Application. The web application would write a record to a database table to "queue" the process. This could contain simply:
User who queued the process
When it was queued
What process was queued (if there would ever be more than one, for example)
Status of the process ("queued" initially)
Anything else you might want to store.
Just insert a record here and then return to the user. At this point the web application has done its job.
The Windows Service (or Console Application) would check this database table for "queued" records. When it finds one, update the status to "processing" (so other executions don't try to run the same one) and invoke the long-running process. When the long-running process is complete, update the status to "complete" (or just delete the record if you don't want it anymore) and notify the user. (Handle error conditions accordingly, of course. Maybe re-try, maybe notify the user of the error, etc.)
By separating the concerns like this you place the appropriate responsibilities in the appropriate application contexts and provide the user with exactly the experience they're looking for. You additionally open the door for future functionality, such as queueing the process by means other than the web application or running reports on queued/running/failed/etc. processes by examining that database table.
Long story short: Don't try to hack a web application so that it doesn't behave like a web application. Use the technologies for their appropriate purposes.
I have some legacy ASMX IIS hosted service. Client applications make subscribe or unsubscribe to the web service. Via some internal to the web service logic it needs to send messages to the subscribed applications periodically.
What is the best way to do the part of the long running task ? I understand opening Thread with long running task not a good idea to do under IIS.
ASMX services cannot do what you're asking for: they cannot just decide to send a message to the client. All they can do is respond if the client requests it.
You can hack around and come up with one method to start the long-running task, and another method to poll for the status of the task. This works, but it can be expensive.
The better model is to perform the long-running task in a separate Windows Service. Have that service host a simple WCF service which will only be used by the main service (the one that talks to the clients). The main (WCF) service would use a Duplex channel to communicate with the clients. That way, it can "call" the clients whenever there is news about one of the long-running tasks.
Usually in such cases when you don't have a way to push the result back, create an unique ID for the long running task and sent it back to the client, after that run the task and have a table in database or something else where you store the status of the task. The client will pull periodically the service to see the task' status by given ID. Once it finds the task is completed it will retrieve the result.
And is completely fine to have a thread running inside IIS doing its job.
I am building a WCF service that will expose several operations, it will run in IIS because it needs HTTPS endpoints. Most of the operations will perform within seconds or less; however, one or two of these operations will take between 5-90 minutes.
The primary consumer of this service will be an ASP.NET MVC application; what is the correct way to do handle this?
Should I jackup the timeout and do some ajax calls? Should I add a table to my database, and have the long running operations update this database, and have the web interface poll this table every minute? I'm not sure what (if there is) the generally accepted best practice for this.
I wrote something similar for my senior project, basically a job scheduling framework.
I chose to go down the path of storing the "status" of the "job" in the database.
I wrote a manager windows service that implemented a WCF client (proxy)
I wrote a WCF Service that implemented my "worker host".
The manager service would read the queue from the database, and hand out work to all of my "worker hosts". The reason I had windows service perform this task as opposed to just having the UI talk directly to the worker host, was because it gave an extra level of control over the whole process.
I didn't like the idea of having "the network cable unplugged" from my worker host, and never getting a status update again from this specific job. So, the windows service gives me the ability to constantly monitor the progress of the WCF worker host, and if a connection error ever occurs (or something else unexpected), I can update the status to failed. Thus, no orphaned jobs.
Take a look at this
WCF Long Running Operations
There could be other options but they are nearly the same. You can also come up with some push notifications (I assume no data is returned) as one int the following link
WCF Push
How would one use SignalR to implement notifications in an .NET 4.0 system that consists of an ASP.NET MVC 3 application (which uses forms authentication), SQL Server 2008 database and an MSMQ WCF service (hosted in WAS) to process data? The runtime environment consists of IIS 7.5 running on Windows Server 2008 R2 Standard Edition.
I have only played with the samples and do not have extensive knowledge of SignalR.
Here is some background
The web application accepts data from the user and adds it to a table. It then calls an one way operation (with the database key) of the WCF service to process the data (a task). The web application returns to a page telling the user the data was submitted and they will be notified when processing is done. The user can look at an "index" page an see which tasks are completed, failed or are in progress. They can continue to submit more tasks (which is independent of previous data). They can close their browser and come back later.
The MSMQ based WCF service reads the record from the database and processes the data. This may take anything from milliseconds to several minutes. When its done processing the data, the record is updated with the corresponding status (error or fail) and results.
Most of the time, the WCF service is not performing any processing, however when it does, users generally want to know when its done as soon as possible. The user will still use other parts of the web application even if they don't have data to be processed by the WCF Service.
This is what I have done
In the primary navigation bar, I have an indicator (similar to Facebook or Google+) for the user to notify them when the status of tasks has changed. When they click on it, they get a summary of what was done and can then view the results if they wish to.
Using jQuery, I poll the server for changes. The controller action checks to see if there is any processes that were modified (completed or failed) and return them otherwise waits a couple of seconds and check again without returning to the client. In order to avoid a time out on the client, it will return after 30 seconds if there was no changes. The jQuery script waits a while and tries again.
The problems
Performance degrades with every user that views a page. There is no need for them to do anything in particular. We've noticed that memory usage of Firefox 7+ and Safari increases over time.
Using SignalR
I'm hoping that switching to SignalR can reduce polling and thus reduce resource requirements especially if nothing has changed task wise in the database. I have trouble getting the WCF service to notify clients that its done with processing a task given the fact that it uses forms based authentication.
By asking this question, I hope someone will give me better insight how they will redesign my notification scheme using SignalR, if at all.
If I understand correctly, you need a way of associating a task to a given user/client so that you can tell the client when their task has completed.
SignalR API documentation tells me you can call JS methods for specific clients based on the client id (https://github.com/SignalR/SignalR/wiki/SignalR-Client). In theory you could do something like:
Store the client id used by SignalR as part of the task metadata:
Queue the task as normal.
When the task is processed and de-queued:
Update your database with the status.
Using the client id stored as part of that task, use SignalR to send that client a notification:
You should be able to retrieve the connection that your client is using and send them a message:
string clientId = processedMessage.ClientId //Stored when you originally queued it.
IConnection connection = Connection.GetConnection<ProcessNotificationsConnection>();
connection.Send(clientId, "Your data was processed");
This assumes you mapped this connection and the client used that connection to start the data processing request in the first place. Your "primary navigation bar" has the JS that started the connection to the ProcessNotificationsConnection endpoint you mapped earlier.
EDIT: From https://github.com/SignalR/SignalR/wiki/Hubs
public class MyHub : Hub
{
public void Send(string data)
{
// Invoke a method on the calling client
Caller.addMessage(data);
// Similar to above, the more verbose way
Clients[Context.ClientId].addMessage(data);
// Invoke addMessage on all clients in group foo
Clients["foo"].addMessage(data);
}
}