I am having an asp.mvc application which resides on a server.From this application, I want to start a process which is a bit long-running operation and will be resource intensive operation.
So what I want to do is I want to have some user agent like 3 which will be there on 3 machines and this user agent will use resources of their respective machines only.
Like in Hadoop we have master nodes and cluster in which tasks are run on the individual cluster and there is 1 master node keeping track of all those clusters.
In Azure, we have virtual machines on which tasks are run and if require Azure can automatically scale horizontally by spinning up the new instance in order to speed up the task.
So I want to create infrastructure like this where I can submit my task to 3 user agents from the mvc application and my application will keep track of this agents like which agent is free, which is occupied, which is not working something like this.
I would like to receive progress from each of this user agent and show on my MVC application.
Is there any framework in .net from which I can manage this background running operations(tracking, start, stop etc..) or what should be the approach for this?
Update : I don't want to put loads of server for this long running operations and moreover I want to keep track of this long running process too like what they are doing, where is error etc.
Following are the approach which I am thinking and I don't know which will make more sense:
1) Install Windows Service in the form of agents of 2-3 computer on premises to take advantage of resp resources and open a tcp/ip connection with this agents unless and until the long running process is complete.
2) Use hangfire to run this long running process outside of IIS thread but I guess this will put load on server.
I would like to know possible problems of above approaches and if there are any better approaches than this.
Hangfire is really a great solution for processing background tasks, and we have used used it extensively in our projects.
We have setup our MVC application on separate IIS servers which is also a hangfire client and just enqueues the jobs needs to be executed by hangfire server. Then we have two hangfire server instances, which are windows service application. So effectively there is no load on the MVC app server to process the background jobs, as it is being processed by separate hangfire servers.
One of the extremely helpful feature of hangfire is its out of the box dashboard, that allows you to monitor and control any aspect of background job processing, including statistics, background job history etc.
Configure the hangfire in application as well as in hangfire servers
public void Configuration(IAppBuilder app)
{
GlobalConfiguration.Configuration.UseSqlServerStorage("<connection string or its name>");
app.UseHangfireDashboard();
app.UseHangfireServer();
}
Please note that you use the same connection string across. Use app.UseHangfireServer() only if you want to use the instance as hangfire server, so in your case you would like to omit this line from application server configuration and use only in the hangfire servers.
Also use app.UseHangfireDashboard() in instance which will serve your hangfire dashboard, which would be probably your MVC application.
At that time we have done it using Windows Service, but if had to do it now, I would like to go with Azure worker role or even better now Azure Web Jobs to host my hangfire server, and manage things like auto scaling easily.
Do refer hangfire overview and documentation for more details.
Push messages to MSMQ from your MVC app and have your windows services listen (or loop) on new messages entering the queue.
In your MVC app create a ID per message queued, so make restful API calls from your windows services back to the mvc app as you make progress on the job?
Have a look at Hangfire, this can manage background tasks and works across VMs without conflict. We have replaced windows services using this and it works well.
https://www.hangfire.io
Give a try to http://easynetq.com/
EasyNetQ is a simple to use, opinionated, .NET API for RabbitMQ.
EasyNetQ is a collection of components that provide services on top of the RabbitMQ.Client library. These do things like serialization, error handling, thread marshalling, connection management, etc.
To publish with EasyNetQ
var message = new MyMessage { Text = "Hello Rabbit" };
bus.Publish(message);
To subscribe to a message we need to give EasyNetQ an action to perform whenever a message arrives. We do this by passing subscribe a delegate:
bus.Subscribe<MyMessage>("my_subscription_id", msg => Console.WriteLine(msg.Text));
Now every time that an instance of MyMessage is published, EasyNetQ will call our delegate and print the message’s Text property to the console.
The performance of EasyNetQ is directly related to the performance of the RabbitMQ broker. This can vary with network and server performance. In tests on a developer machine with a local instance of RabbitMQ, sustained over-night performance of around 5000 2K messages per second was achieved. Memory use for all the EasyNetQ endpoints was stable for the overnight run
Related
I want to create an ASP.MVC App with which I want to send some messages to my telegram channel via Telegram Bot Api at a certain time automatically, even when I am not in my web app.
For example I want to send one of my messages on these dates: 2018/03/23, 2018/03/28, 2018/04/05.
I want to send these messages without any human help and automatically
but I do not know how to do it. Now I can send my message at this time, not at the later time and automatically.
You should take a look at HangFire which is available as nuget packages and may suit your needs perfectly.
An easy way to perform background processing in .NET and .NET Core applications. No Windows Service or separate process required.
Backed by persistent storage. Open and free for commercial use.
Which allows to create :
Delayed jobs are executed only once too, but not immediately, after a certain time interval.
var jobId = BackgroundJob.Schedule(
() => Console.WriteLine("Delayed!"),
TimeSpan.FromDays(7));
With .Net native there is no such option available (at least I am not aware of it).
Either create a windows service which is monitoring the database and performing the operations.
Or, a little bit more simple, create a scheduled task which is running every 5 minutes eg. and executing a console application.
Why you want to implement the same into Asp.Net MVC..? In MVC you can't shedule the task, still if you can but it not proper way of implementation.
As per your requirement you should have to go with windows service, this will be a perfect approach for your requirement implementation.
I want to Scheduling in Asp.net
I have following options to implement this
To write SQLServer JOB(I dont want to do this.Dont want to go outside of .Net environment)
Second option is I will write windows service and this window service will call asp.net
webservice then this webservice calls asp.net method
(I also dont need to do this because my hosting provider might not be allow me to install
window service)
Third option is I call my scheduling method in Application_Start event in global class
(Drawback is, webserver will kill thread any time )
To call Scheduling Code in Page_Load event of Home Page(Might be nobody visits my website for hours
,Also page execution might be slow due to scheduling code)
I also found some online services that calls your page at given interval,some are given below
http://www.cronservice.co.uk/new/
http://scheduler.codeeffects.com
Anybody give me bettor solution of this and also explain why it is bettor?
Thanks in Advance
The ASP.NET application isn't the right place to implement scheduling. I would suggest creating a service or a scheduled task that runs in short intervals.
You don't have many options in a shared hosting environment. My host (WinHost) allows remote access to their database, so I was able to create an executable that ran on a local server with Task Scheduler.
The performance isn't great since the database is accessed over the internet, but it's still better than attempting to run pseudo scheduled tasks with ASP.NET.
Some hosts also offer a service that will request a url within your site on a scheduled basis. However, this didn't work for me because the task I had to run took longer than the request timeout.
There is no one solution that fits all. SQL jobs and windows jobs (scheduled thru windows task scheduler) are very widely used. In one of my previous work places they had jobs that ran on multiple platforms (mainframe,windows,sql server). Failure in some of these jobs, would cost in thousands by the day. So they employed something called ESP. This software monitored jobs on all platforms and sent a message to the control room in case of a failure.
If you throw some more light on the requirement, we might be able to help you better.
ASP.NET is not the right place to house your Scheduled Tasks. I'm using Quartz.net when I have to create Scheduled Tasks.
Create a page that launches your task and place it at the URL http://www.mydomain.com/runtask.
Create a scheduled task on your home PC that sends a request to http://www.mydomain.com/runtask.
You'll need to keep your home PC on all the time.
Ideally I would go with number 1 as you get full control/history/error reporting etc. You can write an SSIS job in .NET and have SQL server schedule it.
However, I have had a similar problem with shared hosting that is very restrictive. What I did was create a page which runs the process on page load (using validation in the querystring for security). I then used a machine I have which is always on to schedule a Windows Task Scheduler (this is part of Windows as standard) to call a bit of VB script that opens the browser and then shuts it.
I am developing a solution in .Net utilising the VMWare Web Service API to create new isolated virtualised development environments.
The front end will be web based. Requestors input details for the specific environment which will be persisted to SQL. So I need to write an engine of some sort to pull the data from SQL and work on the long running task of creating resource pools, switch port groups and cloning existing VM templates etc. As work progresses, events will be raised to write logs and update info back to SQL. This allows requestors to pull data back into a webpage to see how it's progressing or if it's completed.
The thing I am struggling with is how to engineer the engine which will exec the long running task of creating the environment. I cannot write a windows service (which I would like) as we work in a very secure environment and it's not possible (group policy etc). I could write a web service to execute the tasks, extending the httpRuntime executionTimeout to allow the task to complete. But I'm keen to hear what you guys think may be a better solution to use (based on .Net 3.5). The solution needs to be service oriented as we may be using it on other projects within our org. What about WWF, WCF? I have not used any of the newer technologies available since .Net 2.0 as we've only just been approved to move up from .Net 2.0.
First of all, a Windows Service isn't insecure. One of software development devils is discarding a solution by ignorance or because a lack of investigation, requirement analysis and taking decisions collaborately.
Anyway, if you want to do it in a SOA way, WCF is going to be your best friend.
But maybe you can mix a WCF service hosted by Internet Information Services and Message Queue Server (MSMQ).
A client calls a WCF service operation and enqueues a message to some Message Queue Server queue. Later, a Windows scheduled task executed overtime, checks if your queue has at least an incoming message, and task processes this and others until you dequeue all messages.
Doing this way, IIS WCF host will never need to increase its execution time and it'll serve more requests, while the heavy work of executing this long tasks is performed by a Windows scheduled task, for example a console application or a PowerShell cmdlet (or just a PowerShell script, why not?).
Check this old but useful MSDN article about MSMQ: http://msdn.microsoft.com/en-us/library/ms978430.aspx
I don't understand your comment regarding services being insecure, I think that is false.
However I would look into creating something that uses a messaging bus/daemon.
RabbitMQ: http://www.rabbitmq.com/
MassTransit: http://masstransit-project.com/
NServiceBus: http://nservicebus.com/
Then you can basically use something like WCF, Console App or Service as your daemon.
I have a data loading application that has to be executed multiple times per day at irregular intervals. I am planning to write a service to kick off the downloads and import the data to a database server. Are there advantages to using a standard service over a webservice or vice versa?
I think you're missing the point here.
Web Services typically are used for a form of communication or remote execution. You call a remote function on a web-service to either adjust the behavior of the machine it's running on or to retrieve data from it.
Windows Services are background processes that run on a machine without any "logged on user" being required. They can perform tasks and do things while the user is at the login screen, or do elevated operations. You can talk to services to adjust their behavior or retrieve information, but it's general purpose is different than a webservice.
The biggest notable difference here is that web-services must be called, they don't run on their own.
For your application I would suggest using a Windows (Standard) Service, as you can have it execute code once per day. I would only use a web-service if you've got something else to automate the calls to the web-service and you require a response from the server detailing it's execution result (success/fail/warning/etc...)
You could also consider writing a normal (windows or console) application that is triggered by a Windows Scheduled Task. What you've described doesn't necessarily sound like something that would require a service.
Sounds like a good use of a windows service to me. Off the top of my head, I'd use a windows service if:
1. Work is performed on a scheduled basis (regular or irregular intervals) in the background;
2. No interaction is needed - work is just done in the background and kicks off based on polling or some other type of trigger (message dropped in queue, database value trigger, scheduled timespan, etc.);
3. Needs to be monitored (either starts/stops along with logging) and you can take advantage of WMI, perfmon and event log with little effort.
A web service is better for tasks that are interactive (like if you wanted to initiate the download based upon a request received).
Sounds like a windows service is the approach you should take.
Hope this helps. Good luck!
If "irregular interval" does mean, the application is invoked by another application, I would use a web service.
If the action is scheduled, I would use a windows service.
If you are working with SQL Server (scheduled or not), I would also consider SQL Server Integration Services.
I need to implement a background process that runs on a remote windows server 24/7. My development environment is C#/ASP.NET 3.5. The purpose of the process is to:
Send reminder e-mails to employees and customers at appropriate times (say 5:00PM on the day before a job is scheduled)
Query and save GPS coordinates of employees when they are supposed to be out on jobs so that I can later verify that their positions were where they were supposed to be.
If the process fails (which it probably will, especially when updates are added), I need for it to be restarted immediately (or within just a few minutes) as I would have very serious problems if this process failed to send a notification, log a GPS coordinate, or any of the other tasks its meant to perform.
Implement your process as a Windows service.
For a straightforward example of how
to code a Windows service this in
.Net see http://www.developer.com/net/csharp/article.php/2173801 .
To control what happens should the
service fail configure this through
the "Recovery" tab on your service
in services.msc. (see image below)
For higly critical operation you
might look into setting up a server cluster for mitigating single
server failure (see http://msdn.microsoft.com/en-us/library/ms952401.aspx ).
(source: microsoft.com)
You need a Windows Service. You can do non-visual iterative operations in windows services.
Another alternative is to create a normal application and run it on a schedule. Your application is run at certain times a day to perform its actions, depending on how often you need to log GPS coordinates and send reports. If your service doesn't need to run constantly this is usually the recommended approach, as services are supposed to be limited to always-on applications.
As well as being a service, you might want to run on a cluster, and make your service known to the cluster management software.
You can create Windows Service (server programming on Windows) or use scheduler to periodically execute a task.
Depending on the requirements for the high availability, program can be installed on a fail-over cluster where there will be other server (passive node) started and quietly waiting as a hot-backup if the first (active node) dies. This is wide topic. Start with High availablity on Wikipedia.
In my experience if you need to run something 24x7 you need to have (one or more) watchdog process to verify that your service(s) are running correctly. Just relying on the normal service framework cannot guarantee that the program is working correctly - even if it looks like it is running. The watchdog program (which also is a service) can query the service automatically e.g. posting messages checking response times, querying for statistics and so on - when it detects problems it can restart the service (or do some other fail-recovery)
The reason for having a watchdog program as opposed to just rely on user queries to detect errors is that it can be done automatically. This is the preferred method because it allows for a proactive detection.