Develop Reference

.Net framework to manage background running processess on seperate machines

I am having an asp.mvc application which resides on a server.From this application, I want to start a process which is a bit long-running operation and will be resource intensive operation.
So what I want to do is I want to have some user agent like 3 which will be there on 3 machines and this user agent will use resources of their respective machines only.
Like in Hadoop we have master nodes and cluster in which tasks are run on the individual cluster and there is 1 master node keeping track of all those clusters.
In Azure, we have virtual machines on which tasks are run and if require Azure can automatically scale horizontally by spinning up the new instance in order to speed up the task.
So I want to create infrastructure like this where I can submit my task to 3 user agents from the mvc application and my application will keep track of this agents like which agent is free, which is occupied, which is not working something like this.
I would like to receive progress from each of this user agent and show on my MVC application.
Is there any framework in .net from which I can manage this background running operations(tracking, start, stop etc..) or what should be the approach for this?
Update : I don't want to put loads of server for this long running operations and moreover I want to keep track of this long running process too like what they are doing, where is error etc.
Following are the approach which I am thinking and I don't know which will make more sense:
1) Install Windows Service in the form of agents of 2-3 computer on premises to take advantage of resp resources and open a tcp/ip connection with this agents unless and until the long running process is complete.
2) Use hangfire to run this long running process outside of IIS thread but I guess this will put load on server.
I would like to know possible problems of above approaches and if there are any better approaches than this.

Hangfire is really a great solution for processing background tasks, and we have used used it extensively in our projects.
We have setup our MVC application on separate IIS servers which is also a hangfire client and just enqueues the jobs needs to be executed by hangfire server. Then we have two hangfire server instances, which are windows service application. So effectively there is no load on the MVC app server to process the background jobs, as it is being processed by separate hangfire servers.
One of the extremely helpful feature of hangfire is its out of the box dashboard, that allows you to monitor and control any aspect of background job processing, including statistics, background job history etc.
Configure the hangfire in application as well as in hangfire servers
public void Configuration(IAppBuilder app)
{
GlobalConfiguration.Configuration.UseSqlServerStorage("<connection string or its name>");
app.UseHangfireDashboard();
app.UseHangfireServer();
}
Please note that you use the same connection string across. Use app.UseHangfireServer() only if you want to use the instance as hangfire server, so in your case you would like to omit this line from application server configuration and use only in the hangfire servers.
Also use app.UseHangfireDashboard() in instance which will serve your hangfire dashboard, which would be probably your MVC application.
At that time we have done it using Windows Service, but if had to do it now, I would like to go with Azure worker role or even better now Azure Web Jobs to host my hangfire server, and manage things like auto scaling easily.
Do refer hangfire overview and documentation for more details.

Push messages to MSMQ from your MVC app and have your windows services listen (or loop) on new messages entering the queue.
In your MVC app create a ID per message queued, so make restful API calls from your windows services back to the mvc app as you make progress on the job?

Have a look at Hangfire, this can manage background tasks and works across VMs without conflict. We have replaced windows services using this and it works well.
https://www.hangfire.io

Give a try to http://easynetq.com/
EasyNetQ is a simple to use, opinionated, .NET API for RabbitMQ.
EasyNetQ is a collection of components that provide services on top of the RabbitMQ.Client library. These do things like serialization, error handling, thread marshalling, connection management, etc.
To publish with EasyNetQ
var message = new MyMessage { Text = "Hello Rabbit" };
bus.Publish(message);
To subscribe to a message we need to give EasyNetQ an action to perform whenever a message arrives. We do this by passing subscribe a delegate:
bus.Subscribe<MyMessage>("my_subscription_id", msg => Console.WriteLine(msg.Text));
Now every time that an instance of MyMessage is published, EasyNetQ will call our delegate and print the message’s Text property to the console.
The performance of EasyNetQ is directly related to the performance of the RabbitMQ broker. This can vary with network and server performance. In tests on a developer machine with a local instance of RabbitMQ, sustained over-night performance of around 5000 2K messages per second was achieved. Memory use for all the EasyNetQ endpoints was stable for the overnight run

Is it possible to have an ASP.NET application perform a timed action? [duplicate]

Once a day, I want my ASP.NET MVC4 website, which may be running on multiple servers, to email a report to me. This seems like a pretty common thing to want to do, but I'm having a tough time coming up with a good way to do it.
Trying to run a timer on a server to do this work is problematic for a couple of reasons. If I have multiple servers then I'd have the timer running on all of them (in case a server goes down); I'd need to coordinate between them, which gets complicated. Also, trying to access the database via Entity Framework from a background thread adds the complication that I must employ a locking strategy to serialize construction/disposal of the DbContext object between the periodic background thread and the "foreground" Controller thread.
Another option would be to run a timer on the servers, but to have the timer thread perform a GET to a magic page that generates and emails the report. This solves the DbContext problem because the database accesses happen in a normal Controller action, serialized with all of the other Controller accesses to the database. But I'm still stuck with the problem of having potentially more than one timer running, so I'd need some smarts in the Controller action to ignore redundant report requests.
Any suggestions? How is this sort of thing normally done?

You should not be doing this task from your web application as Phil Haack nicely explains it in his blog post.
How is this sort of thing normally done?
You could perform this task from a Windows Service or even a console application that is scheduled to run at regular intervals using the Windows Scheduler.

The proper solution is to create a background service that runs independently of your website. However, if that is not an option there is a hack where you can use the cache as explained in Easy Background Tasks in ASP.NET by Jeff Atwood.

A few options:
If you are hosting on Azure as a Website, check out WebJobs which was released recently in preview (http://azure.microsoft.com/en-us/documentation/articles/web-sites-create-web-jobs/)
If you don't want the pain of extracting out your email logic outside of the website, expose that functionality at a url (with a handler, mvc action, etc.) and then run a Windows Scheduled task that hits that url on a schedule.
Write a simple console app that is executed similarly via a Windows Scheduled task.
Or write a simple Windows Service that internally is looping and checking the time and when reached, hits that url, runs that exe, or has it's own code to send you the email.

I would recommend running Quartz.NET as a Windows Service:
Quartz.NET - Enterprise Job Scheduler for .NET Platform
There's boilerplate code for a Windows Service in the download.

What service type should I use for a SOA solution to exec a long running task?

I am developing a solution in .Net utilising the VMWare Web Service API to create new isolated virtualised development environments.
The front end will be web based. Requestors input details for the specific environment which will be persisted to SQL. So I need to write an engine of some sort to pull the data from SQL and work on the long running task of creating resource pools, switch port groups and cloning existing VM templates etc. As work progresses, events will be raised to write logs and update info back to SQL. This allows requestors to pull data back into a webpage to see how it's progressing or if it's completed.
The thing I am struggling with is how to engineer the engine which will exec the long running task of creating the environment. I cannot write a windows service (which I would like) as we work in a very secure environment and it's not possible (group policy etc). I could write a web service to execute the tasks, extending the httpRuntime executionTimeout to allow the task to complete. But I'm keen to hear what you guys think may be a better solution to use (based on .Net 3.5). The solution needs to be service oriented as we may be using it on other projects within our org. What about WWF, WCF? I have not used any of the newer technologies available since .Net 2.0 as we've only just been approved to move up from .Net 2.0.

First of all, a Windows Service isn't insecure. One of software development devils is discarding a solution by ignorance or because a lack of investigation, requirement analysis and taking decisions collaborately.
Anyway, if you want to do it in a SOA way, WCF is going to be your best friend.
But maybe you can mix a WCF service hosted by Internet Information Services and Message Queue Server (MSMQ).
A client calls a WCF service operation and enqueues a message to some Message Queue Server queue. Later, a Windows scheduled task executed overtime, checks if your queue has at least an incoming message, and task processes this and others until you dequeue all messages.
Doing this way, IIS WCF host will never need to increase its execution time and it'll serve more requests, while the heavy work of executing this long tasks is performed by a Windows scheduled task, for example a console application or a PowerShell cmdlet (or just a PowerShell script, why not?).
Check this old but useful MSDN article about MSMQ: http://msdn.microsoft.com/en-us/library/ms978430.aspx

I don't understand your comment regarding services being insecure, I think that is false.
However I would look into creating something that uses a messaging bus/daemon.
RabbitMQ: http://www.rabbitmq.com/
MassTransit: http://masstransit-project.com/
NServiceBus: http://nservicebus.com/
Then you can basically use something like WCF, Console App or Service as your daemon.

Schedule method invocation C#

I'm using c# to communicate with twitter and i need to code a schedule system to send twitter messages at a user defined date.
The messages to be sent are in a database with the date and time of deliver.
Which is the best method to check the db for scheduled messages and send it when the time arrives?

How accurate do you need the timing to be? Could you get away with polling the database every 5 minutes, saying "Tell me all the messages which need to be delivered before current time + 5 minutes" and then sending them all? (And marking them as sent in the database, of course.) Does it matter if they're a bit early or late?
You can do the scheduling on the C# side to make it more accurate, but unless you really need to I'd stick with a very simple solution.
(Depending on your database there may be clever ways of getting callbacks triggered etc... but again, I'd stick with the simplest solution which works.)

In addition to the windows service option (or background thread), you could just set up a scheduled task to run an app that polls the DB and sends the tweets once every defined interval.
Windows schedules can be setup using C# if needed and are really easy to set up manually.

There are several ways to do this, but I guess the best way is to set up a Windows Service that will periodically poll (frequency is up to you) the DB for any scheduled tweets that hasn't been sent.
Needless to say you'll need to handle scenarios such as the Internet connection or DB being down, etc.

In fact the solution consists in using a windows service but it can't communicate directly with the ASP.NET MVC app. I've added a Web Service that handles the task and a System.Threading.Timer in Windows Service to periodically call the Web Service.

sending email notifications for future dates in asp.net 3.5

I want to develop an Online Reminder service in ASP.NET 2.0 (C#) and SQL2005. But I am not getting the concept of reminder service. What I know is using an online reminder service I can schedule a reminder for future dates, which is sent to me (who schedule reminder) via email or SMS on that date. But in asp.net how to do this, caz anyone can schedule a reminder for any date, how we'll know that when to send that mail to the person. We have to put some loop or what.
So please guide me, what is the concept of an online reminder service and how I can easily develop this application using ASP.NET and SQL
Edited
I am on Shared hosting server, so that solution must be able to work on shared hosting.
Or
Please tell me if anyone knows about any FREE and open-source reminder service CMS which I can download and study it.

Microsoft SQL Server 2005 have scheduling (sql jobs) and email features. You may even donot need to use ASP.NET.

Ideally, you would have a windows service that would periodically (every few minutes) check if any new reminders need to be sent out. Since you are on shared hosting, you probably can't install a service though.
I'm not very familiar with windows shared hosting, but if you have the option of creating scheduled/cron job type tasks you could probably do it that way.
If you can't create a scheduled task on your server, another option would be to create a scheduled task on your home PC with a program/script that runs every few minutes and simply hits a special web page on your site. That page could then have the code that checks for reminders and sends them out. It's a bit of a hack, but it should work.

Have a look at Quartz.Net (http://quartznet.sourceforge.net/). You can create an instance of the quartz scheduler in your Application_Start event and as long as the ASP.Net application is running, it will poll the database and trigger any functions you have registered with it. Since you are on a shared host environment, this is probably your best bet unless your hosting provider has a scheduler that can trigger a WebForm (or ASP.Net MVC Controller) periodically.

First you will obviously need to create a user interface and database to store the reminders. That part you got. The next step is to create a service which periodically queries the database for reminders that are due for notification.
The best way to do this is to write a lightweight Windows Service which, as you suggest, uses a loop and a reasonable sleep time (so as not to monopolize the CPU) to continually check the database for reminders and dispatches notifications. It then processes each reminder based on your requirements.
But since you are on shared hosting, you can't deploy a Windows Service, so the next best thing is to run a background thread on Application_Start of your global.asax. There are many examples of how to do this, e.g.:
http://www.west-wind.com/WebLog/posts/67557.aspx
What are some best practices for managing background threads in IIS?

Shared hosting will not work well with what you are trying to do. You could create a background polling thread on Application start, but it will get shut down at some point and may actually be prohibited by your hosting company. An infinite loop will most likely be detected by your hoster and result in your account being automatically shut down, especially if it is using a fair bit of CPU. As John suggests, there may be a scheduled tasks or hosted cron option with your ISP, but generally, those are just for doing things like nightly backups, not really having the level of granularity you need.
Simple answer is, you most likely need something other than a hosted account. You may need to look into a VPS shared hosting service or you may wish to consider looking into MS Azure or Amazon EC2. To do this right, you need to create an application, or better, a service that runs constantly, something a shared hosting account will not provide.

There also a few services out there who can call a specific web page on your service periodically. You could use that to make the page check if there are any reminders that need to be sent.
However since you're then relying on an external site you can't control this might not be the ideal solution if it is very important that the reminders are always being sent.

1) Create a database for storing messages, with a datestamp
2) Create an SQL job, that selects all messages in a time period
3) From the SQL job, you can initialize an .net based SQL Function, that would send out the emails with the System.Net.Mail namespace.

You might consider a 'hack' using the Cache expiration in for triggering events. Create new cache keys that expire at specific Date-Times to run the reminder or make it recur at defined intervals, checking a queue to see if anything new should be sent.
See:
Easy Background Tasks in ASPNET