I am new in Azure and cloud computing and I want to do a little project in Azure. My idea is to get some data from a webpage and then store it in an Azure SQL database.
I want it to be an automatic script that will run every hour and save the results to the database. What service should I choose in the Azure platform? Worker service?
I have done something similar recently with more complexity(using queues and service bus)
For this Automatic background tasks Worker Role is the ideal choice
use Quartz.NEt for scheduling your job
This link is perfect for your needs just follow the tutorial and you will get it running in no time
Worker role with Quartz.NET
Related
I am having an asp.mvc application which resides on a server.From this application, I want to start a process which is a bit long-running operation and will be resource intensive operation.
So what I want to do is I want to have some user agent like 3 which will be there on 3 machines and this user agent will use resources of their respective machines only.
Like in Hadoop we have master nodes and cluster in which tasks are run on the individual cluster and there is 1 master node keeping track of all those clusters.
In Azure, we have virtual machines on which tasks are run and if require Azure can automatically scale horizontally by spinning up the new instance in order to speed up the task.
So I want to create infrastructure like this where I can submit my task to 3 user agents from the mvc application and my application will keep track of this agents like which agent is free, which is occupied, which is not working something like this.
I would like to receive progress from each of this user agent and show on my MVC application.
Is there any framework in .net from which I can manage this background running operations(tracking, start, stop etc..) or what should be the approach for this?
Update : I don't want to put loads of server for this long running operations and moreover I want to keep track of this long running process too like what they are doing, where is error etc.
Following are the approach which I am thinking and I don't know which will make more sense:
1) Install Windows Service in the form of agents of 2-3 computer on premises to take advantage of resp resources and open a tcp/ip connection with this agents unless and until the long running process is complete.
2) Use hangfire to run this long running process outside of IIS thread but I guess this will put load on server.
I would like to know possible problems of above approaches and if there are any better approaches than this.
Hangfire is really a great solution for processing background tasks, and we have used used it extensively in our projects.
We have setup our MVC application on separate IIS servers which is also a hangfire client and just enqueues the jobs needs to be executed by hangfire server. Then we have two hangfire server instances, which are windows service application. So effectively there is no load on the MVC app server to process the background jobs, as it is being processed by separate hangfire servers.
One of the extremely helpful feature of hangfire is its out of the box dashboard, that allows you to monitor and control any aspect of background job processing, including statistics, background job history etc.
Configure the hangfire in application as well as in hangfire servers
public void Configuration(IAppBuilder app)
{
GlobalConfiguration.Configuration.UseSqlServerStorage("<connection string or its name>");
app.UseHangfireDashboard();
app.UseHangfireServer();
}
Please note that you use the same connection string across. Use app.UseHangfireServer() only if you want to use the instance as hangfire server, so in your case you would like to omit this line from application server configuration and use only in the hangfire servers.
Also use app.UseHangfireDashboard() in instance which will serve your hangfire dashboard, which would be probably your MVC application.
At that time we have done it using Windows Service, but if had to do it now, I would like to go with Azure worker role or even better now Azure Web Jobs to host my hangfire server, and manage things like auto scaling easily.
Do refer hangfire overview and documentation for more details.
Push messages to MSMQ from your MVC app and have your windows services listen (or loop) on new messages entering the queue.
In your MVC app create a ID per message queued, so make restful API calls from your windows services back to the mvc app as you make progress on the job?
Have a look at Hangfire, this can manage background tasks and works across VMs without conflict. We have replaced windows services using this and it works well.
https://www.hangfire.io
Give a try to http://easynetq.com/
EasyNetQ is a simple to use, opinionated, .NET API for RabbitMQ.
EasyNetQ is a collection of components that provide services on top of the RabbitMQ.Client library. These do things like serialization, error handling, thread marshalling, connection management, etc.
To publish with EasyNetQ
var message = new MyMessage { Text = "Hello Rabbit" };
bus.Publish(message);
To subscribe to a message we need to give EasyNetQ an action to perform whenever a message arrives. We do this by passing subscribe a delegate:
bus.Subscribe<MyMessage>("my_subscription_id", msg => Console.WriteLine(msg.Text));
Now every time that an instance of MyMessage is published, EasyNetQ will call our delegate and print the message’s Text property to the console.
The performance of EasyNetQ is directly related to the performance of the RabbitMQ broker. This can vary with network and server performance. In tests on a developer machine with a local instance of RabbitMQ, sustained over-night performance of around 5000 2K messages per second was achieved. Memory use for all the EasyNetQ endpoints was stable for the overnight run
I would like to have a Worker Process to run independently from Websites in Azure. I will use this worker to create Websites and databases dynamically so I don't want WebJobs.
To create theses websites I would like to be able to call the Worker from a WebService in c#.
Also, I want to schedule task to clean databases each day. I read at some places that I need Azure Scheduler or Worker Role with Quartz.net.
I would like to know what is the best solution in Azure for me. Thanks!
Another vote for Azure Automation. You should be able to easily call it from your web service by using the Automation's own API.
If your databases are Azure SQL Databases (the managed Azure service) then you could use Azure Elastic database jobs to perform the daily cleanup.
If not, then Azure Scheduler would be my choice as well.
Based on what I understood it seems like you want to build an automation process and be able to manage it.
I think Azure Automation could possibly be one option for you.
Azure Automation is an Azure service for simplifying cloud management through process automation. Using Azure Automation, manual, frequently-repeated, long-running, and error-prone tasks can be automated to increase reliability, efficiency, and time to value for your organization.
Azure Automation provides a highly-reliable, highly-available workflow execution engine that scales to meet your needs. In Azure Automation, processes can be kicked off manually, by 3rd-party systems, or at scheduled intervals so that tasks happen exactly when needed.
For more details you can take a look at the Azure Automation here:
https://azure.microsoft.com/en-us/documentation/services/automation/
I am developing an email marketing WinForm application. And for the scheduling of campaigns, I decided to use Quartz.NET. I need it to be running as a Windows Service. but I also want the user to be able to add a job (e.g. a campaign that needs to be run everyday at 8 AM which is basically running a .bat file) to the service through the program.
I am also saving all the job schedules in the database so that when you stop/start the OS or Windows Service it can still read from all the jobs that need to be run.
How can I add a job to the service while the service is running? The dynamic addition/removal of the job to the service is of course much preferred. Stopping the service and reading all the jobs again from the database is, to be frank, my last resort.
You can install Quartz.Net as a windows service and configure it to use AdoJobStore.
Your WinForm application will be the interface where you can add, suspend and remove jobs and triggers.
You do not have to stop your windows services while you're doing that.
You do not need to read your database to figure out what's happening in Quartz but you must use the APIs provided.
I've done something similar long time ago; my quartz.net "manager" was a web application.
You can read my answer here.
I guess the steps to do what you're looking for a pretty much the same.
Reference to Quartz.net samples and a free book can be found here and here.
I have been building a Windows Phone 8 app and Windows Azure cloud service that will allow people to store schedules in the cloud. I have implemented a single sign on system and a cloud service used to store the schedule items.
I have also started building a cloud service to send push notifications however the plan is to send notifications based on scheduled times that have been stored in the cloud, the notification system works however only if I send a notification with a tester application.
Does anyone know how to send notifications based on a time record in an SQL database on Azure?
Thanks
As far as i understand you want a user to make a todoitem with a timestamp.?
If this is true, you could do it in some different ways. The easiest and most expensive is to only use azure. Then you need to use azure scheduler, to compare a time stamp with the current time and when the current time exceeds the time stamp, you send a push notification.
A cheaper way is to have an old pc, where you install server capabilities and go to the server and check the database with the time stamp. And then invoke a server api when you have a match.
But be aware of timezones :)
A link about timestamp in Azure tables
How to use Windows Azure Table Storage Timestamp or Etag field
Are you referring to the Azure Scheduler in Mobile Services?
Check out this link.
Otherwise I believe you'd need to have a service running that will run a task on scheduled times and occasionally poll the database for new times to run (pretty simple to do. Sounds like you probably have most of that already).
So here is my requirement,
I have a service provider that generates a files with data in a proprietary format on a text file. I have to download the file, parse it, then process it for specific records, and then update the database accordingly. The process needs to run multiple times on specific days (mostly Mondays & Fridays).
I am using Windows Azure platform. My website is hosted on azure & using SQL Server as my database. Using C# to create the database updater app.
What is the best approach to schedule a database updater job? Do I even need the updater application to be written on there is some better way to achieve the above.
Thanks
Based on the information you've provided I'd start by exploring Web Jobs (http://www.hanselman.com/blog/IntroducingWindowsAzureWebJobs.aspx).
Web Jobs allow you to create a schedule jobs that will run for you at intervals or continually.