We have 20+ console applications for background processing(sending emails, tagging customers based on condition etc). Currently, all are configured using a windows task scheduler and hosted on a different server. it's difficult to manage and schedule manually.
I am new to microservice architecture.
can I create and run all this jobs with MS architecture.
This can be done via programatically and configuration option. Basically save the schedule configuration and apply on application. you may have to design User interface or xml file based configuration to save the schedule details for each job. In java, Quartz framework is quite popular. For .Net - refer Quartz.Net Cron expression
To create a background process or schedule jobs you can do it in your .Net application or via the Windows Service application.
To do it in your .Net Application you can do it with Quartz or HangFire. Both are reliable and straightforward.
To do it in a Microservice environment you can create an independent service and add your new jobs to it.
Also, even you can listen to events inside this specific service and fire it whenever each service wants. It would be quite flexible
Another option is that you can use Worker Services in. NET which helps you to create and run your specific instance of your application (Thanks to dotnet core).
If you ask me which one is the simple one I would ask Quartz.
Related
I'm looking for a better infrastructure setup for managing and deploying internally developed applications which are executed periodically.
The current setup grew into an unmonitorable heterogeneous collection of applications, that can only be executed directly on the scheduler VM.
Current situation:
Windows environment
Bunch of Jobs written in PowerShell and as C# applications some containing rather complex logic, some perform ETL-Operations
Jobs either configured as service or console applications triggered by default Windows scheduler and running on a dedicated VM
Application specific logging into Log-Files (some applications)
Configuration via app.config file for each C# console application
Windows scheduler doesn't provide nice Web-GUI to watch and monitor job executions.
Ideal situation:
Central monitoring: Overview of all jobs (when run, failures)
Trigger manually via a web frontend
Trigger job execution via API with possibility to check whether execution succeeded.
Central job configuration (connection strings, configuration parameters)
Constraints:
No cloud: Due to internal restrictions the software has to reside inside our own network. Our company owns a sufficiently dimensioned server rack for hosting required servers internally.
Considered Options
Azure WebJobs
From what I have read this would be exactly the solution I'm looking for. Due to our "no cloud" policy we'd need to host our own Azure Pack internally, which might require quite some effort to set up and is possibly a technical overkill for these requirements.
Self-written Web-API Project
Another option would be to write a dedicated Web-API project that contains all job functions, has one central configuration and exposes the job functions as Web-API Methods and using Quartz.net for scheduling.
However if possible I'd prefer to use some standard software, so I won't be responsible for maintaining yet another central piece of our infrastructure.
Which option would you choose? Or are there any better alternatives?
I think what you look for is so-called Master Data Management, so you should check for
http://www.talend.com/
https://www.informatica.com/
http://www.tibco.com/
I am developing an email marketing WinForm application. And for the scheduling of campaigns, I decided to use Quartz.NET. I need it to be running as a Windows Service. but I also want the user to be able to add a job (e.g. a campaign that needs to be run everyday at 8 AM which is basically running a .bat file) to the service through the program.
I am also saving all the job schedules in the database so that when you stop/start the OS or Windows Service it can still read from all the jobs that need to be run.
How can I add a job to the service while the service is running? The dynamic addition/removal of the job to the service is of course much preferred. Stopping the service and reading all the jobs again from the database is, to be frank, my last resort.
You can install Quartz.Net as a windows service and configure it to use AdoJobStore.
Your WinForm application will be the interface where you can add, suspend and remove jobs and triggers.
You do not have to stop your windows services while you're doing that.
You do not need to read your database to figure out what's happening in Quartz but you must use the APIs provided.
I've done something similar long time ago; my quartz.net "manager" was a web application.
You can read my answer here.
I guess the steps to do what you're looking for a pretty much the same.
Reference to Quartz.net samples and a free book can be found here and here.
I was wondering if a web application for ASP.NET supports a scheduling feature similar to Cron jobs on a Unix-like platform?
PS. I'm coding in VS2010 using C#, and I need this feature to run C# script on a configurable schedule, based on date & time.
This sounds like a windows service project type.
You coluld use a web project as an editor, to configure various parameters, and persist them to a db or file, and then read them from the windows service.
Using a web app to execute some task at different intervals is not the best ideea.
While this could be done in theory, maybe you can split it in 2projecs: 1 web app that saves config params and maybe diplays the task status, and 1 windows service that reads the params, executes the tasks and maybe updates its status.
Why not just use the Windows Task scheduler to schedule the C# script? Do you not have admin privileges?
I use cron to kickoff my scheduled jobs on unix servers, and the Windows Task Scheduler to run them on Windows servers.
We have a commercial product that may be exactly what you're looking for. JAMS is an enterprise grade task scheduler and it includes ASP.NET controls that you can drop onto your web pages. We have controls to schedule, monitor and manage tasks as well as querying past history.
And, our .NET class library can be called from C# (or any other .NET language)
If you're interested, more information is available at:
http://www.JAMSScheduler.com
I need your advice for scheduling tasks in the MVC3 Webapp.
My task is to create some generic scheduler for different services in the webapp that can be used later in development. For example we have some available tasks that user can schedule whenever he wants.
I didn't want to reinvent the wheel and found the Quartz.Net library that can be used to create the scheduler.
I know that it's not a good idea to host scheduling inside webapp cause webserver can recycle the application pools, etc, so i decided to use it inside a Windows Service and then using Quartz.NET remoting feature to trigger tasks inside my webapp.
But i've found some issues with that. Correct me if I'm wrong, but when i tried to use the Quartz.NET remoting it runs the job inside the Windows Service process, it means that it needs to know about all types inside my webapp, so all the assemblies of the webapp should be referenced by it, and i need to have another config file for database, etc. So in case I write new job class, i can't easily schedule it, i need to stop the service and renew the library for it, so it's not very generic approach.
I can't find any info that Quartz.NET can run jobs only based on it's interface.
So I came up with idea to write my own scheduler that will be hosted in the Windows Service, and will have some IJob interface that will be implemented in the webapp. I will also use .Net remoting using IPC channel.
So the webapp will be like .Net Remoting Server, and when i want to add some new job and then schedule it, i will just have to write new job that implements IJob interface.
I will register it as
IpcChannel channel = new IpcChannel("CurrentIPC");
ChannelServices.RegisterChannel(channel);
RemotingConfiguration.RegisterWellKnownServiceType(
typeof(SimpleJob), "SimpleJob", WellKnownObjectMode.SingleCall);
RemotingConfiguration.RegisterWellKnownServiceType(
typeof(ComplexObject), "ComplexObject", WellKnownObjectMode.SingleCall);
in this case i will have two Job types registered. Then when scheduling the job i will pass the name of the class and on the Windows Service side that will stand for client (executing objects on the webapp side) i will just bind the passed name of the class with IJob like this:
Dictionary<string, IJob> jobs = new Dictionary<string, IJob>();
void AddJob(string name)
{
IJob obj = (IJob)Activator.GetObject(typeof(IJob), string.Format("ipc://CurrentIPC/{0}", name));
jobs.Add(name, obj);
}
So now i don't need to bother about references to my app and other stuff, the scheduler will do it's job without knowing anything, just IJob interface and executing tasks on the webapp side.
If i'm wrong or it's too complex and there are some other simpler methods of doing this, or there are some pitfalls that i'm not aware of, can you help me with that?
Thank you.
P.S.
Also there was an idea to have separate scheduler that will run the web app methods directly by executing a link to specified service in the web app, for example "http://localhost:3030/Request/12" and that's all, but in my web app you should be authorized to execute such request and again we have issues we need to resolve, and we will have additional load to the webserver with such requests in case of thousands of scheduled tasks.
I think you are on the right track, I would create the scheduler using Quartz.NET and host it in a Windows Service because of the app pool recycling issue.
It will trigger tasks/services in your webapp using specific URL:s for each task/service either in your web app or a separate web service instance.
Using this separation the scheduler only needs to know about the urls and the schedule and does not need to reference your app directly. This is also reusable in future projects.
I know that it's not a good idea to host scheduling inside webapp cause webserver can recycle the application pools, etc, so i decided to use it inside a Windows Service...
Why complicate? why not make it simple and use an external service to run webhooks at a period of time?
I use this service and I've been happy, though it's so easy that I could create my own service based on this simple procedure:
http://momentapp.com/
to simplify the call:
private void Run() {
try {
var work = RequestNewMessage(); // get work
ProcessWork(work); // process work
// Log work
}
catch(Exception ex) {
// Log Error
}
finally {
// set job for now plus1 minute
SetRecuringJob(DateTime.UtcNow.AddMinute(1));
}
}
private void SetRecuringJob(DateTime dt) {
PostJob("https://momentapp.com/jobs/new?job[at]={0:s}&job[method]=POST&job[uri]=http://yourapp.com/", dt);
}
A bit late I imagine, but could still be useful for others.
Here's a project I've spent quite a bit of time with that seems to fulfil most the requirements stated - in fact, I use it almost exclusively for callback jobs as described in the question. It's also what we use internally, so I do try to keep it updated.
http://backgroundworker.codeplex.com/
It's pretty similar to Quartz, but more focussed towards managing jobs and their data.
If your issue is you don't want to reference the entire web app from the WindowService, and I agree with that, why don't you simply create a utility dll implementing a generic job able to call your web app via a plain http url and implements the "actions" as a WCF REST service?
This I think will cleanup a bit the architecture and keep insulated ( and reusable ) the scheduler service in your organization.
I might be missing a requirement, but as you seem to be able to install a custom service, it sounds like you have full control over that machine. So you might as well schedule a Windows Task for your recurring job, which should give you a pretty solid basis for recurring tasks.
You can then schedule a VB script to load a url on a timely basis, like here: http://4rapiddev.com/internet/call-or-open-a-web-page-url-by-using-windows-task-scheduler-or-cronjob/
Anything goes of course, from batch file, over powershell to custom executable.
Upsides to this approach:
You can rely on default OS behaviour for what the OS is good at: scheduling :)
There's already a UI for the admin (albeit not a web UI)
You have no dependency but the OS
Possible downsides:
You have to build your own web management around it if you want to expose it to the user (e.g. use one of these as a basis: http://www.codeproject.com/Articles/2407/A-New-Task-Scheduler-Class-Library-for-NET or http://taskscheduler.codeplex.com/).
Security might be an issue if end-users are able to manipulate tasks
Less fine-grained control over what happens on failures or timeouts (not sure how big of an issue, as you can also manipulate how the Scheduler deals with those)
Edit: I did seem to have missed the requirement on the authentication, but when you're using Windows, you could allow for Windows authentication within a certain area of your app. And because you're using Scheduled Tasks which are Windows Authenticated, you have this covered. I never heavily used Windows authentication within an ASP.NET app, but you might even get away with only custom configuration and no extra programming.
I'm building a web application that will need to import data from other database servers when it starts.
I would like to have this import done automatically at regular intervals. I would also like to be able to start and stop the import process from my web application.
What would be the best implementation for the import agent - a Windows Service? Something else?
If your web application needs to have this data in memory, you can use the Cache class.
Set it to expire every X hours, as you need and when it expires, re-fetch the data..
You could create a Windows Service that uses Quartz.Net to run the scheduled tasks.
You should not run scheduled task from your web app, since you don't have any guarantee that your web app is running. You're at IIS app pool management's mercy.
You might want to look at Best way to run scheduled tasks.
Of what I heard this looks like a description for Microsoft Sync Framework. I have just few information about it for myself but will be pleased to see you pointed into that direction.
I'm not sure about your question because you are talking about hourly syncing. When talking web applications, there can't be a nice way to do such a task. You have to create a console app or best task would be a Windows Service Process (which are easier then it sounds)?
Sync Framework Intro
http://msdn.microsoft.com/en-us/sync/bb821992
Sync Framework Tutorial
http://alexduggleby.com/2007/12/16/sync-framework-tutorial-part-1-introduction/
Sync Framework Samples
http://archive.msdn.microsoft.com/sync
And, when I'm editing the answer with links
Nice guide to create a Windows Service (and setup)
http://www.codeproject.com/KB/dotnet/simplewindowsservice.aspx
(if first time, try it on a test project before the production project)
This might be an oversimplification, but can you create a class that does all of this work using a Timer, and then in the application_start of the global.asax, create a BackgroundWorker that kicks off this process?
Your web application could then control the BackgroundWorker object, starting/stopping as necessary.