I am creating a website in asp.net but i have some issue..
I have coded a program which can crawl a give web page i.e. thenextweb.com for its links, and content and images.
Now i want to store these crawled data inside my table *Crawlr_Data*.
I want that the crawler runs after every 30 minutes and updated the table with new links if available.
{ON the Home page of my website i am showing the information stored in the database}
How can i run the crawler on back end and update the database ?
What technology like (web services, WCF) should i use or any other thing in visual studio which i can use so that i if host website online its crawler keeps on running and updating table}
Please suggest
Thanks
There are two ways to do this with the Microsoft stack.
Create a service to run on the server. Have the service itself manage when it wakes up and crawls.
Create a console app to do the crawl. Run the console app as a scheduled task using windows task scheduler as often as you like.
I guess there are other ways to do it -- so saying there are just two is not totally accurate -- there are 3rd party programs that will do it for you also... I expect most if not all of them are implemented as a service. You could also write a program that runs on the server not as a console app or as a service. But this is generally a bad idea.
Related
I have a couple of questions regarding a project I recently started working on. This project requires me to discover devices on the network (broadcasting UDP packets on a specific port), store device information into a database, and then display all the data on a local web page. My questions are:
Can I have a C# program that runs in the background that deals with the device discovery and database management while also running a local webpage for users to interact with?
If so, which kind of Visual Studio project would be the best to
implement this project?
If not, how else could I do this?
We had a problem where we needed a long running process that was initiated by a user but which timed out. We resolved this by running a background Windows service which interacted with the database backing the website. We also have other tasks that we wish to run asynchronously following user action. We run three Windows services on the web server; this solution has been running successfully for a number of years. So, in answer to our specific points:
Yes, as a Windows service separate from your website project
There is a specific project type for this; you will also need to set up an installation project
There may be other ways but I would recommend this approach
The goal is that I want my overall website response time to be instantaneous.
The problem is that I do no have IIS access, my website is hosted using external service and I have no control to the IIS panel.
My current approach now is having a scheduled code that keeps my website alive. The problem with this only approach is the hosting service has an algorithm to shutdown all their hosted website like every some hours.
This is why I need to implement another approach which to warm up / pre-load the website each time it runs.
How to do this when there is no access to the IIS panel?
The solution requires no 3rd party sites; robots; or apps, you merely write a very simple app yourself that periodically performs a trivial web function, perhaps a REST GET. By performing this function say every few minutes not only do you guarentee that the IIS pool won't timeout and be cold for a client, but it also has the nice effect of ensuring your website is up and running in a warm condition (JIT'd; and running) ready for a real request for your non-heartbeat website requests.
e.g.
In your website expose a REST API, say www.misspiggy.com/api/hiyaaaa that does nothing other than to return HTTP 200 OK.
By implemententing this in your ASP.NET app, any request to the above URL will cause your stopped or cold ASP.NET website to be JIT'd during:
first deployment (and even then only during a request is made to it)
after the IIS AppPool has timed out and needs to restart on demand
The client code that makes the REST request can be anything:
a console app
a Windows service
WinForms/WPF app
The console app can be triggered to fire via Windows Task Scheduler say every 5 minutes thus saving you the hastle of building in a scheduler.
My current approach now is having a scheduled code that keeps my website alive. The problem with this only approach is the hosting service has an algorithm to shutdown all their hosted website like every some hours
I suggest you set your ping period to be a matter of minutes rather than hours.
No admin access to server required
The problem is that I do no have IIS access, my website is hosted using external service and I have no control to the IIS panel
It should be pointed out that this solution does not require you to install anything new on the server nor make any changes on the server.
Azure
It is interesting to note that Azure Application Insights has Availability Tests that though designed for testing web site availability, can be used for this exact same purpose of keeping your website alive and warm ready to go for web clients. In fact this is what I do for my web apps.
Doing so keeps response times and latency as low as possible.
There are a number of things you can do but a real simple solution is to use a website monitoring site something like statuscake or uptime robot there are a large number of them out there. You set them up call a page or pages on your website at set intervals to ensure it is still up this has the added bonus of keeping the site warm.
I would also precompile your mvc app if you arent already doing that.
HTH
I want to Scheduling in Asp.net
I have following options to implement this
To write SQLServer JOB(I dont want to do this.Dont want to go outside of .Net environment)
Second option is I will write windows service and this window service will call asp.net
webservice then this webservice calls asp.net method
(I also dont need to do this because my hosting provider might not be allow me to install
window service)
Third option is I call my scheduling method in Application_Start event in global class
(Drawback is, webserver will kill thread any time )
To call Scheduling Code in Page_Load event of Home Page(Might be nobody visits my website for hours
,Also page execution might be slow due to scheduling code)
I also found some online services that calls your page at given interval,some are given below
http://www.cronservice.co.uk/new/
http://scheduler.codeeffects.com
Anybody give me bettor solution of this and also explain why it is bettor?
Thanks in Advance
The ASP.NET application isn't the right place to implement scheduling. I would suggest creating a service or a scheduled task that runs in short intervals.
You don't have many options in a shared hosting environment. My host (WinHost) allows remote access to their database, so I was able to create an executable that ran on a local server with Task Scheduler.
The performance isn't great since the database is accessed over the internet, but it's still better than attempting to run pseudo scheduled tasks with ASP.NET.
Some hosts also offer a service that will request a url within your site on a scheduled basis. However, this didn't work for me because the task I had to run took longer than the request timeout.
There is no one solution that fits all. SQL jobs and windows jobs (scheduled thru windows task scheduler) are very widely used. In one of my previous work places they had jobs that ran on multiple platforms (mainframe,windows,sql server). Failure in some of these jobs, would cost in thousands by the day. So they employed something called ESP. This software monitored jobs on all platforms and sent a message to the control room in case of a failure.
If you throw some more light on the requirement, we might be able to help you better.
ASP.NET is not the right place to house your Scheduled Tasks. I'm using Quartz.net when I have to create Scheduled Tasks.
Create a page that launches your task and place it at the URL http://www.mydomain.com/runtask.
Create a scheduled task on your home PC that sends a request to http://www.mydomain.com/runtask.
You'll need to keep your home PC on all the time.
Ideally I would go with number 1 as you get full control/history/error reporting etc. You can write an SSIS job in .NET and have SQL server schedule it.
However, I have had a similar problem with shared hosting that is very restrictive. What I did was create a page which runs the process on page load (using validation in the querystring for security). I then used a machine I have which is always on to schedule a Windows Task Scheduler (this is part of Windows as standard) to call a bit of VB script that opens the browser and then shuts it.
So, this question has been asked lots, and i have seen many different answers, but nothing finite or absolute for my scenario.
_
What I want to do:
We have a website, with a community of users.
In the admin section of the website, there are buttons to run the following functions:
Email all the users our weekly newsletter [Thursday 4pm],
message users that day's information [Daily 6pm],
post to facebook through facebook connect [Daily 8am and 6pm],
etc, etc. (There will be new requirements coming soon too, but they will follow the same principal)
_
All I want to do is to run these functions automatically, so a member of staff does not need to go to the website, login, then click each of the buttons at the set times.
That is, effectively, have the server click these buttons automatically, at the set times mentioned above.
_
I have seen suggestions for building a service with a timer built in, which will call each function, or use windows task scheduler, build an exe / com etc, but i get the gut feeling it should not be this complicated.
We have the code already written to actually do the tasks in a .net web page, and using some custom built classes etc.
I just do not know how to automatically call these functions at the desired times.
_
Server Info:
Dedicated server running Windows Web Server 2008 R2 (64bit)
Development Enviro:
Visual Studio 2010 SP1, using .NET 4.0
_
Thank you in advance for your help.
Kindest Regards,
Del
Splitting out the code into separate exes/dlls that can be called from the Windows task scheduler as well as your application is the way to go.
There's no sense in re-inventing a scheduler inside your application when one exists in the operating system you are running on. Particularly as your application is a web application which should be doing web things not server things.
The added advantage is that you have made your application more modular and easier to upgrade.
By definition, web servers respond to requests. This fundamental design manifests in all kinds of ways, such as application pools being shut down after a period of inactivity. For this reason, it is not a good idea to execute time-dependent code inside the context of a web server, because it is difficult to guarantee that the code will actually run (without jumping through a lot of hoops that have other negative side effects). Instead, an external time-dependent mechanism should send a request to the web server to execute these functions.
You said that you didn't want to use Windows Scheduler or write an external exe because "it shouldn't be this complicated"; but I don't see what's complicated about having a scheduled task call a web service.
Well, though I agree with other answers cheering for windows task scheduler and that website has nothing to do with scheduling tasks (it can be the source of input parameters, not the task runner itself), you could try using scheduling library like quartz.NET to schedule and run your necessary processes in the background.
Download cURL - a command-line HTTP request utility. Use this utility from task scheduler to call your web page / web service with the specific query string required to execute your functions.
http://curl.haxx.se/
This isnt very secure - anyone that knew the correct query string could cause these functions to execute - but it gives you the general idea. In my opinion its much safer to write a console app that uses your assemblies to directly execute your functionality. Call the console app from task scheduler.
I have a question that is perhaps slightly towards architecture and design rather than syntax problem....
I have a c# winforms app on my desktop which I have built which is similar to a CRM. I make updates to customers records as they submit orders/invoices etc and what I would like to do is to build (or buy) a module that will update a remote database that sits behind a website onto hich registered clients log in. I want the clients to be able to see the status of their invoice/purchase etc as I have updated it on the winforms app.
I can think of a couple of options of the top of my head but would like to know more if you have done something similar
Things I am considering are;
>Replication - I think this is overkill as the updates are not
huge, are one way only, and not
critical they are in real time, and
also I am running SQL express on the
winforms app. This can be changed
but rather not
>create a text/xml file that gets created and uploaded to the web
server to a location that is
monitored every 5 minutes and then
updates the web database. - I am not
hosting the website myself so I do
not have complete control over what
I can install but I suspect I can
install a .NET 'filewatcher'
Anyway, I would appreciate your thought on my 'problem'
thanks
I think your best bet is to create a web service of some kind (I like using ServiceStack.net to create simple REST ones, much cleaner imo than WCF). This will sit on the server and be responsible for the server-side sync piece.
On the client, you could either have the winforms app fire off the call to the web service based on some threshold of activity, or you could have a windows service that you install with the winforms app which does it in a scheduled job or on a timer.
You'll want to be sure that you're doing all your calls over SSL of course, and make sure you're authenticating the clients, but that's the basic architectural approach I'd take.