I need some help with this..
This table I have has a date column in it, and when any of the dates in that column equal the servers date I need to tell my website/program to send out an email or perform some certain notification action to let the user know something.
I was thinking of having a program running on the server polling the data base a certain intervals but the problem with this is if the date is 01/31/11 10:30 AM and my interval is every 5 mins there potential for the polling to be inaccurate i.e. the poll polling at 10:35 AM. In other words I need the database to somehow notify something when "x" date has been hit exactly at that date.
I'd like to avoid having a 1sec interval checking the database as I think that would be a huge performance hit.
I'm using ASP.NET MVC 3 with MSSQL and LINQ Entity framework.
Any creative ideas?
You could use Quartz.net to setup those events. Quartz is pretty flexible and powerful - and it was meant for this sort of thing.
Do not have the database trigger the code. Have a trigger create a row in another table with information about what just happened.
Have a separate program periodically read from the second table to email users or whatever you need to do. Have that program delete the row from the table once it's done with the email.
I don't have any personal experience, but Sql Server CLR Integration might be the answer you are looking for. From the description it sounds like you can write almost anything that will compile against the .NET framework and deploy it to a sql server instance and Sql Server will be able to execute it. http://msdn.microsoft.com/en-us/library/ms254498.aspx
you either need to make use of a scheduler (e.g. DBMS_SCHEDULER in Oracle or SQL Server Jobs, etc) or find some third party tool like maybe Quartz.net as mentioned by another responder. Or maybe code something like the following into a polling app
select all jobs due in next 5 minutes, order by due date
while there are jobs
if the next job is due action it
else sleep for duration of interval till job due
loop
This is bit dirty, but I think it will give you the functionality you're looking for.
In Global.asax.cs
public DateTime LastMaxDateTime;
protected void Application_Start(object sender,EventArgs e)
{
LastMaxDateTime = GetMaxDateTime();
Thread bgThread=new Thread(BackgroundThread_CheckDatabase);
bgThread.IsBackground=true;
bgThread.Start();
}
private void BackgroundThread_CheckDatabase()
{
while(true)
{
DateTime dtMaxDateTime = GetMaxDateTime();
if(dtMaxDateTime > this.LastMaxDateTime)
{
//Send Notifications
this.LastMaxDateTime=dtMaxDateTime;
}
Thread.Sleep(5000); //5 seconds
}
}
private DateTime GetMaxDateTime()
{
//function that returns DateTime from something like "SELECT MAX(DateTimeColumn) FROM [MyTable]"
}
Basically, the code keeps track of the newest DateTime in your table and on each poll, checks to see if there's a newer DateTime in the database since the last time it checked. If so, you can send out your notifications. If you're not expecting many records in your table that could cause a race condition, then I don't see a problem with this as a quick solution.
Most efficient way to do it is to have an application that instead of polling runs event-driven.
For example, have a thread query the database for the earliest scheduled event and sleep until then. Then have another thread synchronously wait for a table change (e.g. in PostgreSQL this would be the NOTIFY/LISTEN statements) and signal the first thread to check if the earliest event has changed.
The easiest way is to keep track of the date of your last check. When you check again, pull all rows greater than the last check date and less than or equal to the new check date. To make sure you execute them, you could add a column for when the action was performed and update that. With an index on that new column there shouldn't be any performance problem with checking it every second for rows with a NULL DateExecuted.
You could also read ahead and sort the upcoming items by trigger date and do a Thread.Wait() until the next one comes up to be precise.
Related
I am trying to make a C# WinForms application that fetches data from a url that is saved in a table named "Links". And each link has a "Last Checked" and "Next Check" datetime and there is "interval" which decides "next check" based on last check.
Right now, what I am doing is fetching ID with a query BEFORE doing the webscraping, and after that I turn Last Checked into DateTime.Now and Next Check into null untill all is completed. Which both then gets updated, after web scraping is done.
Problem with this is if there is any "abort" with an ongoing process, lastcheck will be a date, but nextcheck will be null.
So I need a better way for two processes to not work on same table's same row. But not sure how.
For a multithreaded solution, the standard engineering approach is to use a pool of workers and a pool of work.
This is just a conceptual sketch - you should adapt it to your circumstances:
A worker (i.e. a thread) looks at the pool of work. If there is some work available, it marks it as in_progress. This has to be done so that no two threads can take the same work. For example, you could use a lock in C# to do the query in a database, and to mark a row before returning it.
You need to have a way of un-marking it after the thread finishes. Successful or not, in_progress must be re-set. Typically, you could use a finally block so that you don't miss it in the event of any exception.
If there is no work available, the thread goes to sleep.
Whenever a new work arrives (i.e. INSERT, or a nextcheck is due), one of sleeping threads is awakened.
When your program starts, it should clear any in_progress flags in the event of a previous crash.
You should take advantage of DBMS transactions so that any changes a worker makes after completing its work are atomic - i.e. other threads percieve them as they had happened all at once.
By changing the size of worker pool, you can set the maximum number of simultaneously active workers.
First thing, the separation of controller/workers might be a better pattern as mentioned in other answer. This will work better if the number of threads gets large and te number of links to check is large.
But if your problem is this:
But problem with it is, if for any reason that scraping gets
aborted/finishes halfway/doesn't work properly, LastCheck becomes
DateTime.Now but NextCheck is left NULL, and previous
LastCheck/NextCheck values are gone, and LastCheck/NextCheck values
are updated for a link that is not actually checked
You just need to handle errors better.
The failure will result in exception. Catch the exception and handle it by resetting the state in the database. For example:
void DoScraping(.....)
{
try
{
// ....
}
catch (Exception err)
{
// oh dear, it went wrong, reset lastcheck/nextcheck
}
}
What you reset last/nextcheck to depends on you. You could reset them to what they where at the start if when you determine 'the next thing to do' you also get the values of last/nextcheck and store in variables. Then in the event of failure just set to what they were before.
I've got a Microsoft SQL database containing a table called Bookings with a column EndDate and a column Status.
Whenever a row is inserted into Bookings, it will set the Status to be unavailable and insert an EndDate (like a week from now).
What I want to do is to automatically update Status from unavailable to available when the current date and time is equal to EndDate.
I can't use SQL Server Agent Jobs since I'm running Express Edition.
Thanks
This is a dangerous path that you want to go down. As soon as this process gets missed once you now have rows out there that have invalid data. Not to mention, you're spending an awful lot of processing time (and maintenance) just to mark a column that you should be able to determine at any point in time anyway.
Instead of trying to maintain duplicate data in your database (which will invariably get out of sync), just calculate it at run time or create a view to present it:
CREATE VIEW dbo.Bookings_WithStatus
AS
SELECT
booking_id,
start_date,
end_date,
CASE
WHEN COALESCE(end_date, GETDATE()) > GETDATE() THEN 'Unavailable'
ELSE 'Available'
END AS booking_status
FROM
dbo.Bookings
Although the logic is simple enough that I probably wouldn't even bother with a view myself.
#Sean Lange makes the excellent point that this can also be done as a computed column, which is probably better than using a view:
ALTER TABLE dbo.Bookings
DROP COLUMN booking_status
ALTER TABLE dbo.Bookings
ADD booking_status AS
CASE
WHEN COALESCE(end_date, GETDATE()) > GETDATE() THEN 'Unavailable'
ELSE 'Available'
END
If you need to do this logic constantly and if you can't use SQL Agent, maybe you need to consider a simple application like Windows Service or Console Application.
This can be triggered once per day by Windows Scheduler and that will go through all the records in your database, check if your conditions have met and update records if it is necessary?
You could do this by a service or a script run using sqlcmd, but I'd be inclined to step back and look at the base requirement.
To what end must the Status change at that exact time? The output data that you're looking for sounds much like it could be better accomplished using a view rather than a persisted field, and with much less complexity.
A solution without updating the status column might be sufficient. Your "available" rows are those whose EndDate is greater or equal to GETUTCDATE() assuming EndDate is in UTC
Let's say I wanted to display a Reminder (dynamically created by user) in my ASP.NET MVC View
Every WEEK/MONTH/.... until "infinity".
This reminder has to be seperate database record (or in my case EF Entity Instance object) for each reocurrence, because i'm storing specific data in my database for each occurrence.
How would you guys go about "inserting" these reminder clones into the database?
I Can't insert infinite reminders, .. And choosing an arbitrary date say "2000 years" from now seems wrong, and also inserts a lot of records into the database.
Kind of clueless here...
Thanks in advance for any (alternative) solution/ advice.
*It is key tough that i can store data for each occurence!*
Inserting reminders into the database, IMHO, is a bad way of implementing it. It is just lazy.
My approach would be to store the conditions and then test the conditions on a regular basis. Perhaps just keeping a timestamp of last reminder check and if it is more than 8 hours, re-calculate - something around these lines.
UPDATE
In order to know which reminder has been seen by the user or not, for every type you just keep a timestamp of when it was read. If the difference is more than the item interval, then it must be shown. If timestamp is less than the the time reminder must be shown, then it must be shown.
Is there the remotest possibility that your code will be used in 10 years time?
If not try 10 years.
I am new to threads and in need of help. I have a data entry app that takes an exorbitant amount of time to insert a new record(i.e 50-75 seconds). So my solution was to send an insert statement out via a ThreadPool and allow the user to begin entering the data for the record while that insert which returns a new record ID while that insert is running. My problem is that a user can hit save before the new ID is returned from that insert.
I tried putting in a Boolean variable which get set to true via an event from that thread when it is safe to save. I then put in
while (safeToSave == false)
{
Thread.Sleep(200)
}
I think that is a bad idea. If i run the save method before that tread returns, it gets stuck.
So my questions are:
Is there a better way of doing this?
What am I doing wrong here?
Thanks for any help.
Doug
Edit for more information:
It is doing an insert into a very large (approaching max size) FoxPro database. The file has about 200 fields and almost as many indexes on it.
And before you ask, no I cannot change the structure of it as it was here before I was and there is a ton of legacy code hitting it. The first problem is, in order to get a new ID I must first find the max(id) in the table then increment and checksum it. That takes about 45 seconds. Then the first insert is simply and insert of that new id and an enterdate field. This table is not/ cannot be put into a DBC so that rules out auto-generating ids and the like.
#joshua.ewer
You have the proccess correct and I think for the short term I will just disable the save button, but I will be looking into your idea of passing it into a queue. Do you have any references to MSMQ that I should take a look at?
1) Many :), for example you could disable the "save" button while the thread is inserting the object, or you can setup a Thread Worker which handle a queue of "save requests" (but I think the problem here is that the user wants to modify the newly created record, so disabling the button maybe it's better)
2) I think we need some more code to be able to understand... (or maybe is a synchronization issue, I am not a bug fan of threads too)
btw, I just don't understand why an insert should take so long..I think that you should check that code first! <- just as charles stated before (sorry, dind't read the post) :)
Everyone else, including you, addressed the core problems (insert time, why you're doing an insert, then update), so I'll stick with just the technical concerns with your proposed solution. So, if I get the flow right:
Thread 1: Start data entry for
record
Thread 2: Background calls to DB to retrieve new Id
The save button is always enabled,
if user tries to save before Thread
2 completes, you put #1 to sleep for
200 ms?
The simplest, not best, answer is to just have the button disabled, and have that thread make a callback to a delegate that enables the button. They can't start the update operation until you're sure things are set up appropriately.
Though, I think a much better solution (though it might be overblown if you're just building a Q&D front end to FoxPro), would be to throw those save operations into a queue. The user can key as quickly as possible, then the requests are put into something like MSMQ and they can complete in their own time asynchronously.
Use a future rather than a raw ThreadPool action. Execute the future, allow the user to do whatever they want, when they hit Save on the 2nd record, request the value from the future. If the 1st insert finished already, you'll get the ID right away and the 2nd insert will be allowed to kick off. If you are still waiting on the 1st operation, the future will block until it is available, and then the 2nd operation can execute.
You're not saving any time unless the user is slower than the operation.
First, you should probably find out, and fix, the reason why an insert is taking so long... 50-75 seconds is unreasonable for any modern database for a single row insert, and indicates that something else needs to be addressed, like indices, or blocking...
Secondly, why are you inserting the record before you have the data? Normally, data entry apps are coded so that the insert is not attempted until all the necessary data for the insert has been gathered from the user. Are you doing this because you are trying to get the new Id back from the database first, and then "update" the new empty record with the user-entered data later? If so, almost every database vendor has a mechanism where you can do the insert only once, without knowing the new ID, and have the database return the new ID as well... What vendor database are you using?
Is a solution like this possible:
Pre-calculate the unique IDs before a user even starts to add. Keep a list of unique Id's that are already in the table but are effectively place holders. When a user is trying to insert, reserve them one of the unique IDs, when the user presses save, they now replace the place-holder with their data.
PS: It's difficult to confirm this, but be aware of the following concurrency issue with what you are proposing (with or without threads): User A, starts to add, user B starts to add, user A calculates ID 1234 as the max free ID, user B calculates ID 1234 as the max free ID. User A inserts ID 1234, User B inserts ID 1234 = Boom!
I am developing a web app using ASP.NET 2.0 (C#), where on home page I am displaying recently added records. Adding of records frequency is around 1-5 records per day, so I decided not to put much overhead on the sql server by fetching recent records every time from db server.
So, To make the data cached I have used XML files, I have generated the XML file from dataset, (ds.WriteXML function in .NET), now lets say today (10 Jan 2008 12:30:00) I have created a file recent-cache.xml is created. So, the recent cache file is valid for one day.
Then if the difference between current date and last modified date is greater than or equal 1 day then cache xml file must be generated again, with the new data from the db server.
So, I want the code using which I can get the last modfied date of the xml file and then find the difference between both (current and file last modified-date) dates.
And also please tell me what I thought is the better solution, or we can do anything else, some other easy and speedy technique.
Thanks
You might consider using the ASP.NET Cache API, which exists to do the sort of job you're describing. You can add any object (like an XmlDocument, or a DataSet) to the Cache collection and specify how long you want it in there like so:
Cache.Insert("MyCacheKey", myObjectToCache, null, DateTime.Now.AddDays(1), null);
Then you could get at your cached data with a function like this:
const string CACHE_KEY = "MyCacheKey";
private DataSet RecentlyAdded()
{
if(Cache[CACHE_KEY] == null)
Cache.Insert(CACHE_KEY, GetRecentlyAddedFromDatabase(), null, DateTime.Now.AddDays(1), null);
return Cache[CACHE_KEY];
}
The caching API has lots of other neato features but this would accomplish what you want without having to roll your own file-based solution.
Note that if your app shuts down before the day is up, the Cache will shut down with it, and the "recently added" database query will have to run again the next time the data is requested.
edit: changed cache key to a string constant so it only has to be specified once.