I have been asked to add a feature to some software I have written which needs to send reminder emails.
The reminders can have a recurrence; for example, it might recur every 3 days or every 2 weeks. The reminders also have a start and end date. I have created an SQL table with these fields:
event_id
event_name
event_Description
event_startDate
event_endDate
RecurrenceType (e.g. Daily, Weekly, Monthly,Yearly)
Intarval
event_ActiveFlag
I now need to write a stored procedure that will run every time and send the reminders. I have no problem with sending via a C# console application; the trouble I am having is that I cannot figure out how to get the recurrences for the current day.
No matter what, you cannot get around needing to perform a computation to determine the list of recurrence dates for a given event. By "computation" I mean something like the following pseudocode:
foreach(event in activeEvents)
{
while(recurrenceDate <= currentDate && recurrenceDate <= event.EndDate)
{
event.RecurrenceDates.Add(recurrenceDate);
recurrenceDate.AddDays(event.IntervalLengthInDays);
}
}
I highly recommend using Douglas Day's excellent .NET iCal library (be sure to read the license terms to ensure you are permitted to use this in your app, though they are quite permissive) to calculate recurrence dates for you, because despite appearing straightforward, it's actually very difficult to correctly handle all the corner cases.
This will give you a list of dates that the event occurs on, up to the current date. There are two questions: when and where should you perform this calculation?
As for where, I recommend in the C# app, since you've already stated you have such a component involved. It is absolutely possible to do this in a performant, set-based way in your SQL database; however, the obvious, intuitive way to code this is using loops. C# is great at loops; SQL is not. And doing it the obvious way will make it much easier to debug and maintain when you (or someone else) looks at it six months down the line.
Regarding when, there are two possibilities:
Option 1: On demand: do it as part of the job that sends your daily reminder blast.
Fetch the data for all active events
Compute recurrence lists for them
Examine the last recurrent event (this will be the one closest to the current date) to determine if that event should be sent out as a reminder
Send eligible reminders
Option 2: Once: do it when the event is created and entered in the system.
When an event is created, calculate all recurrence dates through its end date
Store these dates in a table in the DB (event_id FK, event_date)
Your reminder job fetches a list of eligible events (both active and has the appropriate date) from the precomputed table
Send eligible reminders
Which option is better? My money is on #1. Why?
With option 1, if I realize I accidentally entered "daily" instead of "weekly" for my recurrence period, changing the event is much easier, because I don't have recompute and re-store all the recurrence data. This is the reason to calculate recurrence on demand, and should trump all but the most dire of performance related concerns (most of which could likely be fixed by throwing more hardware at the problem, or by better load balancing).
Option 2 is degenerate in the case of events without a defined end date. This is manageable by, say, calculating X years into the future, but what happens X years later?
A daily reminder job probably runs overnight, and so doesn't need to execute super fast. This means cutting down on its execution time isn't a priority. And the recurrence calculation time is probably going to be negligible anyway; I expect the bottleneck to be the actual email sending. However, if you have a lot of active, old events to consider (by "a lot", I mean billions) and/or your app server is grinding to a halt under load, this may become important.
Option 1 saves DB work. I don't mean storage space (although it does use less); that shouldn't matter unless you have a lot (many trillions) of events. I mean that it's less "chatter" back and forth between your app server and the DB, so there's less chance for a dropped connection, concurrency collision, etc. Please note this is incredibly trivial and really doesn't matter either way, unless your production environment has major problems.
Related
So my friend and I developed an Attendance Monitoring System that uses RFID and SMS (broadband stick). The problem we were facing is that it takes probably 3-5 seconds before the next student could time in. What happens after a student times in:
Fetches info (name,grade/section,pic,name and number of guardian/parent)
Displays the info (name, grade/section,pic)
Sends an sms using the name and number of the guardian/parent
Saves the info and date/time as the student's attendance
Clears the displayed info
My friend tried multi threading but still takes a couple of time for all the process to finish.
I was hoping to re develop this in c# but I cant find a way to minimize the time it takes to process a student's time in.
Ps
We're only novices in programming. C# and VB were the only languages we know and few MySQL. We just can't find the right term to look for.
So, you are looking for performance optimisation.
Switching from vb.net to c#.net will change nothing. Check CIL - https://en.m.wikipedia.org/wiki/Common_Intermediate_Language
Everything is translated to the same intermediate language.
What you can try:
Use profilers to narrow down where your code lags: https://learn.microsoft.com/en-us/visualstudio/profiling/?view=vs-2019
Check the performance of any remote apis from which you are getting your data.
Check the performance of your database. Indexes on tables, carefully written queries and nice design, could bring the times down to milliseconds
On multithreading: This would help if your problem was processing power or if your code was blocking in the same thread. Depending on the technology stack, this could or could not be the case.
My guess is that your spend most of your time in round trips to apis and/or databases. When you optimize everything else, then you'll see if there is need for parallelism. Try using the async versions of the functions wherever possible and you'll achieve some inherent parallelism anyway.
Question is pretty forward and it is bothering me for quite some time now. Perhaps any of you has an interesting view on this matter.
When creating an object lets say a memo.
This memo has a string (for the text obviously)
And a user id who has edited said text.
And this memo object is going to be saved to the database.
Is it common to add the current time to the object (eg. add a DateTime field / property) and save that object to db?
Or use the the current time of the database eg. when the row is inserted DateTime.Now in DB?
Considering that your application may run on multiple machines and users which may have different time zones or wrong time on their devices, the first option you proposed could affect data's integrity and you will end up with an unsuncronised database.
I feel it would not make much difference though it really depends on your requirement and project architecture. If you are interested in showing the time to user and you use some queue or background worker to insert row to DB (that is there could be some difference in time between creation of object and insertion into SQL), then it perhaps make sense to get the time at the time of creation of object. Else, if it only for record keeping then DB timestamp should be fine.
Although your example is simple one it may represent typical scenario in enterprise environment. So, instead of one user and memo there may be lots of users, lots of tasks to be executed by those users and there are supervisors who are monitoring how users are performing. With that in mind you should try to log everything that you can since supervisors will ask for different productivity reports. And basic elements of such reports are "Start time" and "End time" of activity.
That said, it is of lesser importance which time you are using as long as activities of all users may be compared. Do have in mind that some activities may be executed in different time zones. This leads us to the fact that time used for events (such as start time and end time) should come from common source. Either some middle layer or database.
Implicitly, you need to log user activities. So you will have a method that will have several parameters that are sufficient for capturing user activities. Execution of such method should be on middle or database layer and therefore time registered should be consistent and comparable.
With this approach you have possibility to extend your definition of captured events (not only start and end time but also some other relevant moments).
I would like some advice on how to best go about what I'm trying to achieve.
I'd like to provide a user with a screen that will display one or more "icon" (per say) and display a total next to it (bit like the iPhone does). Don't worry about the UI, the question is not about that, it is more about how to handle the back-end.
Let's say for argument sake, I want to provide the following:
Total number of unread records
Total number of waiting for approval
Total number of pre-approved
Total number of approved
etc...
I suppose, the easiest way to descrive the above would be "MS Outlook". Whenever emails arrive to your inbox, you can see the number of unread email being updated immediately. I know it's local, so it's a bit different, but now imagine having the same principle but for the queries above.
This could vary from user to user and while dynamic stored procedures are not ideal, I don't think I could write one sp for each scenario, but again, that's not the issue heree.
Now the recommendation part:
Should I be creating a timer that polls the database every minute (for example?) and run-all my relevant sql queries which will then provide me with the relevant information.
Is there a way to do this in real time without having a "polling" mechanism i.e. Whenever a query changes, it updates the total/count and then pushes out the count of the query to the relevant client(s)?
Should I have some sort of table storing these "totals" for each query and handle the updating of these immediately based on triggers in SQL and then when queried by a user, it would only read the "total" rather than trying to calculate them?
The problem with triggers is that these would have to be defined individually and I'm really tring to keep this as generic as possible... Again, I'm not 100% clear on how to handle this to be honest, so let me know what you think is best or how you would go about it.
Ideally when a specific query is created, I'd like to provide to choices. 1) General (where anyone can use this) and b) Specific where the "username" would be used as part of the query and the count returned would only be applied for that user but that's another issue.
The important part is really the notification part. While the polling is easy, I'm not sure I like it.
Imagine if I had 50 queries to be execute and I've got 500 users (unlikely, but still!) looking at the screen with these icons. 500 users would poll the database every minute and 50 queries would also be executed, this could potentially be 25000 queries per miuntes... Just doesn't sound right.
As mentioned, ideally, a) I'd love to have the data changes in real-time rather than having to wait a minute to be notified of a new "count" and b) I want to reduce the amount of queries to a minimum. Maybe I won't have a choice.
The idea behind this, is that they will have a small icon for each of these queries, and a little number will be displayed indicating how many records apply to the relevant query. When they click on this, it will bring them the relevant result data rather than the actual count and then can deal with it accordingly.
I don't know if I've explained this correctly, but if unclear, please ask, but hopefully I have and I'll be able to get some feedback on this.
Looking forward to your feeback.
Thanks.
I am not sure if this is the ideal solution but maybe a decent 1.
The following are the assumptions I have taken
Considering that your front end is a web application i.e. asp.net
The data which needs to be fetched on a regular basis is not hugh
The data which needs to be fetched does not change very frequently
If I were in this situation then I would have gone with the following approach
Implemented SQL Caching using SQLCacheDependency class. This class will fetch the data from the database and store in the cache of the application. The cache will get invalidated whenever the data in the table on which the dependency is created changes thus fetching the new data and again creating the cache. And you just need to get the data from the cache rest everything (polling the database, etc) is done by asp.net itself. Here is a link which describes the steps to implement SQL Caching and believe me it is not that difficult to implement.
Use AJAX to update the counts on the UI so that the User does not feel the pinch of PostBack.
What about "Improving Performance with SQL Server 2008 Indexed Views"?
"This is often particularly effective for aggregate views in decision
support or data warehouse environments"
I'm developing an event booking application and am having difficulty figuring out how to manage the booking process. I know about db transactions and a little bit about locking but I have a lot of business rules to validate before a booking can be committed and I'm worried about performance bottlenecks.
Here's a summary of what's going on:
An Event has a maximum number of slots
A user can book one spot in the Event
Each user has an account with money in and each event costs a certain amount
Given the above parameters, the following business rules are what I need to validate for a booking to take place:
The user hasn't already booked a spot for this Event
The user has enough funds to book the Event
The Event has at least one spot available
The Event does not conflict with other events the user has booked (not so much of an issue as I can check this when displaying the page and hide this Event from the user)
My main worry is that if I pull all the information from the db up front (i.e. Event, User, Account, and existing Bookings) that by the time I run all the validation and come to commit the new booking, the state of the system will have possibly changed (i.e. someone else has booked the last spot, money has left my account, etc).
If I was to lock the code/database tables around this booking process, then I've potentially (??) got the lock for quite a while affecting other operations in the system and causing performance issues at peak times.
Can anyone suggest an approach whereby I can manage or at least limit these concerns.
I'm building an asp.net app in c# and using sql server 2005.
I think a good example to look at is how Ticketmaster reserves seats for tickets that MAY get purchased. They tell you that you have so many minutes until the seats are put back into inventory. It pushes the purchaser to make a decision or someone else will have a chance at the seats. This is really your biggest hurdle. As for checking the business rules, you'll have to do that. There is no magic around what needs to be done there. If you need the data to validate a rule then that's what you need to do. With some proper mapping and outlining you can find the efficiencies. I hope that answered your question.
Good luck!
One solution:
Pre-emptively book the spot (with a status of "hold").
Validate.
If the booking can't be kept due to business rules, delete it. If not, change status to "booked"
If you go back to the 80s and read literature published on the topic of transaction processing you'll find that one of the most discussed example was the airline reservation systems. And for good reason, as it was one of the OLTP topics that exposed all the issues around transaction processing: correctness, troughput, contention, deadlocks. What you describe is a very similar problem, but instead of air flight seats you have event slots. So yes, you will have all those issues.
There is no magic pixie dust. This is a hard problem. But there are some guiding lines:
Forgetful Fred cannot lock a slot for ever. Forgetful Fred is the user that opens the reservation screen, picks a seat, then goes to lunch without finishing the transaction. If this is allowed, then the system will slowly 'leak' slots that aren't used
Database locks are too expensive to be held while waiting for user input.
Throughput can only be achieved with granular locks.
The business logic should not attempt concurent updates on correlated items.
Everything displayed to the user should be treated as 'tentative'.
The user interface should be prepared to handle update conflicts.
The update logic should always follow the same hierachy (eg. if the agreed update logic is Account->User->Event->Booking then a rogue transaction trying to update Booking->Event->User will cause deadlocks).
And as a matter of fact there is an approach that limits these concerns: workflow processing backed by transactional queues that leverage correlated items exclusive lock out. Not your everyday ASP task for sure, so I'd recommend you stick with what you know.
I created two (or more) threads to insert data in a table in database. When inserting, there is a field CreatedDateTime, that of course, stores the datetime of the record creation.
For one case, I want the threads to stay synchronized, so that their CreatedDateTime field will have exactly the same value. When testing with multi threading, usually I've got different milliseconds...
I want to test different scenarios in my system, such as:
1) conflicts inserting record exactly at the same time.
2) problems with ordering/selection of records.
3) Problems with database connection pooling.
4) Problems with multiple users (hundred) accessing at same time.
There may be other test cases I haven't listed here.
Yes, that's what happens. Even if by some freak of nature, your threads were to start at exactly the same time, they would soon get out of step simply because of resource contention between them (at a bare minimum, access to the DB table or DBMS server process).
If they stay mostly in step (i.e., never more than a few milliseconds out), just choose a different "resolution" for your CreatedDateTime field. Put it in to the nearest 10th of a second (or second) rather than millisecond. Or use fixed values in some other way.
Otherwise, just realize that this is perfectly normal behavior.
And, as pointed out by BC in a comment, you may misunderstand the use of the word "synchronized". It's used (in Java, I hope C# is similar) to ensure two threads don't access the same resource at the same time. In actuality, it almost guarantees that threads won't stay synchronized as you understand the term to mean (personally I think your definition is right in terms of English usage (things happening at the same time) but certain computer languages have suborned the definition for their own purposes).
If you're testing what happens when specific timestamps go into the database, you cannot rely on threads "behaving themselves" by being scheduled in a specific order and at specific times. You really need to dummy up the data somehow, otherwise it's like trying to nail jelly to a tree (or training a cat).
One solution is to not use things such as getCurrentTime() or now() but use a specific set of inserts which have known timestamps. Depending on your actual architecture, this may be difficult (for example, if you just call an API which itself gets the current timestamp to millisecond resolution).
If you control the actual SQL that's populating the timestamp column, you need to change that to use pre-calculated values rather the now() or its equivalents.
If you want to have the same timestamps on multiple rows being inserted; you should create a SQL thread which will do a multirow insert in one query which will allow you to get the same timestamps. Other than this, I agree with everyone else, you cannot get an exact timestamp at a huge resolution with multithreads unless you were to insert the timestamp as it is seen in the application and share that timestamp to be inserted. This of course, throws the caveat issues of threads out the window. It's like saying, I'm going to share this data, but I don't want to use mutexes because they stop the other thread from processing once it hits a lock().