Below are my environment settings
SQL Server 2008 R2
ASP.NET MVC
Entity Framework
In my application I want to do post processing of transaction after they are inserted in database. So basically I want a separate application to be listening to database events, and whenever a new row is created in table Orders, do some post processing (calling another webservice to process orders) on it.
I checked with EF and it seems it doesn't provide any such feature.
Few other requirements
at this moment i am not looking for solution involving polling as it will increase db load.
Want some kind of event to get triggered when a row is inserted.
What would be the best way to achieve the same?
Create a T-SQL trigger on that table that inserts a small "command" entry into a separate "Command" table, with all the relevant information (e.g. OrderId and possibly others).
Then have your stand-alone application check that "Command" table on a regular basis (scheduled to check every 10 minutes - or whatever makes sense to you).
Based on the data read from the "Command" table, your application can then do any post-processing necessary, and update the underlying database table with the new information.
Warning: DO NOT put all this post-processing logic directly into the trigger! That would be a cardinal sin - a trigger should never do heavy and lengthy processing since that blocks the currently running transaction and kills your database's system performance for sure!
You can make use of a Service bus, something like MassTransit. Configure it to make use of it's Subscription service.
Once the order has been saved, raise an event on the bus, and have another service listen for this event. Once the other service picks up the event, you can continue processing the order.
This is the MassTransit main page
http://masstransit-project.com/
And this is the configuration for the subscription service
http://docs.masstransit-project.com/en/latest/overview/publishing.html
Related
I am developing a program in WPF.Net, and I need to know when somebody makes a change over any table of the database.
The idea is receive a event from the database when it was changed. I was reading a lot of articles but I can't find a method to resolve my problem.
Kind Regards
The best solution is to use a message queue. After your app commits a change to the database, the app also publishes a message on the message queue. Other clients then just wait for notifications on that message queue.
There are a few other common solutions, but all of them have disadvantages.
Polling. If a client is interested in recent changes, they run a query searching for new data every N seconds.
The downside is you have to keep polling even during times when there are no changes. You might have to poll very frequently, depending on how promptly you need to notice the changes. This adds to database load just to support the polling queries.
Also it costs more if you have many clients all polling for queries. In one system I supported, the database was struggling to process 30,000 queries per second just for clients running polling.
Change Data Capture. Using the binary log as a de facto message queue, because it records all the changes. Use a client tool such as Debezium, or write your own binlog tail client (this is a lot of work).
The downside is the binlog records all changes, not just those you want to be notified about. You have to filter it somehow. Also you have to learn how to use Debezium or equivalent tool.
Triggers. Write a trigger on the table that invokes a UDF to post notification outside the database. This is a bad idea, because the trigger executes when your insert/update/delete executes, not when the transaction commits. Clients could be notified of changes before the changes are committed, so if they go query the database right after they get the notification, the change is not visible to them yet.
Also a disadvantage because it requires you install a UDF extension in MySQL Server. MySQL doesn't normally have any way of posting an external notification.
I'm not a C# developer so I can't suggest specific code. But the general methods above are similar regardless of which language the app is written in.
I don't think this is possible with MySQL, DBs like MondgoDB have this sort of feature.
You may like to use the method described in this answer.
Essentially have date/time fields on rows where you can pull data since a certain date time. Or you could use a CQRS/Event stratagem and maybe use a message queue.
How can i get notification on every new insertion in sql table, i want to get the top recent entries to show in Grid View against a user, the way i was doing this is calling ajax function using JavaScript on setInterval/setTimeout, but the problem is this its hitting the database in mentioned time interval, and i just want to load the entries (call the function) only when there is new insertion in Database just to minimize the Database Hits.
Thanks in Advance :)
This is possible using triggers and extended stored procedures that can access processes outside the SQL application. I would say though that this is not the right architecture, and extended stored procedures have additional concerns. It would be preferable to use which ever application layer is writing to the database to perform the necessary communications signalling that a new message has arrived.
I am currently working on an asp.net web application and have been interested in implementing a SQL dependency into a part of it. There is a process where users will update some rows in a SQL server database table. A windows service hosted elsewhere will receive a sqldependency query notification, and process the data. What I want is for the web application to post back once it detects that the data processing has completed. The way I'm trying to do this at the moment is to create a SQLdependency with a query that monitors the relevant rows. The only notification I care about is when the query no longer returns rows (meaning the data has finished processing). Is there any clean way to make this work? My current implementation receives notifications for every change made, which come as the data is processed.
I believe you can use a field on a table to track the changes. But you make the field an int (this will be similar to how ##Trancount works). The premise is this:
Each service that begins updates will increase that field by one.
As they finish, they decrease that field by one (and your web app can be the one controlling that aspect)
(NB) On error the services need to be aware to decrease the field by one.
Your SqlDependency will have a query which monitors this table, selecting that field when it changes. When it is zero, it means that no more rows are returned.
My current situation is that I have an application that needs to be notified when new data arrives in a database table. The data is coming from an external source (that I have no control over--this is this only integration option). When new data arrives, my application needs to take certain actions--basically query for the new data, handle it, insert the result into a local table, etc.
I want to avoid polling if possible, as the data is expected to be handled in real time. That said, making sure no data ever gets missed is the #1 priority.
My questions:
Is SqlDependency generally considered reliable?
Do I need to be concerned about race conditions, e.g. I am handling one change when another arrives?
What happens when the database gets rebooted? Will my app recover and start receiving changes again, or will I need a fail-safe timer of some sort that will resubscribe to notifications periodically?
Most of the articles I have read on the topic address SQL Server 2005. I am using SQL Server 2008 R2. Is there a newer technique that is preferred over SqlDependency?
(Edit)Also, What if the application goes down? I guess I would have to query for missed data on start up?
1) Yes, I consider it reliable as in it does correctly the purpose was designed to do (cache invalidation)
2) No. This is why you can only subscribe by issuing a query, this ensures that there is no race between the fetching of the data and new updates notifying
3) Database (or instance) restart signals all pending query notifications with an SqlNotificationInfo value of Restart. Read how SqlDependency and is based on Query Notification for a better understanding. As SqlDependency keeps an open connection to the database all the time, a database unavailability will be detected by SqlDependency even before any explicit query notification
4) No. More on this further down...
5) There is no 'missed data'. Query Notification (and hence SqlDependency) never notify you about what data changed. It only notifies you that it changed. You are always supposed to go back and read all the data back to
see what had changed (and I refer you back to question/answer no. 2). A newly started application had not yet queried the data to begin with, so there is no change to be notified of. Only after it has first queried the data can it receive a notification.
From the description of your problem I'm not convinced you need query notifications. It seems to me that you want to act on any change, not matter when it happened, even if your application was not running. This is certainly not cache invalidation, it is change tracking. Therefore you need to deploy a change tracking technology, like Change Data Capture or Change Tracking, both of which are SQL Server 2008 and later only (not available in SQL Server 2005). With SQL Server 2005 is not uncommon to deploy a trigger and queue a message for Service Broker to handle the same problem you are trying to handle (detect changes, react to each row of new data).
Coming at it from the point of view of a .net developer to just wants to use it for cache invalidation it has been a real pain and isn't completely reliable.
Set up and troubleshooting has been particularly painful, we get it working okay in one environment but then it doesn't work in another. Figuring out why has been difficult and time-consuming.
Even when it is all running it isn't completely reliable. SQL Server can drop notifications if it under heavy load and there are known issues with it restarting and notifications not resuming: http://connect.microsoft.com/SQLServer/feedback/details/543921/sqldependency-incorrect-behaviour-after-sql-server-restarts.
I would avoided if there is an alternative technology the does what you want and is less troublesome.
I have a table with a heavy load(many inserts/updates/deletes) in a SQL2005 database. I'd like to do some post processing for all these changes in as close to real time as possible(asynchronously so as not to lock the table in any way). I've looked a number of possible solutions but just can't seem to find that one neat solution that feels right.
The kind of post processing is fairly heavy as well, so much so that the windows listener service is actually going to pass the processing over to a number of machines. However this part of the application is already up and running, completetly asynchronous, and not what I need help with - I just wanted to mention this simply because it affects the design decision in that we couldn't just load up some CLR object in the DB to complete the processing.
So, The simple problem remains: data changes in a table, I want to do some processing in c# code on a remote server.
At present we've come up with using a sql trigger, which executes "xp_cmdshell" to lauch an exe which raises an event which the windows service is listening for. This just feels bad.
However, other solutions I've looked at online feel rather convoluted too. For instance setting up SQLCacheDependancy also involves having to setup Service broker. Another possible solution is to use a CLR trigger, which can call a webservice, but this has so many warnings online about it being a bad way to go about it, especially when performance is critical.
Idealy we wouldn't depnd on the table changes but would rather intercept the call inside our application and notify the service from there, unfortunately though we have some legacy applications making changes to the data too, and monitoring the table is the only centralised place at the moment.
Any help would be most appreciated.
Summary:
Need to respond to table data changes in real time
Performance is critical
High volume of traffic is expected
Polling and scheduled tasks are not an option(or real time)
Implementing service broker too big (but might be only solution?)
CLR code is not yet ruled out, but needs to be perfomant if suggested
Listener / monitor may be remote machine(likely to be same phyisical network)
You really don't have that many ways to detect changes in SQL 2005. You already listed most of them.
Query Notifications. This is the technology that powers SqlDependency and its derivatives, you can read more details on The Mysterious Notification. But QN is designed to invalidate results, not to pro-actively notify change content. You will only know that the table has changes, without knowing what changed. On a busy system this will not work, as the notifications will come pretty much continously.
Log reading. This is what transactional replication uses and is the least intrusive way to detect changes. Unfortunately is only available to internal components. Even if you manage to understand the log format, the problem is that you need support from the engine to mark the log as 'in use' until you read it, or it may be overwritten. Only transactional replication can do this sort of special marking.
Data compare. Rely on timestamp columns to detect changes. Is also pull based, quite intrussive and has problems detecting deletes.
Application Layer. This is the best option in theory, unless there are changes occuring to the data outside the scope of the application, in which case it crumbles. In practice there are always changes occuring outside the scope of the application.
Triggers. Ultimately, this is the only viable option. All change mechanisms based on triggers work the same way, they queue up the change notification to a component that monitors the queue.
There are always suggestions to do a tightly coupled, synchronous notification (via xp_cmdshell, xp_olecreate, CLR, notify with WCF, you name it), but all these schemes fail in practice because they are fundamentally flawed:
- they do not account for transaction consistency and rollbacks
- they introduce availability dependencies (the OLTP system cannot proceed unless the notified component is online)
- they perform horribly as each DML operation has to wait for an RPC call of some form to complete
If the triggers do not actually actively notify the listeners, but only queue up the notifications, there is a problem in monitoring the notifications queue (when I say 'queue', I mean any table that acts as a queue). Monitoring implies pulling for new entries in the queue, which means balancing the frequency of checks correctly with the load of changes, and reacting to load spikes. This is not trivial at all, actually is very difficult. However, there is one statement in SQL server that has the semantics to block, without pulling, until changes become available: WAITFOR(RECEIVE). That means Service Broker. You mentioned SSB several times in your post, but you are, rightfuly so, scared of deploying it because of the big unknown. But the reality is that it is, by far, the best fit for the task you described.
You do not have to deploy a full SSB architecture, where the notificaition is delivered all the way to the remote service (that would require a remote SQL instance anyway, even an Express one). All you need to accomplice is to decouple the moment when the change is detected (the DML trigger) from the moment when the notification is delivered (after the change is commited). For this all you need is a local SSB queue and service. In the trigger you SEND a change notification to the local service. After the original DML transaction commits, the service procedure activates and delivers the notification, using CLR for instance. You can see an example of something similar to this at Asynchronous T-SQL.
If you go down that path there are some tricks you'll need to learn to achieve high troughput and you must understant the concept of ordered delivery of messages in SSB. I reommend you read these links:
Reusing Conversations
Writing Service Broker Procedures
SQL Connections 2007 Demo
About the means to detect changes, SQL 2008 apparently adds new options: Change Data Capture and Change Tracking. I emphasizes 'apparently', since they are not really new technologies. CDC uses log reader and is based on the existing Transactional replication mechanisms. CT uses triggers and is very similar to existing Merge replication mechanisms. They are both intended for occasionally connected systems that need to sync up and hence not appropiate for real-time change notification. They can populate the change tables, but you are left with the task to monitor these tables for changes, which is exactly from where you started.
This could be done in many ways. below method is simple since you dont want to use CLR triggers and sqlcmd options.
Instead of using CLR triggers you can create the normal insert trigger which updates the dedicated tracking table on each insert.
And develop dedicated window service which actively polls on the tracking table and update the remote service if there is any change in the data and set the status in tracking table to done (so it wont be picked again)..
EDIT:
I think Microsoft sync services for ADO.Net can work for you. Check out the below links. It may help you
How to: Use SQL Server Change Tracking - sql server 2008
Use a Custom Change Tracking System - below sql server 2008
In similar circumstances we are using CLR trigger that is writing messages to the queue (MSMQ). Service written in C# is monitoring the queue and doing post-processing.
In our case it is all done on the same server, but you can send those messages directly to the remote queue, on a different machine, totally bypassing "local listener".
The code called from trigger looks like this:
public static void SendMsmqMessage(string queueName, string data)
{
//Define the queue path based on the input parameter.
string QueuePath = String.Format(".\\private$\\{0}", queueName);
try
{
if (!MessageQueue.Exists(QueuePath))
MessageQueue.Create(QueuePath);
//Open the queue with the Send access mode
MessageQueue MSMQueue = new MessageQueue(QueuePath, QueueAccessMode.Send);
//Define the queue message formatting and create message
BinaryMessageFormatter MessageFormatter = new BinaryMessageFormatter();
Message MSMQMessage = new Message(data, MessageFormatter);
MSMQueue.Send(MSMQMessage);
}
catch (Exception x)
{
// async logging: gotta return from the trigger ASAP
System.Threading.ThreadPool.QueueUserWorkItem(new WaitCallback(LogException), x);
}
}
Since you said there're many inserts running on that table, a batch processing could fit better.
Why did just create a scheduled job, which handle new data identified by a flag column, and process data in large chunks?
Use the typical trigger to fire a CLR on the database. This CLR will only start a program remotely using the Win32_Process Class:
http://motevich.blogspot.com/2007/11/execute-program-on-remote-computer.html