I have the need from an asp.net web site to send out many SMS messages at once, and also poll a POP3 account for an incoming mail, and then SMS that out to many recipients, one at a time.
The way I am thinking of doing this is a windows service that would connect to my sql back-end to see if there are SMS messages to be sent out, like every 10-20 seconds or so. If so, get all the messages in a list, delete them from the table, and then proceed to send them out.
Same way with the pop account.
Any ideas on how to best provide this service without causing blocking in the asp.net web page when it is kicked off (e.g. messages added to sql server)?
Platform is windows server 2003 R2, sql 2008 standard, asp.net 3.5 SP1.
Thanks for your advice.
We have implemented similar scenarios using SQL Server service broker's Queueing mechanism. The idea is that every inserted SMS record is caught by a trigger which inserts a message containing the SmsID into the service broker Queue.
You then need a stored procedure which receives messages from the Queue. If there are no messages, your procedure will run until the next entry is inserted. That's OK, since it does not take up resources to listen to the Queue.
Next you'll need a Windows service who continuously (recursively) calls the STP, assembles the SMS and sends it.
The Advantage of the Service Broker Queue over a flag in a table is thread safety. This way you could have as many instances of your Service as you want w/o having to worry too much about concurrency issues.
You can find a nice Service Broker tutoial here: http://www.developer.com/db/article.php/3640771
Instead of using an Sql Server for the queuing you could use MSMQ (Microsoft Message Queuing) for this.
MSMQ is quite easy to set up and once it is up and running it is more scalable than Sql Server.
So what you could do was to setup a new queue in MSMQ that would receive the messages you wanted to send. The message would normally be some sort of Message object that describe the message, the sender and the recipient.
Then you would either setup a service that would poll the queue at a regular interval or you could setup MSMQ to start a class of your choice each time a new Message was sent to the queue.
If you need a log of the messages you could have the service / sender object write to a log in sql server when the message was sent.
Related
In want to use RabbitMQ to send events to a server from a mobile C# app. The user records a lot of events in the whole day (number of products manufactured, consumed water, ...) and they need to be delivered in a server to be processed. I understand that RabbitMQ creates a queue in the server but also, I would like to have a queue en in client side, in the mobile app. It is usual that in some parts of the factory Internet fails, so when the user records any event, it needs to be sent using the RabbitMQ client, but if Internet fails, it should remain in an "internal" queue, waiting to be sent in the next synchronization.
Which is the best approach for this problem? Does have RabbitMQ client library a feature for this purpose?
No RabbitMQ does not provide any such thing , typically for a user case like your it is best to use a local light weight database. You can go for something like SQLite.
Keep the data locally till it is synchronized and once done you may delete it from local.
I am currently using a SqlDependency with a SQL Server 2012 Service Broker and I want to be able to have two servers configured both listening to the service broker and pull off the queue but message should only be pulled off the queue once total. Each machine should try and pull down what it can but if too many are coming in it should share a balance in pulling in what it can. Right now I start two instances of the program and both are listening. Once a new message is added they both pull off the same message off the queue and run the code.
Is SqlDependency not the solution to what I want to do? What is the better solution to something like this?
Once a new message is added they both pull off the same message off the queue and run the code
The behavior you describe is how SQLDependency is designed to work. If there are multiple listeners, all listeners are notified. For example, you can see this described in the SignalR SQL Backplane documentation
Notice how all VMs receive notification from SQL Server, including the VM that initiated the update.
If you want to distribute SQL Notifications across a pool of worker VMs, you need a way to share state. Note that the SQL Notification is only an indication that something changed and doesn't indicate what changed. One approach is to add a table to the database to act as a queue of jobs or actions. Subscribers can query this queue on each notification and claim the action by updating or deleting from this table. (Appropriate locks would have to be configured on the table)
Alternatively, you can do this using other tools for shared state, such as a message queue (eg. RabbitMQ), or distributed cache (eg. Redis)
You don't need SQL Notifications or SQLDependency. Each instance can execute:
WAITFOR(
RECEIVE TOP(1) * FROM {NameOfQueue}
), TIMEOUT #timeoutvalue;
This command will WAIT, leaving the connection open, until either a message is available or the timeout has occurred. On the timeout you receive no message so just connect and try again.
Each message can only be RECEIVED by a single process. Internally the row in the Server Broker queue is locked, and other readers will READPAST locked rows.
Because the SQL can be a little bit tricky, I've written what I think is a helpful wrapper class that you are free to use.
I need to create a windows service that processes a job queue.
A trigger will fill up the queue, and the windows service will process that queue.
I know how to create a windows service (using a timer), but I'm not sure about how tell the trigger (CLR project) to fill the queue, and how to process it.
I'm new in C#, so any example, ideas, guidances are wellcome
Thank you
Sounds like you need a MQ (message queueing) system. Try looking into this (RabbitMQ) as a starting point, then update your question with specifics if need be.
http://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html
Here is Microsoft's implementation of it (MSMQ):
http://msdn.microsoft.com/en-us/library/ms711472(v=vs.85).aspx
Thank you Bill Sambrone.
I found SQL Server Service Broker (http://technet.microsoft.com/en-us/library/ms345108%28v=sql.90%29.aspx)
It is a service that uses queues to send messages that are guaranteed to arrive to destiny, it can send messages between databases, remote databases, and also to external apps. I'm sending the messages to my c# external app. And the best is that it comes bundled inside SQL server.
I need to implement a queuing mechanism for WCF service requests. The service will be called by clients in a one-way manner. These request messages should be stored in a SQL Server database and a Windows Service queues the messages. The time at which the requests are processed will be configurable. If there happens error in processing the message, it need to be retried up to 100 times and if still fails it need to be terminated.
Also there should be a mechanism to monitor the number of transaction made on a day and number of failures.
QUESTIONS
If I were using MSMQ, clients could have forwarded the message to queue without knowing the service endpoint. But I am using SQL Server to store the request messages. How the clients can put the requests to SQL Server?
Is the solution feasible? Do we have any article/book that explains how to implement the above?
What are the steps to prevent service and client reaching faulted state in this scenario?
What is the best method to store incoming message to database?
What is the best method to implement retry mechanism? Anything already exist so that I don't have to reinvent the wheel?
Is there any book/article that explains this implementation?
NOTES
Content of the message will be complex XML. For example Travel expense items of an employee or a list of employees.
READING
Logging WCF Request to Database
Guaranteed processing of data in WCF service
MSMQ vs. SQL Server Service Broker
Is it possible to persist and then forward WCF messages to destination services?
WCF 4 Routing Service - protocol bridging issue
https://softwareengineering.stackexchange.com/questions/134605/designing-a-scalable-and-robust-retry-mechanism
Integrating SQL Service Broker and NServiceBus
Can a subscriber also publish/send message in NServiceBus?
I'm a DBA, so that flavors my my response, but here's what I'd do:
If you're using SQL 2005+, use Service Broker to store the messages
in the database rather than storing them in a table. You get a
queueing mechanism with this, so you can get rid of MSMQ. You'll also have a table, but it's just going to store the conversation handle (essentially, a pointer to the message) along with how many times it attempted this message. Lastly, you'll want some sort of a "dead letter box" where messages that reach your retry threshold go.
In your message processing code, do the following:
Begin a transaction
Receive a message off of the queue
If the retry count is greater than the threshold, move it to the dead letter box and commit
Increment the counter on the table for this message
Process the message
If the processing succeeded, commit the transaction
If the processing failed, put a new message on the queue with the same contents and then commit the transaction
Notice that there aren't any planned rollbacks. Rollbacks in Service Broker can be bad; if you rollback 5 times without a successful receive, the queue will become disabled for both enqueuing and dequeuing. But you still want to have transactions for the case when your message processor dies in the middle of processing (i.e. the server crashes).
1. If I were using MSMQ, clients could have forwarded the message to queue without knowing the service endpoint.
Yes - but they would need to know the MSMQ endpoint in order to send their message to the queue.....
But I am using SQL Server to store the request messages. How the clients can put the requests to SQL Server?
The clients won't put their requests into SQL Server - that's what the service on the server will do. The client just call a service method, and the code in there will store the request into the SQL Server table.
2. Is the solution feasible? Do we have any article/book that explains how to implement the above?
Sure, I don't see any big issue. The only point unclear to me right now is: how will the clients know their results?? Do they need to go get results from another service or something??
3. What are the steps to prevent service and client reaching faulted state in this scenario?
As always - just make sure your service code catches all exceptions and either handles them internally, or returns interoperable SOAP faults instead of .NET exceptions.
It sounds like what you want to do is similar to this:
In this case you can use netMsmqBinding between your service and your service consumers.
The only thing you won't get out of the box is the retrying. However if you make the queue transactional then this functionality can be implemented in your service code.
If there is a failure in your dequeue operation the message will not be removed from the queue. It will therefore be available for further dequeue attempts.
However, you would need to implement retry attempt threshold code which fails a message after a certain number of attempts.
I would suggest a different approach to the ones suggested here. If you are able to, I would consider the introduction of a messaging framework such as NServiceBus. It satifies many of the requirements that you have right out of the box. Let me try and address this in context of your requirements.
The service will be called by clients in a one-way manner.
All communication between endpoints in NServiceBus is one way. The underlying transport NServiceBus uses is MSMQ, so much like your WCF approach, your client is communicating with queues, rather than specific service endpoints.
These request messages should be stored in a SQL Server database and a Windows Service queues the messages.
If you wanted to store your request messages in a database then you can configure NServiceBus to forward all messages sent to your request processing endpoint to another "audit" queue, which you can use to persist to the database. This has the added benefit of separating your application logic from your auditing implementation.
The time at which the requests are processed will be configurable.
NServiceBus allows you to defer when a mesage is sent. Normally a message is sent via the Send method of a Bus instance - Bus.Send(msg). You can use The Defer method to send the message some time in the future eg. Bus.Defer(DateTime.Now.AddDays(1), msg); There's nothing more you really have to do, NserviceBus will handle the message once the specified time has been reached.
If there happens error in processing the message, it need to be retried up to 100 times and if still fails it need to be terminated.
By default, NServiceBus will enlist your message in a transaction as soon as your message leaves the queue. This ensures that in the event of failure that the message is rolled back to the originating queue. In such an event, NServiceBus will automatically try to reprocess the message a configurable number of times. The default being 5. You can of course set this to whatever you want, although I am not sure why you would want to set this to 100. At any rate, NServiceBus uses this setting to stop an endless loop of automatic retries. Once the limit has been reached the message is sent to an error queue where it sits until you fix whatever issues caused the exception or until you decide to push the message back to the queue for processing. Either way, you are assured that the message is never lost.
Also there should be a mechanism to monitor the number of transaction made on a day and number of failures.
The beauty of using MSMQ as the transport is that performance monitoring can be a achieved at a infrastructure level. How your applications perform, can be measured by how long they sit in the queue. NServiceBus comes with performance monitors that track the length of time a message is in the queue and you can also add perf mons that come built into windows to track other activity. To monitor errors, all you need to do is check the number of messages in the error queue.
One of the main features of NServiceBus is reliability. WCF will only do so much for you, and then you are on your own. That's a lot of code, complexity and frankly hugely error prone. The things I have described here are all standard features of NServiceBus and I have barely scratched the surface with all the other things that you can do with it. I recommend you check it out.
How do you 'verify' that a message sent using MSMQ to a private local queue was actually delivered? I'm especially thinking of a scenario where the listener (a C# service in my case) is not running and therefore delivery can't be successful.
You can only guarantee that it can get to the queue without taking extra steps. To deal with the "not running receiver" scenario, you would need to code the receiver to send a message back to the server when it processes the message. The original sender would be responsible for tracking the sent messages and verifying that the client has recieved them.
That's one of the decisions you should be taking when deciding whether or not to use MSMQ as opposed to a remoting or a web service scenario. For example, we had a project used for notifying all of our retail locations when an emergency occurred (such as a product recall/food safety issue.) We needed to know immediately if the store's listener was running so we chose remoting, and when the sender received an error indicating one of the listeners was not listenting, we would need to pick up the phone and call.
Just something to keep in mind.
Edit - clarification
I was really giving out two options above.
Code the client to send a message back to the sender when it receives a message.
Use another option, such as remoting, where you can detect if the client is running and receives the message.
It's always sent to the queue.
If your service isn't running to receive it, it just sits there, waiting patiently, until someone receives it.
You know it's been sent to the queue because .Send() returns without crashing.
You can probably pull this info out using administrative queues
When you send a message you can specify the AcknowledgeType which will allow you find out (through positive or negative acknowledgement) whether the message reached the queue and/or was received from the queue. Acknowledgements are sent as messages, by MSMQ, to the AdministrativeQueue so make sure you assign that property on the Message object.
You can check the administrative queue for acknowledgements by correlation ID which is ID of the original message.