I'm designing a service for sending out emails for our eCommerce site (order confirmations, alerts etc...)
The plan is to have a "SendEmail" method, that generates a chunk of XML representing the email to be sent, and sticks it on an Amazon SQS queue.
My web app(s) and other applications will use this to "send" emails.
I then require a way of checking the queue, and physically sending out the email messages.
(I know how I'm going to be dispatching emails)
I'm curious as to what the best way to "poll" the queue would be?
Should I create a windows service, and use something like Quartz.net to schedule it to check the queue every x number of minutes for example?
Is there a better way of doing this?
Sounds more like Amazon SNS is your huckleberry. (not really sure what that means, but I saw it in a movie once).
SQS is more of a "hopefully someone comes looking for this message at some point before it expires!" where SNS seems more like a "I need to make sure this gets to whoever needs it right away!"
It even includes email as a pre-built transport. (not even sure if that's the right word)
Amazon SNS provides a simple web services interface that can be used to create topics you want to notify applications (or people) about, subscribe clients to these topics, publish messages, and have these messages delivered over clients’ protocol of choice (i.e. HTTP, email, etc.). Amazon SNS delivers notifications to clients using a “push” mechanism that eliminates the need to periodically check or “poll” for new information and updates.
To be sure, they have the same freemium model as the rest of the services with limitations on email:
You can get started with Amazon SNS for free. Each month, Amazon SNS customers pay no charges for the first 100,000 Amazon SNS Requests, no charges for the first 100,000 Notifications over HTTP and no charges for the first 1,000 Notifications over Email.
If I were you, and if I REALLY wanted to make SQS work for this scenario, I would create a windows service that retrieves any messages from the queue every 10 minutes and then dispatches them.
If I could potentially use a different service, I would seriously consider using something like Postmark (Which I just found out about today thanks to a comment on Jeff Atwood’s blog post). You would just submit your email message info to their api, and they would do the rest. They even have a api for checking for bounced emails. I have never used this service, but I think it sounds great and would seriously consider using it in the future.
Looks like SES is what you need. http://aws.amazon.com/ses/ It is very inexpensive too compared to other similar services or building other systems.
You can use the AWS Beanstalk service to consume the tasks in the Queue.
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
Windows service is the best way, but you must not take it. To run background job in ASP.NET environment there is HostingEnvironment.QueueBackgroundWorkItem. A good starting point to implement continues polling (via long polling) is sqs-consumer.
Related
In want to use RabbitMQ to send events to a server from a mobile C# app. The user records a lot of events in the whole day (number of products manufactured, consumed water, ...) and they need to be delivered in a server to be processed. I understand that RabbitMQ creates a queue in the server but also, I would like to have a queue en in client side, in the mobile app. It is usual that in some parts of the factory Internet fails, so when the user records any event, it needs to be sent using the RabbitMQ client, but if Internet fails, it should remain in an "internal" queue, waiting to be sent in the next synchronization.
Which is the best approach for this problem? Does have RabbitMQ client library a feature for this purpose?
No RabbitMQ does not provide any such thing , typically for a user case like your it is best to use a local light weight database. You can go for something like SQLite.
Keep the data locally till it is synchronized and once done you may delete it from local.
I've been working on sending and receiving ActiveMQ messages between .net applications and everything work great when there is a active Consumer Connection.
However there are scenarios that I won't be able to have a active Consumer, example: web application consuming active messages.
I notice that any message not delivered to a consumer is place in a "Messages Enqueued" bucket.
Is there a way I can consume an already send and Enqueued message for a particular topic?
I am workign with Topic only, as shown on the example below
any tips from .net or even java might help me out thanks
Topics (except for durable topics) are inherently going to throw messages away if you don't have a consumer connected at the time the message is sent. That's just how topics work. My guess is that the web UI's column header is misleading- it probably should say "messages sent". You cannot retrieve them.
More on durable topics and queues here: http://activemq.apache.org/how-do-durable-queues-and-topics-work.html
We are creating a bulk SMS messages sending Web application using ASP.NET web Forms and C#.
There could be multiple logged-in users, sending bulk (1-5000) messages to their clients.
As per initial R&D we have selected to use Twilio's SMS API for this purpose.
But I am anxious about potential timeout issue while sending messages and saving response in loop, Also does Twilio accept multiple request instances?
Twilio has a limit or 1 message per second Then we can only send 3600 messages in one Hour, that would be a limitation.
Twilio provide Rest API but we are not familiar with MVC, so REST API idea has been dropped.
Please suggest other alternative, best practice and strategy to accomplish the task.
------------------UPDATE------------------------
Short Code Rest API Documentation
https://www.twilio.com/docs/api/rest/short-codes
Twilio Sending Documentation
https://www.twilio.com/docs/api/rest/sending-messages
Request Data From Twilio Documentation
https://www.twilio.com/docs/api/twiml/sms/twilio_request
With those volumes, there is no way you are going to be able to get by without using an SMS shortcode. Even if you could technically get it to work with a regular number, I suspect you would be shutdown as a spammer pumping out so many messages on a regular phone number.
A short code will let you send at at least 30 messages per second (1800/minute), and supposedly they can up that for you if you have a justified need.
https://www.twilio.com/sms/shortcodes
and this:
https://www.twilio.com/help/faq/sms/can-my-twilio-sms-messages-be-blacklisted-as-spam
Can my Twilio SMS messages be blacklisted as spam?
Recipient carriers always reserve the right to filter out messages
from certain numbers, and routinely do so to protect their users from
spam.
If you’ll be sending SMS messages as part of a bulk campaign (with 30
or more identical or similar messages going out within a few minutes),
we discourage you from using regular 10-digit numbers. Instead, we
highly recommend using an SMS short code.
I have a email queue with email to be send. A webservice calls a SOAP webservice that processes the queue one by one.
We send email using an external vendor using their REST API. My problem is that calls to this API can take from 0.1ms to 12s. We sent thousands of emails to customer that subscribe to our notices and it important that in each batch there's not to much delay between the first compared to the last in the queue (ideally they'd be sent in simultaneously).
I've complained to the vendor but as they suck I'm quite sure they will not do anything about this.
Can I somehow Threadify this process, instantiating simultaneous calls to the server? The server is also my web server so I can't use all the juice. How many threads is appropriate? Is this a good idea? What's the best way to generically manage these threads?
You shouldn't be creating threads within an ASP.Net application. If you have a large enough queue to warrant multithreading you should create a windows service to handle the queue.
I would queue the email in a database table and generate a separate windows service that reads from the table and spawns a thread for each email, up to some max thread limit. The database can also be used to capture throughput time.
You also should find out how many simultaneous web service requests your vendor can handle. BCC yourself on the emails to find out if simultaneous submissions on your end end up as a single-threaded transmission on their end. And perhaps start shopping for an alternative to this vendor (you did say they suck).
If you want to get fancy and offload the effort from your own server, you send a batch of emails to a cloud service (Amazon Web Services, Microsoft Azure, or Google App Server) and spawn a process on the cloud to spray the emails to your vendor simultaneously.
You can also send the emails directly from the cloud, at least you can with Amazon. They provide a default limit, but then here's a link on how to remove the limit: http://aws.amazon.com/contact-us/ec2-email-limit-request/.
I have had some success with ThreadPool.QueueUserWorkItem() for a ASP.NET app. You can google for some usage examples.
There is no need to spawn threads yourself. The class generated by visual studio to access a web service already contains asynchronous methods. For each webservice call Foo, you will see that there is a BeginFoo and EndFoo method. The BeginFoo method will immediately return an IAsyncResult object while the webservice call is done in another thread.
See this MSDN topic for more information on how to use IAsyncResult.
What are some ideas (using .NET and SQL 2005) for implementing a service that sends emails? The emails are to be data-driven. The date and time an email is to be sent is a field in a table.
I have built several high volume email notification services used to send data driven emails in the past. A few recommendations:
Look into using a high quality email service provider that specializes in managing bounces, unsubscribes, isp and black list management, etc. If sending email is critical to your business, but not your main business it will be worth it. Most will have an api for sending templated messages, click tracking, open rates and will have provide triggers etc.
Look into the SQL Server Service Broker to queue the actual messages, otherwise you can consider Microsoft Message Queuing Services. There is no need to build your own queuing service. We spent too much time dealing with queing infrastructure code when this was already solved.
Develop a flexible set of events on your business tier to allow for the triggering of such messages and put them in your queue asynchronously, this will save you alot of grief in the long run as opposed to polling on the DB or hacking it in with Database triggers.
You can use triggers to send emails on UPDATE/DELETE/INSERT. The triggers can be implemented with .Net, just send mails from there using the classes in System.Net.Mail namespace.
Here is a good article how to implement CLR (.Net) triggers in .Net.
For a light-weight SMPT server, and to minimize the delays, you can use the one, recommended in Kenny's answer.
Thanks everyone for the feedback. For simplicity's sake I've started out with a SP that looks up the reminders to be sent and uses sp_ send_ dbmail (SQL Database Mail) to send the emails. This runs on a job every minute. I update the record to indicate the reminder was sent with the MailItemId sent back from sp_ send_ dbmail. The volume of reminders expected is worst case in the 10^2 range per day.
I'd love to hear feedback about any shortcomings people think this solution may have.
By the way, I can't believe Vista doesn't come with a local SMTP server! Luckily Google is more generous, I used Gmail's server for testing.
Usually, I just spin up a process such as http://caspian.dotconf.net/menu/Software/SendEmail/
I was going to suggest SQL Server Notification Services, which will handle the job nicely. But I see that's been dropped from SQL Server 2008, so you probably don't want to go there.
Data Driven SSRS Subscriptions? Just a thought.