Develop Reference

RabbitMQ publish a message with REST API

I have an application written in Python (PIKA) and C# (official NuGet package). Those applications are publishing new messages into RabbitMQ queues.
Until now, I used this syntax to publish a new message into the queue:
model.BasicPublish(exchange, routingKey, basicProperties, body);
I found that BasicPublish function always returns with success. I also read in RabbitMQ documentation that in case of broker destroyed, the messages that didn't send yet will be removed without sending it to RabbitMQ.
I want to avoid the loss of messages. So, I found 3 options to submit those messages to publish:
Transaction - Very slow.
Confirmation - I found it tricky to implement in a multi-threaded environment.
with REST API - What do you think about that?
I think that it will be ideal for me yo use REST API for inserting messages into queues.
The Question:
The way that I found to send a message with API is to send POST message to this endpoint:
http://localhost:15672/api/exchanges/vhost/amq.default/publish
As you can see, this port (15672) belongs to the RabbitMQ management system.
Is this the right way to use RabbitMQ with REST API?
Can I trust the RabbitMQ management system in the production environment?
Can you recommend an alternative to REST API that will accept to message enqueue immediately after insertion (blocking)?

No, don't use the HTTP API. It is not intended for production use for publishing or consuming messages.
You must use publisher confirms. The technique described in this tutorial applies to the .NET client library as well.
You could also investigate libraries written on top of the official .NET library that may correctly implement publisher confirms for you. EasyNetQ is one such library.
Another good resource to read with concern to 100% reliability is this blog post.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.

.NET architecture design issue

Im just starting to work on a particular piece of development.
We have a .NET WCF application, MySql/EF DAL/ORM, that is called by a threaded job scheduler that pulls data from one client, stores it in our DB and passes the latest data to a another client and vice versa.
So to think about it as messages,
ClientB sends an order to ClientA through our system which transforms the order into a readable format for ClientA.
ClientA then can send messages to the ClientB through our system to say stuff like "your order is shipped" or "your order is late".
I need to take these messages and relay them onto ClientB but I want it to be transactional and for us to have full control over failed messages etc.
My current thoughts are, for simplicity sake, to have a OrderMessages table in our DB which receives messages, with a state of "Ready" which can then be processed by a factory and forwarded to the relevant client using a configuration stored against the clients.
Sorry for this being all over the place, but hopefully I've explained what im trying to do :/
Neil

Your proposed architecture is a classic queue table pattern. Remus Rusanu is the canonical resource for building such a thing with SQL Server. The ideas apply to other databases as well.
There is nothing wrong with this architecture. Note, that in case of an error when messaging a client you cannot know whether the message was received or not. There is no 100% solution for this problem. It is the Two Generals Problem.
If you make the clients pull directly from the database you can avoid this problem. Clients can use their own transactions in this case.

Consider leveraging a message platform for publishers and subscribers.
Specifically, consider using a hub and spoke pattern.
Also, BizTalk specializes in workflows across distributed systems.
Also consider the effort involved:
Transactions (short and long)
Error handling
Expected message formats
Orchestrations

How to scale out signalr to a large number of users

I have read a lot on scaling out in signalr and the favourites seem to be those mentioned in :
http://www.asp.net/signalr/overview/signalr-20/performance-and-scaling/scaleout-in-signalr
Namely the following service buses:
- SQL
- Redis
- Azure
The problem is is even stated in the text however:
"Using a backplane, the maximum message throughput is lower than it is when clients talk directly to a single server node. That's because the backplane forwards every message to every node, so the backplane can become a bottleneck."
I am creating...wait for it... YEP! A chat application. And I want to be able to scale it out to MILLIONS of users. Regardless of whether I make it big (ha!) or not, I plan on documenting the step by step process. Now I have most of the app ready, and I am wondering about this scale out issue. I watched this, highly useful video:
http://channel9.msdn.com/Events/Build/2013/3-502
Skip to 55 minutes. "Custom scale-out". And the other ideas such as filtering the message bus.
Now hopefully you're excited and not contemplating suicide over the boredom I am ushering unto you...
My idea is to do as follows:
- per popular rooms give a single server
- each room therefore can easily cope with the traffic and signalr can work nicely broadcasting to the clients and storing the message log to a GROUPS server (ie holds all group messaging per group)
- Then private messaging will need to either use a backplane or server push
- the user connections will therefore need to be updated in a sql server DB (easy enough) and the data posted via ajax rather than signalr
However, I want to explore all options. (Please post any better ideas if you have them) I want to also try testing REDIS for the private messaging. WHY?!! Because what if I want the users to be able to have private messaging groups... and users 1,2 and 3 are all highly annoying and are on servers 1,2 and 3. (Ah you little ...!) For better performance though, I will want to implement a Redis message filter to only send to the servers with the clients on them!
So, what exactly am I looking for? Basically I need resources. I can't find any useful Redis message bus examples (asp.net example has no filter. yes, I can add the AddResolverblabla line! :) )
I also need examples of the following:
- server to server ajax post: I am a server noobie!
- a load balancer example to specify a certain room per chat room (or just some page)
- how many messages can the Redis message bus handle? Will it easily be a bottle neck even with the filter? I cant find any example of performances WITH a filter
Finally I need your brains! ;) If you are sat there thinking there's a better way, please let me know.
Many thanks to all who read this essay, I look forward to your replies. (Please up-vote this if you find it useful! There's a lot of forums with similar questions, but no proper answers)

I plan to start answering this as I find documentation. Hopefully more will join!
1.
How to define a connection string to a SQL Server 2008 database?
2.
SQL Server filter:
http://msdn.microsoft.com/en-us/library/windowsazure/microsoft.servicebus.messaging.sqlfilter.aspx
3. web farm
http://weblogs.asp.net/scottgu/archive/2010/09/08/introducing-the-microsoft-web-farm-framework.aspx
4. WF tutorials:
http://www.asp.net/web-forms/tutorials/deployment/configuring-server-environments-for-web-deployment/creating-a-server-farm-with-the-web-farm-framework
3/12/2014
A possible better solution is to use memecache - what facebook is based on
still need to find whether you can specify: use signalr or use redis message bus.
still need to find redis filter tutorials
14/03
GlobalHost.DependencyResolver.UseRedis("server", port, "password", "AppName");
Defines the servers to use redis. Need a filter
6/10/2014
After more research on scaling out, a possibleanswer is to not think of servers as a web of communications, but self-contained. The server uses sends to update the DB and using a timer you can get all the required information (messages etc) each loop from the DB for the users CURRENTLY logged in to that server. As such, it will scale out much easier. Not cheaply however.

C# Mail Server?

I'm trying to understand exactly what it is I need to be implementing and am just after a little guidance. I know very little about email standards such as POP3, IMAP, SMTP which is probably what's making the getting started difficult.
What'd I'd like to do consists of a couple of things:
Be notified of the arrival of an email at a specific custom address. e.g. hi.me#myapp.com.
Handle multiple addresses per user.
Be able to process this email and either delete it, leave it and be able to display in a web email client or forward it on based on various rules.
I've seen a few SMTP libraries and a POP3 one, but I'm not 100% sure what I need to do this. I'm guessing the best way would be to write a full C# mail server - but I'm hoping for a little guidance/suggestion.
Thanks

If you really want to build a mailserver from scratch, the protocol specifications are all available as RFCs.
For IMAP that would be RFC-3501 and for SMTP RFC-2821, you can google for the rest.
I've been working on a C# SMTP server in my (very limited) spare time, mostly out of frustration with existing MTAs and as an exercise in C# server programming. I can tell you that there's a bit more to it than you might think. So if you want fast results, I'd seriously consider hooking into an existing MTA/mailserver setup.
Depending on how fast you want to process messages, it may be sufficient to poll one or more mailboxes. You could process the messages and, for example, forward them to another mailbox using the built-in .NET SmtpClient.
On top of that, all mail servers I know of implement not just the SMTP protocol but also a slew of anti-spam measures. Most of these measures have matured over many years. You get all that for free if you build on top of an existing mail server infrastructure.
Then again, configuration of just about any mail server I know seems to be a fine art, hence my frustration with them.

Email Notification Service

What are some ideas (using .NET and SQL 2005) for implementing a service that sends emails? The emails are to be data-driven. The date and time an email is to be sent is a field in a table.

I have built several high volume email notification services used to send data driven emails in the past. A few recommendations:
Look into using a high quality email service provider that specializes in managing bounces, unsubscribes, isp and black list management, etc. If sending email is critical to your business, but not your main business it will be worth it. Most will have an api for sending templated messages, click tracking, open rates and will have provide triggers etc.
Look into the SQL Server Service Broker to queue the actual messages, otherwise you can consider Microsoft Message Queuing Services. There is no need to build your own queuing service. We spent too much time dealing with queing infrastructure code when this was already solved.
Develop a flexible set of events on your business tier to allow for the triggering of such messages and put them in your queue asynchronously, this will save you alot of grief in the long run as opposed to polling on the DB or hacking it in with Database triggers.

You can use triggers to send emails on UPDATE/DELETE/INSERT. The triggers can be implemented with .Net, just send mails from there using the classes in System.Net.Mail namespace.
Here is a good article how to implement CLR (.Net) triggers in .Net.
For a light-weight SMPT server, and to minimize the delays, you can use the one, recommended in Kenny's answer.

Thanks everyone for the feedback. For simplicity's sake I've started out with a SP that looks up the reminders to be sent and uses sp_ send_ dbmail (SQL Database Mail) to send the emails. This runs on a job every minute. I update the record to indicate the reminder was sent with the MailItemId sent back from sp_ send_ dbmail. The volume of reminders expected is worst case in the 10^2 range per day.
I'd love to hear feedback about any shortcomings people think this solution may have.
By the way, I can't believe Vista doesn't come with a local SMTP server! Luckily Google is more generous, I used Gmail's server for testing.

Usually, I just spin up a process such as http://caspian.dotconf.net/menu/Software/SendEmail/

I was going to suggest SQL Server Notification Services, which will handle the job nicely. But I see that's been dropped from SQL Server 2008, so you probably don't want to go there.

Data Driven SSRS Subscriptions? Just a thought.