Develop Reference

Never ending messages with RabbitMQ

We are using MassTransit(3.5.2) together with RabbitMQ(3.6.6). We are having a problem with a message that never gets removed from the queue (even if we have read and consumed the message).
To read from the queue we have implemented an IConsumer class.
The only thing we are doing is closing the sender (abrupt).
After that the message never gets acknowledged and rabbitmq is continuing to send the same message to our consumer all the time.
Have anyone else got the same problem and how did you solve this issue?
:: Update from comments ::
We had already checked the log files and it says: "closing AMQP connection <0.18285.1> ([::1]:57008 -> [::1]:5672): client unexpectedly closed TCP connection ".
That doesnt feel that wierd since i am actually closing the tcp connection unexpectedly with killing the .exe file :)
Regarding the log files from masstransit we have also done that and we do not get any error, we only gets two debug messages.
One that we have received and one that we are sending the result.
DEBUG 47 MassTransit.Messages - RECEIVE rabbitmq://localhost/[VirtualHost]/[ConsumerName] N/A ContractCommand CommandConsumer(00:00:00.0364932)
DEBUG 30 MassTransit.Messages - SEND rabbitmq://localhost/[VirtualHost]/bus-[ComputerName]-[Sende‌​rName].Server.vshost‌​-4bayyydsf9rfs3qzbdk‌​gx8bbr1?durable=fals‌​e&autodelete=true d0700000-762f-c85b-f03a-08d4679c39d4 Result
One observation that I have made in my consumer is that at the same time as I am force closing my sender I get an MessageNotConfirmedException followed by some AlreadyClosedException from RabbitMQ.
And it's after that we get in the infinite loop when MT does not set the ACK/NACK. (and in the infinite loop I do not get any MessagenNotConfirmedException).
Also for my consumer to properly work again I need to restart my consumer then it will be ACK/NACKed.
MessageNotConfirmedMessage: "'MassTransit.RabbitMqTransport.MessageNotConfirmedException‌​' in mscorlib.dll Additional information: rabbitmq://localhost/[VirtualHost]/bus-[ComputerName]-[Servi‌​ce].Server.vshost-4b‌​ayyydsf9rfsf3ybdkgxg‌​5h8b => The message was not confirmed by RabbitMQ
'RabbitMQ.Client.Exceptions.AlreadyClosedException' Additional information: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=404, text="NOT_FOUND - no exchange 'bus-[ComputerName]-[ServiceName].Server.vshost-4bayyydsf9rf‌​s3qzbdkgx8bbr1' in vhost '[VirtualHost]'", classId=60, methodId=40, cause=

I have updated our code to use MassTransit 3.5.7 and after that I have not been able to reproduce this issue.

When you are done processing your queue object, you can mark it as successful by BasicAck(e.DeliveryTag, false)

Why is message put into a dead-letter queue (MSMQ)?

Preamble:
I have a web application, which sends MSMQ messages (with UseDeadLetterQueue = true) from time to time. I can also see that some of those messages are put into a system dead letter queue. However, I can't see the failure reason directly from "Computer Management" console.
I found the following resource:
https://msdn.microsoft.com/en-us/library/ms789035(v=vs.110).aspx,
which allows to check message status and failure. But it's usable only in WCF scope (as it uses OperationContext).
Basically checking message status and failure reason is what I need, but how can I configure a WCF service to listen to a dead-letter queue and track all the messages put inside, regardless of sender?
Or is there any other (non-WCF) way to get reason of why specific message was put into a dead letter queue?

The linked article you found tells you how to create a WCF service that takes messages off the DLQ. Note that it tells you you must use AddressFilterMode = AddressFilterMode.Any to pull all messages off regardless of sender.
Note: by the time the message ends up on the DLQ, you will not have information about the reason this occurred. Regardless of whether or not you're using WCF that information isn't stored in the queue.
If for example your recipient is a WCF listener using the MSMQ binding any exception thrown will cause the message to fail to get delivered. MSMQ will retry for a while and then eventually give up and put the message in the DLQ. So if you wanted to know the original reason you need to add some tracing in your WCF service to catch and trace thrown exceptions.

MSMQ Event Logging doesn't show dropped messages?

So I'm trying to get MSMQ messages forwarded from one machine to another (which is dead easy - I was surprised), but one of the requirements from the ops side of the house is that we need to be able to see a log entry somewhere when the remote server decides not to accept a message. For example, if I try to send to a nonexistent queue, like so:
MessageQueue remoteQueue = new MessageQueue(#"FormatName:Direct=OS:machinename\private$\notarealqueue");
remoteQueue.Send("Test", MessageQueueTransactionType.Single);
The message goes into the local delivery queue, and appears to get sent across the network, but because the queue doesn't exist, the remote MSMQ manager discards the message. However, there's no entry in the Event Log that I can find about the message being dropped on the floor, and that makes people nervous. The Microsoft/Windows/MSMQ/EndToEnd log only seems to involve successful messages, which doesn't seem particularly useful. Is there a log I'm not seeing somewhere?

You can use MSMQ dead letter queues for that.
message.UseDeadLetterQueue = true;
With that enabled, if message can't be delivered it will be sent to one of two system dead letter queues - one for transactional and one for non transactional messages. You'll also find there the reason why message was not delivered, which was the original destination queue, full message body, label, etc.
You can use one of tools for managing queues to resend or recover these messages.

The event log is solely for the health state of MSMQ. What happens to a single message is trivial and not logged in the event log. Imagine what would happen if a million messages were discarded and had to be logged in the event log.

MSMQ via C# - ACK that message received?

I'm sending a message to a private queue via c# :
MessageQueue msgQ = new MessageQueue(#".\private$\aaa");
msgQ.Formatter = new XmlMessageFormatter(new[] { typeof (String) });
msgQ.Send(msg);
It does work and I do see the message in the queue.
However, is there any way to get an ACK whether the message got to the queue with success ?
ps
BeginPeek and PeekCompleted is an event which is raised when a message becomes available in the queue or when the specified interval of time has expired. it is not helping me because I need to know if the message that I sent was received by msmq. BeginPeek will be raised also if someone else entered a message to the queue. and the last thing I want is to check via BeginPeek - from who this message comes from.
How can I do that?
ps2
Or maybe I don't have to worry since msgQ.Send(msg); will raise an exception if a message wasn't inserted....?

I think what you are trying to do should not be handled in code. When you send the message, it is placed in the outgoing queue. There are numerous reasons why it would not reach the destination, such as a network partition or the destination queue being full. But this should not matter to your application - as far as it is concerned, it sent the message, it committed transaction, it received no error. It is a responsibility of the underlying infrastructure to do the rest, and that infrastructure should be monitored to make sure there are no technical issues.
Now what should really be important to your application is the delivery guarantees. I assume from the scenario that you are describing that you need durable transactional queues to ensure that the message is not lost. More about the options available can be read here
Also, if you need some identifier to display to the user as a confirmation, a common practice is to generate it in the sending code and place it in the message itself. Then the handling code would use the id to do the required work.

Using transactional queues and having all your machines enroll in DTC transactions likely would provide what you're looking for. However, it's kinda a pain in the butt and DTC has side effects - like all transactions are enrolled together, including DB transactions.
Perhaps a better solution would to be use a framework like MassTransit or NServiceBus and do a request-response, allowing the reviecer to respond with actual confirmation message say not only "this has been delivered" but also "I acknowledge this" with timeout options.

As Oleksii have explained about reliable delivery.
However this can effect on performance.
What I can suggest is:
Why not create a MSMQ server on the machine that is sending MSG to other system.
What I am thinking is
Server 1 sends MSMSQ to Server 2
Server 2 receives adds to queue
Server 2 process queue/fire your code here to send a MSMQ msg to Server 1
Server 1 receives MSG (any successful msg with MSGId)
Do your further task
This approach can be an extra mile, but will keep your servers out of performance Lag.

Suggestions for developing a TCP/IP based message client

I've got a server side protocol that controls a telephony system, I've already implemented a client library that communicates with it which is in production now, however there are some problems with the system I have at the moment, so I am considering re-writing it.
My client library is currently written in Java but I am thinking of re-writing it in both C# and Java to allow for different clients to have access to the same back end.
The messages start with a keyword have a number of bytes of meta data and then some data. The messages are always terminated by an end of message character.
Communication is duplex between the client and the server usually taking the form of a request from the Client which provokes several responses from the server, but can be notifications.
The messages are marked as being on of:
C: Command
P: Pending (server is still handling the request)
D: Data data as a response to
R: Response
B: Busy (Server is too busy to handle response at the moment)
N: Notification
My current architecture has each message being parsed and a thread spawned to handle it, however I'm finding that some of the Notifications are processed out of order which is causing me some trouble as they have to be handled in the same order they arrive.
The duplex messages tend to take the following message format:
Client -> Server: Command
Server -> Client: Pending (Optional)
Server -> Client: Data (optional)
Server -> Client: Response (2nd entry in message data denotes whether this is an error or not)
I've been using the protocol for over a year and I've never seen the a busy message but that doesn't mean they don't happen.
The server can also send notifications to the client, and there are a few Response messages that are auto triggered by events on the server so they are sent without a corresponding Command being issued.
Some Notification Messages will arrive as part of sequence of messages, which are related for example:
NotificationName M00001
NotificationName M00001
NotificationName M00000
The string M0000X means that either there is more data to come or that this is the end of the messages.
At present the tcp client is fairly dumb it just spawns a thread that notifies an event on a subscriber that the message has been received, the event is specific to the message keyword and the type of message (So data,Responses and Notifications are handled separately) this works fairly effectively for Data and response messages, but falls over with the notification messages as they seem to arrive in rapid sequence and a race condition sometimes seems to cause the Message end to be processed before the ones that have the data are processed, leading to lost message data.
Given this really badly written description of how the system works how would you go about writing the client side transport code?
The meta data does not have a message number, and I have not control over the underlying protocol as it's provided by a vendor.

The requirement that messages must be processed in the order in which they're received almost forces a producer/consumer design, where the listener gets requests from the client, parses them, and then places the parsed request into a queue. A separate thread (the consumer) takes each message from the queue in order, processes it, and sends a response to the client.
Alternately, the consumer could put the result into a queue so that another thread (perhaps the listener thread?) can send the result to the client. In that case you'd have two producer/consumer relationships:
Listener -> event queue -> processing thread -> output queue -> output thread
In .NET, this kind of thing is pretty easy to implement using BlockingCollection to handle the queues. I don't know if there is something similar in Java.
The possibility of a multi-message request complicates things a little bit, as it seems like the listener will have to buffer messages until the last part of the request comes in before placing the entire thing into the queue.
To me, the beauty of the producer/consumer design is that it forces a hard separation between different parts of the program, making each much easier to debug and minimizing the possibility of shared state causing problems. The only slightly complicated part here is that you'll have to include the connection (socket or whatever) as part of the message that gets shared in the queues so that the output thread knows where to send the response.
It's not clear to me if you have to process all messages in the order they're received or if you just need to process messages for any particular client in the proper order. For example, if you have:
Client 1 message A
Client 1 message B
Client 2 message A
Is it okay to process the first message from Client 2 before you process the second message from Client 1? If so, then you can increase throughput by using what is logically multiple queues--one per client. Your "consumer" then becomes multiple threads. You just have to make sure that only one message per client is being processed at any time.

I would have one thread per client which does the parsing and processing. That way the processing would be in the order it is sent/arrives.
As you have stated, the tasks cannot be perform in parallel safely. performing the parsing and processing in different threads is likely to add as much overhead as you might save.
If your processing is relatively simple and doesn't depend on external systems, a single thread should be able to handle 1K to 20K messages per second.
Is there any other issues you would want to fix?

I can recommend only for Java-based solution.
I would use some already mature transport framework. By "some" I mean the only one I have worked with until now -- Apache MINA. However, it works and it's very flexible.
Regarding processing messages out-of-order -- for messages which must be produced in the order they were received you could build queues and put such messages into queues.
To limit number of queues, you could instantiate, say, 4 queues, and route incoming message to particular queue depending on the last 2 bits (indeces 0-3) of the hash of the ordering part of the message (for example, on the client_id contained in the message).
If you have more concrete questions, I can update my answer appropriately.