Is it possible to set the delivery tag on the message that I publish in my publisher client code? Is it possible to get the delivery tag for the message that I publish in my publisher client?
Here is my situation:
1) I am creating a job (job a) that creates many messages that might take a long time to process. Multiple jobs can be running at one time.
2) I am using a single queue
3) I am using a single receiver
4) I want the ability to "delete" all messages for job a.
If I knew what the delivery tag is then I can just ack each of the message to "delete" just the ones I want from the queue.
Thanks in advance.
You can set custom headers for every message in the following way:
Publisher:
byte[] messageBodyBytes = System.Text.Encoding.UTF8.GetBytes("Hello, world!");
IBasicProperties props = model.CreateBasicProperties();
props.ContentType = "text/plain";
props.DeliveryMode = 2;
props.Headers = new Dictionary<string, object>();
props.Headers.Add("header", value);
model.BasicPublish(exchangeName,
routingKey, props,
messageBodyBytes);
Reference: https://www.rabbitmq.com/dotnet-api-guide.html#publishing
At the consumer you have access to the BasicProperties.
Consumer:
public override void HandleBasicDeliver(string consumerTag, ulong deliveryTag, bool redelivered, string exchange, string routingKey, IBasicProperties properties, byte[] body)
{
header_value = // Read the value from the properites variable
if (header_value matches) {
// Reject or delete the message
_channel.BasicReject(deliveryTag, false);
}
else {
// Accept the message and do your processing
_channel.BasicAck(deliveryTag, false);
}
}
Reference: https://www.tutlane.com/tutorial/rabbitmq/csharp-read-messages-from-rabbitmq-queue
I am using RabbitMQ with pika which is a python client. I am not that fluent with C# or .NET, but I hope this helps!
Related
I am using Azure.Messaging.ServiceBus nuget package to work with Azure service bus. We have created a topic and a subscription. The subscription has 100+ messages. We want to read all the message and continue to read message as they arrive.
Microsoft.Azure.ServiceBus package (deprecated now) provided RegisterMessageHandler which use to process every incoming message. I am not able to find similar option under Azure.Messaging.ServiceBus nuget package.
I am able to read one message at a time but I have to call await receiver.ReceiveMessageAsync(); every time manually.
To receive multiple messages (a batch), you should use ServiceBusReceiver.ReceiveMessagesAsync() (not plural, not singular 'message'). This method will return whatever number of messages it can send back. To ensure you retrieve all 100+ messages, you'll need to loop until no messages are available.
If you'd like to use a processor, that's also available in the new SDK. See my answer to a similar question here.
As suggested by #gaurav Mantri, I used ServiceBusProcessor class to implement event based model for processing messages
public async Task ReceiveAll()
{
string connectionString = "Endpoint=sb://sb-test-today.servicebus.windows.net/;SharedAccessKeyName=manage;SharedAccessKey=8e+6SWp3skB3Aedsadsadasdwz5DU=;";
string topicName = "topicone";
string subscriptionName = "subone";
await using var client = new ServiceBusClient(connectionString, new ServiceBusClientOptions
{
TransportType = ServiceBusTransportType.AmqpWebSockets
});
var options = new ServiceBusProcessorOptions
{
// By default or when AutoCompleteMessages is set to true, the processor will complete the message after executing the message handler
// Set AutoCompleteMessages to false to [settle messages](https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement#peeklock) on your own.
// In both cases, if the message handler throws an exception without settling the message, the processor will abandon the message.
AutoCompleteMessages = false,
// I can also allow for multi-threading
MaxConcurrentCalls = 1
};
await using ServiceBusProcessor processor = client.CreateProcessor(topicName, subscriptionName, options);
processor.ProcessMessageAsync += MessageHandler;
processor.ProcessErrorAsync += ErrorHandler;
await processor.StartProcessingAsync();
Console.ReadKey();
}
public async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
Console.WriteLine(body);
// we can evaluate application logic and use that to determine how to settle the message.
await args.CompleteMessageAsync(args.Message);
}
public Task ErrorHandler(ProcessErrorEventArgs args)
{
// the error source tells me at what point in the processing an error occurred
Console.WriteLine(args.ErrorSource);
// the fully qualified namespace is available
Console.WriteLine(args.FullyQualifiedNamespace);
// as well as the entity path
Console.WriteLine(args.EntityPath);
Console.WriteLine(args.Exception.ToString());
return Task.CompletedTask;
}
I was able to consume multiple messages that are sent by multiple producers to the same exchange with different routing key using the above code and was able to insert each message to database.
But this will consume too much of resources as messages will be inserted into DB one after the other. So I decided to go for batch insert and I found I can set BasicQos
After setting the message limit to 10 in BasicQos, my expectation is the Console.WriteLine must write 10 messages, but it is not as expected.
My expectation is to consume N number messages from the queue and do bulk insert and on successful send ACK else No ACK
Here is the piece of code I use.
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueBind(queue: "queueName", exchange: "exchangeName", routingKey: "Producer_A");
channel.QueueBind(queue: "queueName", exchange: "exchangeName", routingKey: "Producer_B");
channel.BasicQos(0, 10, false);
var consumer = new EventingBasicConsumer(channel);
channel.BasicConsume(queue: "queueName", noAck: false, consumer: consumer);
consumer.Received += (model, ea) =>
{
try
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
// Insert into Database
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);
Console.WriteLine(" Recevier Ack " + ea.DeliveryTag);
}
catch (Exception e)
{
channel.BasicNack(deliveryTag: ea.DeliveryTag, multiple: false, requeue: true);
Console.WriteLine(" Recevier No Ack " + ea.DeliveryTag);
}
};
Console.ReadLine();
}
}
BasicQos = 10 means that the client fetch only 10 messages at time, but when you consume it you will see always one message a time.
Read here: https://www.rabbitmq.com/consumer-prefetch.html
AMQP specifies the basic.qos method to allow you to limit the number
of unacknowledged messages on a channel (or connection) when consuming
(aka "prefetch count").
for your scope you have to download the messages, put it inside a temporary list and then insert into the DB.
an then you can use:
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: true);
void basicAck()
Parameters:
deliveryTag - the tag from the received
AMQP.Basic.GetOk or AMQP.Basic.Deliver
multiple - true to acknowledge
all messages up to and including the supplied delivery tag; false to
acknowledge just the supplied delivery tag.
Example
final List<String> myMessagges = new ArrayList<String>();
channel.basicConsume("my_queue", false, new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
myMessagges.add(new String(body));
System.out.println("Received...");
if (myMessagges.size() >= 10) {
System.out.println("insert into DB...");
channel.basicAck(envelope.getDeliveryTag(), true);
myMessagges.clear();
}
}
});
Batch size based consumption can be done using the channel.basicQos().
Channel channel = connection.createChannel();
channel.basicQos(10);
It specifies the maximum no of messages to be fetched without sending ACK for each.
Use the DefaultConsumer class and override its methods.
Consumer batchConsumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
}
#Override
public void handleCancelOk(String consumerTag) {
}
};
Consume 10 messages using channel.basicConsume()
channel.basicConsume(QUEUE_NAME, false, batchConsumer);
When channel.basicConsume() is called it will fetch a batch of 10 messages. 'false' is set to disable auto ack, and ACK to be sent only once after consuming entire batch.
channel.basicAck(getLastMessageEnvelope().getDeliveryTag(), true);
Here 'true' means we are sending ACK for multiple messages.
Detailed explanation can be found in
RabbitMQ Batch Consumption
I have a socket application that allows thousands of clients to connect. It stores them in a ConcurrentDictionary<int, Socket> and operates solely for request-response situations:
When I need data, I find the relevant socket and send a request, asking for the data I need.
After sending the request, I receive bytes until it sends the response. Then I stop receiving.
Like this:
public Task<Message> Request(int clientId, Message message)
{
Socket client;
return Clients.TryGetValue(clientId, out client)
? RequestInternal(client, message);
: _EmptyTask;
}
public async Task<Message> RequestInternal(Socket client, Message message)
{
await SendAsync(client, message).ConfigureAwait(false);
return await ReceiveOneAsync(client).ConfigureAwait(false);
}
Now I need to change this application to allow clients to send me anything, anytime; even without I make a request for it. Which -I think- will require constantly receiving from the sockets and a completely different approach.
Questions:
What are the bets-known approaches (best-practices) for this kind of applications?
Any gotchas you can tell me about or any guides you can point me to?
What I have in mind:
Disclaimer: This part is a little long and completely hypothetical. You can skip that if you have an answer to the above questions.
What I have in mind:
Receiving bytes constantly and adding the assembled PDUs to a BlockingCollection<Message>.
Creating a thread, dedicated to process received messages using BlockingCollection's GetConsumingEnumerable method.
The processing thread will do this:
foreach (var message in Messages.GetConsumingEnumerable())
ProcessMessage(message);
With this I can receive and process everything the clients send but distinguishing the messages that are sent to reply my requests from the messages that are sent because the client needed to would be an issue.
I think I can send a unique identifier byte (unique to that particular client) with the request. Then the client can send that identifier back to me in its response and I can use it to distinguish the response.
ProcessMessage(Message msg)
{
// msg is a message from msg.Sender.
if (msg.Id == 0)
{
// msg is not a response, do processing.
}
else
{
// msg is a response to the message that's sent with msg.Id.
// Find the request that:
// * ...is made to msg.Sender
// * ...and has the msg.Id as identifier.
// And process the response according to that.
}
}
This means I also have to store the requests.
Here is an hypothetical version of RequestInternal:
Edit: Replaced Wait calls with awaits after Stephen Cleary's answer.
private async Task RequestInternal(Socket client, Message message)
{
var request = new Request(client, message);
Requests.Add(request);
await SendAsync(client, message).ConfigureAwait(false);
return await request.Source.Task.ConfigureAwait(false);
}
And the Request class:
private sealed class Request
{
public readonly byte Id;
public readonly Socket Client;
public readonly Message Message;
public readonly TaskCompletionSource<Message> Source;
public Request(Socket client, Message message)
{
Client = client;
Message = message;
Source = new TaskCompletionSource<Message>();
// Obtain a byte unique to that socket...
Id = GetId(client);
}
}
And ProcessMessage becomes this:
ProcessMessage(Message msg)
{
if (msg.Id == 0)
OnReceived(msg); // To raise an event.
else
{
// Method to find a request using msg.Sender and msg.Id
var request = Requests.Find(msg);
if (request != null)
request.Source.SetResult(msg);
}
}
Although I have no idea what kind of collection type Requests would be.
Edit: I've used a ConcurrentDictionary<Key, Request> where Key is a private struct with an Int32 (id of the socket) and a Byte (id of the message) fields. it also implements IEquatable<T>.
I wrote a TCP/IP .NET Sockets FAQ a few years ago that addresses some common problems (such as message framing, continuous reading, and explanations of common errors). The code samples all use the Socket class, but the same concepts apply to all TCP/IP sockets.
Regarding your protocol design and request/response matching, the overall approach sounds good. You'll need to ensure you're threadsafe (e.g., Requests would probably be a ConcurrentDictionary). Also, you should await SendAsync rather than calling Wait.
An alternative approach that I've played around with but haven't put into production is based on TPL Dataflow. You can create one block that represents the "output" for each client and another block for the "input". Then you can layer your message framing on that, and layer your request/response matching on that, and then send any remaining (unsolicited) messages to a single shared BufferBlock.
So your "end-user" API would end up looking like this:
// Send a request and asynchronously receive a matching response.
Task<Message> RequestAsync(int clientId, Message message);
// Endpoint for unsolicited messages.
IReceivableSourceBlock<Tuple<int, Message>> UnsolicitedMessages { get; }
You could then hook up an ActionBlock to UnsolicitedMessages to execute a delegate whenever one comes in.
At this line of code i am getting the error as i mentioned
I declared MSMQ_NAME as string as follows
private const string MSMQ_NAME = ".\\private$\\ASPNETService";
private void DoSomeMSMQStuff()
{
using (MessageQueue queue = new MessageQueue(MSMQ_NAME))
{
queue.Send(DateTime.Now); //Exception raises
queue.Close();
}
}
Can you first verify the queue is existing with the name 'ASPNETService' at below location?
Computer Management -> Services and Applications -> Message Queuing -> Private Queues
I had a similar problem. I was confused because my code worked on my local development machine, but not in production. Even stranger, the queues were created the exact same way.
It turns out that IIS doesn't have access to them by default. I just opened up the permissions.
Computer Management -> Private Queues -> right-click queue name -> Properties -> Security Tab -> click "Everyone" user -> click Full Control/Allow checkbox -> click OK
This fixed it for me, and in my case it's not an issue, but you may want to think about the ramifications of just opening it up for all users.
Also, I had to do this across all queues on all servers. There doesn't seem to be a way to multi-select queues or folders in order to set permissions for multiple queues simultaneously.
I was having the same problem.
I had created a new private queue and gave Full Permission to Everyone.
But I was still catching a "Queue does not exist or you do not have sufficient permissions to perform the operation" when trying to Send() to the queue. And I was able to verify that MessageQueue.Exists(".\\private$\\myqueue") was returning true.
Restarting the Message Queuing Service resolved my the problem for me.
I had same problem and I did like below where I check whether queue exists or not. If yes send message else create queue and then send message
MessageQueue msgQueue = null;
string queuePath = ".\\Private$\\billpay";
Payment newPayment = new Payment()
{
Payee = txtPayee.Text,
Payor = txtPayor.Text,
Amount = Convert.ToInt32(txtAmount.Text),
DueDate = dpDueDate.SelectedDate.Value.ToShortDateString()
};
Message msg = new Message();
msg.Body = newPayment;
msg.Label = "Gopala - Learning Message Queue";
if (MessageQueue.Exists(queuePath) == false)
{
//Queue doesnot exist so create it
msgQueue = MessageQueue.Create(queuePath);
}
else
{
msgQueue = new MessageQueue(queuePath);
}
msgQueue.Send(msg);
I was facing the same problem, I had resolved it using the following class to create queue
private MessageQueue messageQueue;
public const string DEFAULT_QUEUE_NAME = "newQueue";
public const string QUEUENAME_PREFIX = ".\\Private$\\";
public static string QueueName
{
get
{
string result = string.Format("{0}{1}", QUEUENAME_PREFIX, DEFAULT_QUEUE_NAME);
return result;
}
}
public void SendMessage()
{
string queuePath = QueueName;
MessageQueue messageQueue = MessageQueue.Create(queuePath);
messageQueue.Send("msg");
}
Create message queue in same manner for receiving the message.
For others struggling with this and pulling their hair out like I have been, I finally found something that works when all of the upvoted suggestions failed.
Even if you think the host name of your target queue's hosting system is being resolved correctly, don't believe it. Try replacing the host name with an IP address and see if it works. It does for me. I can WRITE to a public queue using a host name on my remote server without problems, but trying to READ from it produces exactly the error listed for this question.
For example, if I declare the following:
private static string QueueName = #"FormatName:DIRECT=TCP:SOMEHOST\MyQueue";
private static System.Messaging.MessageQueue Queue = new System.Messaging.MessageQueue(QueueName);
Where "MyQueue" is a public queue on server SOMEHOST, the following code will successfully insert messages to the queue, but always fails on the Receive():
Queue.Formatter = new XmlMessageFormatter(new Type[] { typeof(String) });
// The Receive() call here is a blocking call. We'll wait if there is no message in the queue, and processing
// is halted until there IS a message in the queue.
//
try
{
Queue.Send("hello world", System.Messaging.MessageQueueTransactionType.Single);
var msg = Queue.Receive(MessageQueueTransactionType.Single);
}
catch (Exception ex)
{
// todo error handling
}
One simple change in how I specify the queue location is all that's needed to make the Receive() stop failing with the dreaded "queue does not exist or you do not have sufficient permissions" error:
private static string QueueName = #"FormatName:DIRECT=TCP:192.168.1.100\MyQueue";
(Obviously I've obfuscated IP addresses and other sensitive info). Using the IP address is not obviously a production-worthy scenario, but it did point me to some type of name resolution problem as being the possible cause of the error. I cannot explain why Send() works but Receive() does not when I am using a host name instead of IP, but I can reproduce these results consistently. Until I can figure out what's going on with the name resolution, I'm no longer wasting a day trying to read messages from a queue.
I am using Rabbit MQ in C#. This is my scenario
A separate process publishes messages to the queue
Client has to read set of N messages from queue
Process the N messages
Acknowledge the N messages
Under the same channel, I receive the messages and then process them and then acknowledge them. The server process keeps publishing messages. The problem I am facing is, when I try to get next set of messages, they do not come in the same order as it was published by the publishing process. The messages come in a random order. Only the first set of messages come in the correct order.
Does any one what is going wrong here? Is creating a new channel to access the next set of messages not right? Below is the sample code:
while (true)
{
using (IModel getChannel = MQConnection.CreateModel())
{
// Create a consumer
QueueingBasicConsumer consumer = CreateQueueConsumer(getChannel, exchangeName, queueName);
int numberOfMessages = 100;
// Next Recieve
List<object> msgSet = GetNextSetOfMessages(consumer, getChannel, exchangeName, queueName, numberOfMessages, out finalDeliverytag);
// Do some processing
if (finalDeliverytag > 0)
AckFinishedMessages(exchangeName, queueName, finalDeliverytag, getChannel);
if (finalDeliverytag == 0)
break;
}
}
Kindly help. Thanks!