I have found the following question (How to configure the RequiresDuplicateDetection for AzureServiceBus topics) about how to set the RequiresDuplicationDetection property when configuring a publish topic from a producer application in MassTransit. However, I have not been able to find out how to do it for commands that are transmitted to a queue with Send rather than Publish.
Additionally, I have found that when configuring a consumer of one the queues in question I can set the property easily, as shown below. This however is not ideal for my use case, if possible I would much rather the producer set this property when it starts and creates the queue.
cfg.ReceiveEndpoint(queue, e =>{
e.RequiresDuplicateDetection = true;
e.ConfigureConsumer<JobEventConsumer>(registrationContext, consumerConfig =>{
consumerConfig.UseMessageRetry(r =>{
r.Interval(10, TimeSpan.FromMilliseconds(200));
r.Ignore<ValidationException>();
});
});
});
Update: After a bit more investigation I have also found that setting the property to true at the global config level doesn't seem to work either. Code shown below
class Program {
static async Task Main(string[] args) {
EndpointConvention.Map<ExtractionRequest>(new Uri("queue:test-queue"));
var busControl = Bus.Factory.CreateUsingAzureServiceBus(cfg =>{
cfg.Host("My connection string");
cfg.RequiresDuplicateDetection = true;
cfg.EnablePartitioning = true;
});
await busControl.StartAsync();
try {
do {
string value = await Task.Run(() =>{
Console.WriteLine("Enter message (or quit to exit)");
Console.Write("> ");
return Console.ReadLine();
});
if ("quit".Equals(value, StringComparison.OrdinalIgnoreCase)) break;
await busControl.Send<ExtractionRequest>(new {});
}
while (true);
}
finally {
await busControl.StopAsync();
}
}
}
public interface ExtractionRequest {}
Any advice on how to turn RequiresDuplicationDetection on for a queue from the producer is welcomed.
Thanks in advance, James.
You can't set queue properties from a message sender, it's the responsibility of the receive endpoint.
The receive endpoint is the responsible component because it's declaring the queue and related attributes.
The reason publish is different is because topics can be configured by the producer, since there may be multiple consumer subscriptions on a single topic.
Related
I am using Azure.Messaging.ServiceBus nuget package to work with Azure service bus. We have created a topic and a subscription. The subscription has 100+ messages. We want to read all the message and continue to read message as they arrive.
Microsoft.Azure.ServiceBus package (deprecated now) provided RegisterMessageHandler which use to process every incoming message. I am not able to find similar option under Azure.Messaging.ServiceBus nuget package.
I am able to read one message at a time but I have to call await receiver.ReceiveMessageAsync(); every time manually.
To receive multiple messages (a batch), you should use ServiceBusReceiver.ReceiveMessagesAsync() (not plural, not singular 'message'). This method will return whatever number of messages it can send back. To ensure you retrieve all 100+ messages, you'll need to loop until no messages are available.
If you'd like to use a processor, that's also available in the new SDK. See my answer to a similar question here.
As suggested by #gaurav Mantri, I used ServiceBusProcessor class to implement event based model for processing messages
public async Task ReceiveAll()
{
string connectionString = "Endpoint=sb://sb-test-today.servicebus.windows.net/;SharedAccessKeyName=manage;SharedAccessKey=8e+6SWp3skB3Aedsadsadasdwz5DU=;";
string topicName = "topicone";
string subscriptionName = "subone";
await using var client = new ServiceBusClient(connectionString, new ServiceBusClientOptions
{
TransportType = ServiceBusTransportType.AmqpWebSockets
});
var options = new ServiceBusProcessorOptions
{
// By default or when AutoCompleteMessages is set to true, the processor will complete the message after executing the message handler
// Set AutoCompleteMessages to false to [settle messages](https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement#peeklock) on your own.
// In both cases, if the message handler throws an exception without settling the message, the processor will abandon the message.
AutoCompleteMessages = false,
// I can also allow for multi-threading
MaxConcurrentCalls = 1
};
await using ServiceBusProcessor processor = client.CreateProcessor(topicName, subscriptionName, options);
processor.ProcessMessageAsync += MessageHandler;
processor.ProcessErrorAsync += ErrorHandler;
await processor.StartProcessingAsync();
Console.ReadKey();
}
public async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
Console.WriteLine(body);
// we can evaluate application logic and use that to determine how to settle the message.
await args.CompleteMessageAsync(args.Message);
}
public Task ErrorHandler(ProcessErrorEventArgs args)
{
// the error source tells me at what point in the processing an error occurred
Console.WriteLine(args.ErrorSource);
// the fully qualified namespace is available
Console.WriteLine(args.FullyQualifiedNamespace);
// as well as the entity path
Console.WriteLine(args.EntityPath);
Console.WriteLine(args.Exception.ToString());
return Task.CompletedTask;
}
I'm using MS newly. I'm thinking to build something like this. I have two queues, ConsumeQueue and DelayQueue.
DelayQueue has no consumer and defined dead letter exchange ConsumeQueue.
DelayQueue "x-message-ttl" queue argument setted for example 10 seconds.
Consumer consuming ConsumeQueue and process data. Handle unsuccessful datas and send message to DelayQueue.
After 10 seconds message moving to dead-letter ConsumeQueue and consuming less data and process.
note: I don't want to scheduled message plugin.
Startup configuration, please note if it can be better:
services.AddMassTransit(configure =>
{
configure.AddBus(factory =>
{
return Bus.Factory.CreateUsingRabbitMq(mqConfig =>
{
mqConfig.Host(eventBusOptions.Host, hostConfigurator =>
{
hostConfigurator.Username(eventBusOptions.Username);
hostConfigurator.Password(eventBusOptions.Password);
});
mqConfig.ReceiveEndpoint($"ConsumeQueue", endpointConfig =>
{
endpointConfig.Consumer<ConsumeMessage>(factory);
});
mqConfig.ReceiveEndpoint($"DelayedQueue", endpointConfig =>
{
endpointConfig.SetQueueArgument("x-message-ttl", TimeSpan.FromSeconds(10));
endpointConfig.BindDeadLetterQueue($"{nameof(FundValueConsumer)}", $"{nameof(FundValueConsumer)}", configure: a =>
{
a.Lazy = true;
a.Durable = true;
});
});
});
});
});
And inside ConsumeQueue consumer, the send message to DelayQueue part:
...
var endpoint = await _busControl.GetSendEndpoint(new Uri("queue:DelayedQueue"));
await endpoint.Send(retryCodeMessage);
//Also i tried this too
await endpoint.Send(retryCodeMessage, (context) =>
{
context.TimeToLive = TimeSpan.FromSeconds(15);
});
...
When send the message line throwing:
The AMQP operation was interrupted: AMQP close-reason,
initiated by Peer, code=406, text='PRECONDITION_FAILED
- inequivalent arg 'x-message-ttl' for queue 'DelayedQueue' in vhost 'dev':
received none but current is the value '10000' of type 'signedint'', classId=50, methodId=10
What should i do? What i did, it could be wrong. Please show the correct implementation of the plan I want
I created a MassTransit quickstart program to interact with my localhost RabbitMQ:
namespace ConsoleApp1
{
public static class Program
{
public class YourMessage
{
public string Text { get; set; }
}
public static async Task Main(params string[] args)
{
var bus = Bus.Factory.CreateUsingRabbitMq(sbc =>
{
var host = sbc.Host(new Uri("rabbitmq://localhost"), h =>
{
h.Username("guest");
h.Password("guest");
});
sbc.ReceiveEndpoint(host, "test_queue", ep =>
{
ep.Handler<YourMessage>(async context => await Console.Out.WriteLineAsync($"Received: {context.Message.Text}"));
});
});
await bus.StartAsync();
await bus.Publish(new YourMessage{Text = "Hi"});
Console.WriteLine("Press any key to exit");
Console.ReadKey();
await bus.StopAsync();
}
}
}
Everything looked fine untill I actually checked the underlying RabbitMQ management and found out that just for this very simple program, MassTransit created 3 exchanges and 2 queues.
Exchanges, all fanouts:
ConsoleApp1:Program-YourMessage: Durable
VP0003748_dotnet_bus_6n9oyyfzxhyx9ybobdmpj8qeyt: Auto-delete and Durable?
test_queue: Durable
Queues:
VP0003748_dotnet_bus_6n9oyyfzxhyx9ybobdmpj8qeyt: x-expire 60000
test_queue: Durable
I would like to know why all of that is necessary or is the default configuration? In particular, I am not really sure to get the point of creating so "many".
It is all described in the documentation.
ConsoleApp1:Program-YourMessage is the message contract exchange, here messages are being published.
test_queue is the endpoint exchange. It binds to the message exchange. This way, when you have multiple consumers for the same message type (pub-sub), they all get their copy of the message.
test_queue is the queue, which binds to the endpoint exchange. Publish-subscribe in RMQ requires exchanges and queues can find to exchanges, so messages get properly delivered.
Both non-durable queue and exchange with weird names are the endpoint temp queue and exchange, which are used for request-response.
Just wondering the best way (even if via Portal, Powershell, or C#) to purge the messages off of a Service Bus Topic's Subscription.
Imagine we have a topic with 4 subscriptions, and we only want to purge the messages from one of the subscriptions.
I have a feeling the only way may be to read the messages in a while loop, but hoping for something better.
UPDATE:
Apart from using code, you can use the Server Explorer as suggested in the answer - right click subscription and purge messages:
You can most certainly do it via code. If you're using Service Bus SDK, you could do something like the following:
static void PurgeMessagesFromSubscription()
{
var connectionString = "Endpoint=sb://account-name.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=access key";
var topic = "topic-name";
var subscription = "subscription-name";
int batchSize = 100;
var subscriptionClient = SubscriptionClient.CreateFromConnectionString(connectionString, topic, subscription, ReceiveMode.ReceiveAndDelete);
do
{
var messages = subscriptionClient.ReceiveBatch(batchSize);
if (messages.Count() == 0)
{
break;
}
}
while (true);
}
What this code will do is fetch messages from the subscription (100 at a time) in Receive & Delete mode so that as soon as messages are fetched, they are deleted from the subscription automatically.
I believe Service Bus Explorer tool also has the capability to purge messages. You can use that as well instead of writing the code.
If you have a lot of messages and can tolerate a bit of downtime on subscriber side, it might be faster to just drop the subscription and create a new one with the same name.
Thank you #Gaurav Mantri, I used slightly changed code without the batch option with version 5.2.0 of Microsoft.Azure.ServiceBus Nuget Package:
var connectionString = "Endpoint=sb://";
var topic = "topic";
var subscription = "subscription";
var subscriptionClient = new SubscriptionClient(connectionString, topic, subscription, ReceiveMode.ReceiveAndDelete);
subscriptionClient.RegisterMessageHandler(
(message, token) =>
{
Console.WriteLine($"Received message: SequenceNumber:
{message.SystemProperties.SequenceNumber}");
return Task.CompletedTask;
},
(exceptionEvent) =>
{
Console.WriteLine("Exception = " + exceptionEvent.Exception);
return Task.CompletedTask;
});
Can I somehow choose not to create _skipped queues in a request/response scenario?
We have chosen to build our app like this (don't ask why, it has its reasons). We are doing a PublishRequest to multiple consumers and only one consumer responds. all others will generate a skipped queue (timeout error). I don't want to change this structure. But if I could disable the creation of skipped queues in RabbitMQ, then it would be perfect.
BusControl for request:
BusControl = Bus.Factory.CreateUsingRabbitMq(busFactoryConfig =>
{
busFactoryConfig.Host(new Uri(ServerUrl), hostConfig =>
{
hostConfig.Username(Username);
hostConfig.Password(Password);
});
busFactoryConfig.AutoDelete = true;
busFactoryConfig.Durable = true;
});
Request method:
var requestClient = BusControl.CreatePublishRequestClient<TRequest, TResponse>(TimeSpan.FromMilliseconds(5000));
TResponse response = TaskUtil.Await(() => requestClient.Request(request));
BusControl for respond:
BusControl = Bus.Factory.CreateUsingRabbitMq(busFactoryConfig =>
{
IRabbitMqHost rabbitMqHost = busFactoryConfig.Host(new Uri(ServerUrl), hostConfig =>
{
hostConfig.Username(Username);
hostConfig.Password(Password);
});
busFactoryConfig.ReceiveEndpoint(rabbitMqHost, receiveEndpointConfig =>
{
if (RetryLimit > 0)
receiveEndpointConfig.UseRetry(Retry.Incremental(3, 5000, 5000));
receiveEndpointConfig.Consumer<TConsumer>();
receiveEndpointConfig.PurgeOnStartup = false;
receiveEndpointConfig.AutoDelete = true;
receiveEndpointConfig.Durable = true;
});
});
Respond method in consumer:
return context.RespondAsync(response);
If I want to timeout a consumer which will generate a dead-letter message and a skipped queue, then I return this:
return Task.CompletedTask;
But I don't want the skipped queue which is visible in the RabbitMQ GUI, other than that it is working great.
I also asked the same question to Chris Patterson a few months ago but didn't get an answer: https://stackoverflow.com/a/39236139.
I just don't want that the app creates so many skipped queues, if it creates just one skipped queue then I can still live with it but there are so many because there is an identifier in the queuename...:
It is a default middleware so at this moment there is no way to turn it off. However, you can create an empty consumer for JObject and it will consume all messages, which mean no matter what message you receive, there will be at least one consumer. Then dead letter queue middleware will not be engaged.