Let's imagine that we have Q named "NotificationQ" and have a consumer who gets a task from that Q and sends emails to customers.
Emailing process sends an email by API from mailgun. That API request does not turn 200 every time(the reason is not important). In that time I need to tell RabbitMQ that tasks fail. I know there is a feature called autoAck but if a request fails how the RabbitMQ client pack understood that a fail.
Am I manually trigger ack to say that request failed?
I using https://www.nuget.org/packages/RabbitMQ.Client/ pack to handle RabbitMQ tasks.
var channel = RabbitPrepareFactory.GetConnectionFactory();
channel.BasicQos(0, 1, false);
var notificationPack = channel.BasicGet("notification", true);
var message = System.Text.Encoding.UTF8.GetString(notificationPack.Body.ToArray());
var task = JsonConvert.DeserializeObject<ForgetPasswordEmailNotification>(message);
var isEmailSendSuccessful = SomeFakeEmailSendFunctions(task.Email);
if (!isEmailSendSuccessful)
{
//something for tell RabbitMQ that task fail and not delte that task in q
.......
}
I think this could be usefull. I would use something like a Dead Letter
https://www.rabbitmq.com/dlx.html
So everytime a message is failing for whatever reason, you push the message to that queue.
Once your messaged was recieved by your consumer and the scope of the operation finished, that message is acknowledged so that other consumers will not take a already processed message.
[Edit]
I dont't think its a good ideea to process a message from a queue and afterwards to leave it there if something happens to your BackEnd. If you implement the dead letter queue you could try to reprocess those messages at some time ( Maybe a CronJob ) or if you really don't wanna have dead letter queues you could try to implement in your Client a Retry Mechanism. Polly could work very well in your case https://github.com/App-vNext/Polly
I'm up to send and receive messages over ActiveMQ Artemis with C# applications. In Anycast-mode, everything is working.
When i tried to send and receive in multicast-mode, i can send, but i don't receive any of the messages from the queue.
I tried the trick from java, set the "multicast" flag before the tcp uri, but an error message shows up that there isn't an implementation for "multicast"
private void Receiver()
{
IConnectionFactory factory = new NMSConnectionFactory("multicast:tcp://172.29.213.150:61616");
IConnection connection = factory.CreateConnection("artemis", "simetraehcapa");
connection.Start();
ISession session = connection.CreateSession(AcknowledgementMode.AutoAcknowledge);
IDestination destination = SessionUtil.GetDestination(session, "hund");
IMessageConsumer receiver = session.CreateConsumer(destination);
receiver.Listener += new MessageListener(Message_Listener);
}
Normally I would receive the messages, because I only switched from anycast to multicast, but actually I receive nothing.
If using the AcitveMQ OpenWire NMS client you don't apply that odd multicast thing you've done to the URI, that will give you an error. The client should just work if you use the Session API and not that confusing SessionUtil API that has resulted in many people running into issues.
I'd use Session.CreateTopic to get an ITopic instance and then create a consumer using that which should map over into Artemis Multicast addresses without you needing to do anything. You do of course need to be subscribed before any messages are sent as Topics don't retain messages if no consumers are around when the are sent.
I am working on some POC project and trying to solve the following problem.
I have a Publisher which is sending a messages to the Queue:
bus.PublishAsync<IBaseScenario>(new TestScenario())
.ContinueWith(task =>
{
if (task.IsCompleted && !task.IsFaulted)
Console.WriteLine("TestScenario queued with success.");
else
Console.WriteLine(task.Exception.Message);
});
And some Consumers which are consuming a messages:
bus.SubscribeAsync<IBaseScenario>("test_1_consumer",
message => Task.Factory.StartNew(() =>
{
var testScenario = message as TestScenario;
var anotherTestScenario = message as AnotherTestScenario;
ResolveScenario(testScenario);
ResolveScenario(anotherTestScenario);
}).ContinueWith(task =>
{
if (task.IsCompleted && !task.IsFaulted)
Console.WriteLine("Task ended up with success.");
else
Console.WriteLine(task.Exception.Message);
}));
At this point everything is working as needed, but here is what I would like to achieve.
My Message is some kind of Scenario which contains steps, each Scenario is sent to the Queue and then maintained by a Consumer.
I would like to get a some kind of ACK info from Consumer sent to Publisher everytime when the each Step is done on the Consumer site (for example if its ended up with success or not.
I would like to get also an info about which Consumer got the Message.
Every Message (Scenario) should be treated as atomic operation, so there should not be possible to doing steps on different Consumers and if some Step will end without success, then the whole scenario should be treated as failed.
Are these 2 requirements possible to solve using the following architecture or do I need to use something more?
The easiest thing to do would be to use EasyNetQ's request response model described here https://github.com/EasyNetQ/EasyNetQ/wiki/Request-Response
In the response you can put the identity of the consumer that processed the message and the final status of the scenario. If one scenario is sent in one message, and that scenario contains all the steps necessary then all steps would be processed by a single consumer.
That said, message duplication is always a problem due to either sending the message twice or a message being requeued after a consumer fails. If it is critical that a scenario NEVER be processed more than once, then you will need to implement message deduplication or make each scenario idempotent. That is a general fact of life when working with RabbitMQ.
If you add expiration's to entities you are adding into Redis e.g. in ServiceStack.Redis:
redisClient.Set(elementKey, "some cached value", DateTime.Now.AddMinutes(2));
how can you then subscribe to the element's expiration. The desired outcome would be something ala:
redisClient.Subscribe(elementKey, "expire", DoSomethingBasedOnKey)
Actually you can subscribe to expired keys events, but like Matias said it may take some time until Redis will publish the event.
Redis has Keyspace notifications, you can read about it here,
Keyspace notifications allows clients to subscribe to Pub/Sub channels in order to receive events affecting the Redis data set in some way.
Type of events
Keyspace notifications are implemented sending two distinct type of events for every operation affecting the Redis data space. For instance a DEL operation targeting the key named mykey in database 0 will trigger the delivering of two messages, exactly equivalent to the following two PUBLISH commands:
PUBLISH keyspace#0:mykey del
PUBLISH keyevent#0:del mykey
So what you need is to subscribe to the channel that will publish a message on expired command of keyevent(work also when ttl is reached), it prefix will be like so:
"keyevent#0:expired"
Timing accuracy wasn't matter at my case so I've implemented it like so using the ServiceStack C# Redis client:
string EXPIRED_KEYS_CHANNEL = "__keyevent#0__:expired";
using (IRedisClient client = redisClient.GetClient())
{
using (var cacheSubscription = client.CreateSubscription())
{
cacheSubscription.OnMessage += (ch, expiredKey) =>
{
FireOnKeyExpired(expiredKey);
};
cacheSubscription.SubscribeToChannels(EXPIRED_KEYS_CHANNEL);
}
}
Update:
Make sure to configure redis.conf to allow key events on expired keys:
notify-keyspace-events Ex
Or on the fly like so(configuration may be lost when instance is restarted)
config set notify-keyspace-events Ex
Maybe you don't know that expire messages won't get exactly published when keys get expired:
Timing of expired events
Keys with a time to live associated are
expired by Redis in two ways:
When the key is accessed by a command and is found to be expired.
Via a background system that looks for expired keys in background, incrementally, in order to be able to also
collect keys that are never accessed.
I would suggest you that your best bet is using some task scheduler like the built-in Windows Task Scheduler or Quartz.NET to schedule a task to publish a message to some custom Redis PubSub channel exactly when the key must get expired.
I'm using RabbitMQ in C# with the EasyNetQ library. I'm using a pub/sub pattern here. I still have a few issues that I hope anyone can help me with:
When there's an error while consuming a message, it's automatically moved to an error queue. How can I implement retries (so that it's placed back on the originating queue, and when it fails to process X times, it's moved to a dead letter queue)?
As far as I can see there's always 1 error queue that's used to dump messages from all other queues. How can I have 1 error queue per type, so that each queue has its own associated error queue?
How can I easily retry messages that are in an error queue? I tried Hosepipe, but it justs republishes the messages to the error queue instead of the originating queue. I don't really like this option either because I don't want to be fiddling around in a console. Preferably I'd just program against the error queue.
Anyone?
The problem you are running into with EasyNetQ/RabbitMQ is that it's much more "raw" when compared to other messaging services like SQS or Azure Service Bus/Queues, but I'll do my best to point you in the right direction.
Question 1.
This will be on you to do. The simplest way is that you can No-Ack a message in RabbitMQ/EasyNetQ, and it will be placed at the head of the queue for you to retry. This is not really advisable because it will be retried almost immediately (With no time delay), and will also block other messages from being processed (If you have a single subscriber with a prefetch count of 1).
I've seen other implementations of using a "MessageEnvelope". So a wrapper class that when a message fails, you increment a retry variable on the MessageEnvelope and redeliver the message back onto the queue. YOU would have to do this and write the wrapping code around your message handlers, it would not be a function of EasyNetQ.
Using the above, I've also seen people use envelopes, but allow the message to be dead lettered. Once it's on the dead letter queue, there is another application/worker reading items from the dead letter queue.
All of these approaches above have a small issue in that there isn't really any nice way to have a logarithmic/exponential/any sort of increasing delay in processing the message. You can "hold" the message in code for some time before returning it to the queue, but it's not a nice way around.
Out of all of these options, your own custom application reading the dead letter queue and deciding whether to reroute the message based on an envelope that contains the retry count is probably the best way.
Question 2.
You can specify a dead letter exchange per queue using the advanced API. (https://github.com/EasyNetQ/EasyNetQ/wiki/The-Advanced-API#declaring-queues). However this means you will have to use the advanced API pretty much everywhere as using the simple IBus implementation of subscribe/publish looks for queues that are named based on both the message type and subscriber name. Using a custom declare of queue means you are going to be handling the naming of your queues yourself, which means when you subscribe, you will need to know the name of what you want etc. No more auto subscribing for you!
Question 3
An Error Queue/Dead Letter Queue is just another queue. You can listen to this queue and do what you need to do with it. But there is not really any out of the box solution that sounds like it would fit your needs.
I've implemented exactly what you describe. Here are some tips based on my experience and related to each of your questions.
Q1 (how to retry X times):
For this, you can use IMessage.Body.BasicProperties.Headers. When you consume a message off an error queue, just add a header with a name that you choose. Look for this header on each message that comes into the error queue and increment it. This will give you a running retry count.
It's very important that you have a strategy for what to do when a message exceeds the retry limit of X. You don't want to lose that message. In my case, I write the message to disk at that point. It gives you lots of helpful debugging information to come back to later, because EasyNetQ automatically wraps your originating message with error info. It also has the original message so that you can, if you like, manually (or maybe automated, through some batch re-processing code) requeue the message later in some controlled way.
You can look at the code in the Hosepipe utility to see a good way of doing this. In fact, if you follow the pattern you see there then you can even use Hosepipe later to requeue the messages if you need to.
Q2 (how to create an error queue per originating queue):
You can use the EasyNetQ Advanced Bus to do this cleanly. Use IBus.Advanced.Container.Resolve<IConventions> to get at the conventions interface. Then you can set the conventions for the error queue naming with conventions.ErrorExchangeNamingConvention and conventions.ErrorQueueNamingConvention. In my case I set the convention to be based on the name of the originating queue so that I get a queue/queue_error pair of queues every time I create a queue.
Q3 (how to process messages in the error queues):
You can declare a consumer for the error queue the same way you do any other queue. Again, the AdvancedBus lets you do this cleanly by specifying that the type coming off of the queue is EasyNetQ.SystemMessage.Error. So, IAdvancedBus.Consume<EasyNetQ.SystemMessage.Error>() will get you there. Retrying simply means republishing to the original exchange (paying attention to the retry count you put in the header (see my answer to Q1, above), and information in the Error message that you consumed off the error queue can help you find the target for republishing.
I know this is an old post but - just in case it helps someone else - here is my self-answered question (I needed to ask it because existing help was not enough) that explains how I implemented retrying failed messages on their original queues. The following should answer your question #1 and #3. For #2, you may have to use the Advanced API, which I haven't used (and I think it defeats the purpose of EasyNetQ; one might as well use RabbitMQ client directly). Also consider implementing IConsumerErrorStrategy, though.
1) Since there can be multiple consumers of a message and all may not need to retry a msg, I have a Dictionary<consumerId, RetryInfo> in the body of the message, as EasyNetQ does not (out of the box) support complex types in message headers.
public interface IMessageType
{
int MsgTypeId { get; }
Dictionary<string, TryInfo> MsgTryInfo {get; set;}
}
2) I have implemented a class RetryEnabledErrorMessageSerializer : IErrorMessageSerializer that just updates the TryCount and other information every time it is called by the framework. I attach this custom serializer to the framework on a per-consumer basis via the IoC support provided by EasyNetQ.
public class RetryEnabledErrorMessageSerializer<T> : IErrorMessageSerializer where T : class, IMessageType
{
public string Serialize(byte[] messageBody)
{
string stringifiedMsgBody = Encoding.UTF8.GetString(messageBody);
var objectifiedMsgBody = JObject.Parse(stringifiedMsgBody);
// Add/update RetryInformation into objectifiedMsgBody here
// I have a dictionary that saves <key:consumerId, val: TryInfoObj>
return JsonConvert.SerializeObject(objectifiedMsgBody);
}
}
And in my EasyNetQ wrapper class:
public void SetupMessageBroker(string givenSubscriptionId, bool enableRetry = false)
{
if (enableRetry)
{
_defaultBus = RabbitHutch.CreateBus(currentConnString,
serviceRegister => serviceRegister.Register<IErrorMessageSerializer>(serviceProvider => new RetryEnabledErrorMessageSerializer<IMessageType>(givenSubscriptionId))
);
}
else // EasyNetQ's DefaultErrorMessageSerializer will wrap error messages
{
_defaultBus = RabbitHutch.CreateBus(currentConnString);
}
}
public bool SubscribeAsync<T>(Func<T, Task> eventHandler, string subscriptionId)
{
IMsgHandler<T> currMsgHandler = new MsgHandler<T>(eventHandler, subscriptionId);
// Using the msgHandler allows to add a mediator between EasyNetQ and the actual callback function
// The mediator can transmit the retried msg or choose to ignore it
return _defaultBus.SubscribeAsync<T>(subscriptionId, currMsgHandler.InvokeMsgCallbackFunc).Queue != null;
}
3) Once the message is added to the default error queue, you can have a simple console app/windows service that periodically republishes existing error messages on their original queues. Something like:
var client = new ManagementClient(AppConfig.BaseAddress, AppConfig.RabbitUsername, AppConfig.RabbitPassword);
var vhost = client.GetVhostAsync("/").Result;
var aliveRes = client.IsAliveAsync(vhost).Result;
var errQueue = client.GetQueueAsync(Constants.EasyNetQErrorQueueName, vhost).Result;
var crit = new GetMessagesCriteria(long.MaxValue, Ackmodes.ack_requeue_false);
var errMsgs = client.GetMessagesFromQueueAsync(errQueue, crit).Result;
foreach (var errMsg in errMsgs)
{
var innerMsg = JsonConvert.DeserializeObject<Error>(errMsg.Payload);
var pubInfo = new PublishInfo(innerMsg.RoutingKey, innerMsg.Message);
pubInfo.Properties.Add("type", innerMsg.BasicProperties.Type);
pubInfo.Properties.Add("correlation_id", innerMsg.BasicProperties.CorrelationId);
pubInfo.Properties.Add("delivery_mode", innerMsg.BasicProperties.DeliveryMode);
var pubRes = client.PublishAsync(client.GetExchangeAsync(innerMsg.Exchange, vhost).Result, pubInfo).Result;
}
4) I have a MessageHandler class that contains a callback func. Whenever a message is delivered to the consumer, it goes to the MessageHandler, which decides if the message try is valid and calls the actual callback if so. If try is not valid (maxRetriesExceeded/the consumer does not need to retry anyway), I ignore the message. You can choose to Dead Letter the message in this case.
public interface IMsgHandler<T> where T: class, IMessageType
{
Task InvokeMsgCallbackFunc(T msg);
Func<T, Task> MsgCallbackFunc { get; set; }
bool IsTryValid(T msg, string refSubscriptionId); // Calls callback only
// if Retry is valid
}
Here is the mediator function in MsgHandler that invokes the callback:
public async Task InvokeMsgCallbackFunc(T msg)
{
if (IsTryValid(msg, CurrSubscriptionId))
{
await this.MsgCallbackFunc(msg);
}
else
{
// Do whatever you want
}
}
Here, I have implemented a Nuget package (EasyDeadLetter) for this purpose, which can be easily implemented with the minimum changes in any project.
All you need to do is follow the four steps :
First of all, Decorate your class object with QeueuAttribute
[Queue(“Product.Report”, ExchangeName = “Product.Report”)]
public class ProductReport { }
The second step is to define your dead-letter queue with the same QueueAttribute and also inherit the dead-letter object from the Main object class.
[Queue(“Product.Report.DeadLetter”, ExchangeName =
“Product.Report.DeadLetter”)]
public class ProductReportDeadLetter : ProductReport { }
Now, it’s time to decorate your main queue object with the EasyDeadLetter attribute and set the type of dead-letter queue.
[EasyDeadLetter(DeadLetterType =
typeof(ProductReportDeadLetter))]
[Queue(“Product.Report”, ExchangeName = “Product.Report”)]
public class ProductReport { }
In the final step, you need to register EasyDeadLetterStrategy as the default error handler (IConsumerErrorStrategy).
services.AddSingleton<IBus>
(RabbitHutch.CreateBus(“connectionString”,
serviceRegister =>
{
serviceRegister.Register<IConsumerErrorStrategy,
EasyDeadLetterStrategy>();
}));
That’s all. from now on any failed message will be moved to the related dead-letter queue.
See more detail here :
GitHub Repository
NuGet Package