I'm currently evaluating Azure Functions and I'm trying to find a way/pattern to reliable and idempotent send Emails (and store them in a db). I already read a lot about Sagas, 2PC, Eventual Consistency, but I don't know how to apply these concepts to my situation.
I already have a few business objects stored in a database. Now I would like to add an endpoint which e.g. sends a project summary based on a template. Therefore I created a http triggered function and a CreateEmail method. This is the pseudo code of it
public static async void CreateEmail(QueueClient queue, Guid id)
{
// add the message to the queue, but keep it hidden for 3 min
var sendReceipt = await queue.SendMessageAsync(id.ToString(), TimeSpan.FromSeconds(180))
.ConfigureAwait(false);
//message.PopReceipt is now populated, and only this client can operate on the message until visibility timeout expires
try
{
//Create the mail entity in the db and commit
CreateEmailEntityAndCommit(id);
}
catch (Exception)
{
// Delete the SendMail queue message, because an error occured in db operations
queue.DeleteMessage(sendReceipt.Value.MessageId, sendReceipt.Value.PopReceipt);
throw;
}
// Everything is fine. Mark the message as visible to the email send function
queue.UpdateMessage(sendReceipt.Value.MessageId, sendReceipt.Value.PopReceipt,
visibilityTimeout: TimeSpan.Zero);
}
The code actually does not send the mail, but only creates a database entity and queues a message to the Azure Queue Storage. Another, queue triggered function picks up the messages, sends the mail and updates the status in the db:
public void Run([QueueTrigger("myqueue-items")]string id, ILogger log)
{
if (CheckEmailStatus() == Status.Sent)
{
// Message received twice
return;
}
SendEmail();
UpdateEmailStatus(Status.Sent); // How do we deal with exceptions here? email sent successfully, but status not updated...
}
And here is my problem: If anything goes wrong immediately after sending the mail, the status is not updated. When azure delivers the message again, the Mail would be send again. I guess there is a pattern to avoid such a situation.
Since you are using storage queue, you need to handle idempotency or deduplication at the receiver function end using some identifier of the entity. For example, you can maintain a cache of ids which you can look up to check if currently received id exists, the cache can be set with a reasonable TTL of your desired time window.
Note: Duplicate detection is out of the box in Service Bus queue.
Also you might want to look at Durable Functions.
Related
As per myunderstanding on masstransit
Publish: Sends a message to subscribers .
Send: Used to send messages in fire and forget fashion
Requests: uses request/reply pattern to just send a message and get a response
In my requirement i need to validate my request before calling the Send method. Here the validation should occur at DB level to check for say duplicate records.
I tried to use publish before my send method , but send method doesnt wait for the publish consumer to execute.
My scenario is if (validation is success) proceed with saving data ie the send request job to save data.
So should i use request response pattern here for doing the validation. I am a newbie to masstransit and microservice.
MyTestController{
if(validation success) // how to validate here
Send request to save data.
}
It sounds like you want to validate the data before it is sent out. Something conceptually like this.
class MyTestController
{
// ..
public async Task<IActionResult> Post(SomeData data)
{
if(DataIsValid(data))
_publishEndpoint.Publish(new Message())
}
}
You can validate the data (like null checking) like any other code before you publish. Nothing special here, so I'm guessing its something else.
You want to validate the data using some other data in a database. If its the same database that the website/api is using - that is also not a special thing, so I'm guessing that is not it either.
You want to, some how, validate the data is correct before sending the message. But that you need to use the data of the application that the message is going to. That is typically where I see people get tripped up.
Assuming its something like number three. Let's call the sending service "Service A" and the receiving service "Service B". Today it sounds like you are trying to put the validation in "Service A" but it really has to be in "Service B". What I would do is implement a Saga in "Service B". The first step would be to take request (creating an instance of a saga), then validate the data, then if it passes validation, the saga can take the next step that you want in the process. That should give you what you want in terms of validation before action (we just need to move it to "Service B").
Now "Service B" can expose the state of the Saga at an endpoint like /saga-instance/42 where the controller takes the 42, digs into the database, grabs the saga data and converts it into an API response. Service A can poll that endpoint to get updated status details.
Ultimately, I hope you see that there are a lot of variables at play, but that there is a path forward. You may have to simply adjust where certain actions are taken.
I have created a solution based on Azure Functions and Azure Service Bus, where clients can retrieve information from multiple back-end systems using a single API. The API is implemented in Azure Functions, and based on the payload of the request it is relayed to a Service Bus Queue, picked up by a client application running somewhere on-premise, and the answer sent back by the client to another Service Bus Queue, the "reply-" queue. Meanwhile, the Azure Function is waiting for a message in the reply-queue, and when it finds the message that belongs to it, it sends the payload back to the caller.
The Azure Function Activity Root Id is attached to the Service Bus Message as the CorrelationId. This way each running function knows which message contains the response to the callers request.
My question is about the way I am currently retrieving the messages from the reply queue. Since multiple instances can be running at the same time, each Azure Function instance needs to get it's response from the client without blocking other instances. Besides that, a time out needs to be observed. The client is expected to respond within 20 seconds. While waiting, the Azure Function should not be blocking other instances.
This is the code I have so far:
internal static async Task<(string, bool)> WaitForMessageAsync(string queueName, string operationId, TimeSpan timeout, ILogger log)
{
log.LogInformation("Connecting to service bus queue {QueueName} to wait for reply...", queueName);
var receiver = new MessageReceiver(_connectionString, queueName, ReceiveMode.PeekLock);
try
{
var sw = Stopwatch.StartNew();
while (sw.Elapsed < timeout)
{
var message = await receiver.ReceiveAsync(timeout.Subtract(sw.Elapsed));
if (message != null)
{
if (message.CorrelationId == operationId)
{
log.LogInformation("Reply received for operation {OperationId}", message.CorrelationId);
var reply = Encoding.UTF8.GetString(message.Body);
var error = message.UserProperties.ContainsKey("ErrorCode");
await receiver.CompleteAsync(message.SystemProperties.LockToken);
return (reply, error);
}
else
{
log.LogInformation("Ignoring message for operation {OperationId}", message.CorrelationId);
}
}
}
return (null, false);
}
finally
{
await receiver.CloseAsync();
}
}
The code is based on a few assumptions. I am having a hard time trying to find any documentation to verify my assumptions are correct:
I expect subsequent calls to ReceiveAsync not to fetch messages I have previously fetched and not explicitly abandoned.
I expect new messages that arrive on the queue to be received by ReceiveAsync, even though they may have arrived after my first call to ReceiveAsync and even though there might still be other messages in the queue that I haven't received yet either. E.g. there are 10 messages in the queue, I start receiving the first few message, meanwhile new messages arrive, and after I have read the 10 pre-existing messages, I get the new messages too.
I expect that when I call ReceiveAsync for a second time, that the lock is released from the message I received with the first call, although I did not explicitly Abandon that first message.
Could anyone tell me if my assumptions are correct?
Note: please don't suggest that Durable Functions where designed specifically for this, because they simply do not fill the requirements. Most notably, Durable Functions are invoked by a process that polls a queue with a sliding interval, so after not having any requests for a few minutes, the first new request can take a minute to start, which is not acceptable for my use case.
I would consider session enabled topics or queues for this.
The Message sessions documentation explains this in detail but the essential bit is that a session receiver is created by a client accepting a session. When the session is accepted and held by a client, the client holds an exclusive lock on all messages with that session's session ID in the queue or subscription. It will also hold exclusive locks on all messages with the session ID that will arrive later.
This makes it perfect for facilitating the request/reply pattern.
When sending the message to the queue that the on-premises handlers receive messages on, set the ReplyToSessionId property on the message to your operationId.
Then, the on-premises handlers need to set the SessionId property of the messages they send to the reply queue to the value of the ReplyToSessionId property of the message they processed.
Then finally you can update your code to use a SessionClient and then use the 'AcceptMessageSessionAsync()' method on that to start listening for messages on that session.
Something like the following should work:
internal static async Task<(string?, bool)> WaitForMessageAsync(string queueName, string operationId, TimeSpan timeout, ILogger log)
{
log.LogInformation("Connecting to service bus queue {QueueName} to wait for reply...", queueName);
var sessionClient = new SessionClient(_connectionString, queueName, ReceiveMode.PeekLock);
try
{
var receiver = await sessionClient.AcceptMessageSessionAsync(operationId);
// message will be null if the timeout is reached
var message = await receiver.ReceiveAsync(timeout);
if (message != null)
{
log.LogInformation("Reply received for operation {OperationId}", message.CorrelationId);
var reply = Encoding.UTF8.GetString(message.Body);
var error = message.UserProperties.ContainsKey("ErrorCode");
await receiver.CompleteAsync(message.SystemProperties.LockToken);
return (reply, error);
}
return (null, false);
}
finally
{
await sessionClient.CloseAsync();
}
}
Note: For all this to work, the reply queue will need Sessions enabled. This will require the Standard or Premium tier of Azure Service Bus.
Both queues and topic subscriptions support enabling sessions. The topic subscriptions allow you to mix and match session enabled scenarios as your needs arise. You could have some subscriptions with it enabled, and some without.
The queue used to send the message to the on-premises handlers does not need Sessions enabled.
Finally, when Sessions are enabled on a queue or a topic subscription, the client applications can no longer send or receive regular messages. All messages must be sent as part of a session (by setting the SessionId) and received by accepting the session.
It seems that the feature can not be achieved now.
You can give your voice here where if others have same demand, they will vote up your idea.
We currently have a NServiceBus 5 system, which contains two recurring Sagas. Since they act as dispatcher to periodically pull multiple sorts of data from an external system, we're using the Timeouts to trigger this: We created a generic and empty class called ExecuteTask, which is used by the Saga to handle the timeout.
public class ScheduleSaga1 : Saga<SchedulerSagaData>,
IAmStartedByMessages<StartScheduleSaga1>,
IHandleMessages<StopSchedulingSaga>,
IHandleTimeouts<ExecuteTask>
And the other Saga is almost identically defined:
public class ScheduleSaga2: Saga<SchedulerSagaData>,
IAmStartedByMessages<StartScheduleSaga2>,
IHandleMessages<StopSchedulingSaga>,
IHandleTimeouts<ExecuteTask>
The timeout is handled equally in both Sagas:
public void Handle(StartScheduleSaga1 message)
{
if (_schedulingService.IsDisabled())
{
_logger.Info($"Task '{message.TaskName}' is disabled!");
}
else
{
Debugger.DoDebug($"Scheduling '{message.TaskName}' started!");
Data.TaskName = message.TaskName;
// Check to avoid that if the saga is already started, don't initiate any more tasks
// as those timeout messages will arrive when the specified time is up.
if (!Data.IsTaskAlreadyScheduled)
{
// Setup a timeout for the specified interval for the task to be executed.
Data.IsTaskAlreadyScheduled = true;
// Send the first Message Immediately!
SendMessage();
// Set the timeout
var timeout = _schedulingService.GetTimeout();
RequestTimeout<ExecuteTask>(timeout);
}
}
}
public void Timeout(ExecuteTask state)
{
if (_schedulingService.IsDisabled())
{
_logger.Info($"Task '{Data.TaskName}' is disabled!");
}
else
{
SendMessage();
// Action that gets executed when the specified time is up
var timeout = _schedulingService.GetTimeout();
Debugger.DoDebug($"Request timeout for Task '{Data.TaskName}' set to {timeout}!");
RequestTimeout<ExecuteTask>(timeout);
}
}
private void SendMessage()
{
// Send the Message to the bus so that the handler can handle it
Bus.Send(EndpointConfig.EndpointName, Activator.CreateInstance(typeof(PullData1Request)));
}
Now the problem: Since both Sagas are requesting Timeouts for ExecuteTask, it gets dispatched to both Sagas!
Even worse, it seems like the stateful Data in the Sagas gets messed up, since both Sagas are sending both message.
Therefore, it seems like the Timeouts are getting sent to all the Saga Instances which are requesting it.
But looking at the example https://docs.particular.net/samples/saga/simple/ there is no special logic regarding multiple Saga instances and their state.
Is my assumption correct? If this is the case, what are the best practices to have multiple Sagas requesting and receiving Timeouts?
The only reason I can think of when this is happening is that they share the same identifier to uniquely identify the saga instance.
Both ScheduleSaga1 and ScheduleSaga2 are using the same SchedulerSagaData for storing state. NServiceBus sees an incoming message and tries to retrieve the state, based on the unique identifier in the incoming message. If both StartScheduleSaga1 and StartScheduleSaga2 come in with identifier 1 for example, NServiceBus will search for saga state in the table SchedulerSagaData with unique identifier 1.
Both ScheduleSaga1 and ScheduleSaga2 will then share the same row!!!
Timeouts are based on SagaId in the TimeoutEntity table. Because both sagas share the same SagaId, it's logical they are both executed once the timeout arrives.
At the minimum you should not reuse the identifier to schedule tasks. It's probably better to not share the same class for storing saga state. Also easier to debug.
I am looking for recommendations on how to handle the ability to queue and send different types of emails using Azure WebJobs.
Before sending the email there needs to do some business logic to compose, populate and then send. So say that I have 10 different types of emails
2 examples of emails
1) when a booking is confirmed I would like to something like
var note = new BookingConfirmNotification() { BookingId = 2 }
SomeWayToQueue.Add(note)
2) or when I need reminder the user to do something
var reminder = new ReminderNotification() { UserId = 3 }
SomeWayToQueue.Add(reminder, TimeSpan.FromDays(1)
Is it better to create a queue for each of different types of emails with QueueTrigger on WebJob .. or is there a better way to do this?
Update
So I thought you might be able to add different Function/Methods with different strongly-typed classes to trigger different methods, but that doesnt seem to work.
public void SendAccountVerificationEmail(
[QueueTrigger(WebJobHelper.EmailProcessorQueueName)]
AccountVerificationEmailTask task, TextWriter log)
{
log.WriteLine("START: SendAccountVerificationEmail: " + task.UserId);
const string template = "AccountVerification.cshtml";
const string key = "account-verification";
PrepareAndSend(user.Email, "Account confirmation", template, key, task, typeof(AccountVerificationEmailTask));
log.WriteLine("END: SendAccountVerificationEmail: " + task.UserId);
}
public void SendForgottonPasswordEmail(
[QueueTrigger(WebJobHelper.EmailProcessorQueueName)]
ForgottonPasswordEmailTask task, TextWriter log)
{
const string template = "ForgottonPassword.cshtml";
const string key = "forgotton-password";
PrepareAndSend(user.Email, "Forgotton password", template, key, task, typeof(ForgottonPasswordEmailTask));
}
This doesn't work - different methods are fired randomly when a message is added to the queue
How can I implement something like this using WebJobs?
Are you saying the "compose/populate" logic is somewhat heavy, that's why you want to queue it? What process/entity is creating these "send email" tasks?
I could see you having an Azure Queue where email requests are queued. Assuming you're using the WebJobs SDK (you should be if you're dealing with Azure Storage Queues, etc.) you can have a function monitoring that queue that sends the messages. The WebJobs SDK Extensions include a SendGrid email binding. You can see an example close to what you're describing here. This sample sends emails based on incoming queue messages. Using the SDK in this way, you'll get automatic retry support, poison queue handling, etc.
Regarding your updated question. When multiple consumers consume a queue, they will "round-robin" meaning that the messages are distributed between all the available consumers. This explains why the methods appear to be firing randomly.
Something to think about. You might want to invert things and use queues to capture your user's interactions.
Off the back of that you can perform one or more actions.
For example.
_bus.Send(new BookingCreatedEvent{Ref="SomeRef", Customer="SomeCustomer"});
or
_bus.Send(new BookingCancelledEvent{Ref="SomeRef");
The benefit of this is you can choose what to do on the consuming side. Right now you want to send an email but what if you want to also log to a database or send a record into your CRM?
If you switch to Azure Service Bus Topics/Subscriptions, you can have multiple handlers for the same event.
public static void SendBookingConfirmation([ServiceBusTrigger("BookingCreated","SendConfirmation")] BookingCreatedEvent bookingDetails)
{
// lookup customer details from booking details
// send email to customer
}
public static void UpdateBookingHistory([ServiceBusTrigger("BookingCreated","UpdateBookingHistory")] BookingCreatedEvent bookingDetails)
{
// save booking details to CRM
}
Current Setup includes a windows service which picks up a message from the local queue and extracts the information and puts in to my SQL database.According to my design
Service picks up the message from the queue.(I am using Peek() here).
Sends it to the database.
If for some reason i get an exception while saving it to the database the message is back into the queue,which to me is reliable.
I am logging the errors so that a user can know what's the issue and fix it.
Exception example:If the DBconnection is lost during saving process of the messages to the database then the messages are not lost as they are in the queue.I don't comit untill i get an acknowledgement from the DB that the message is inserted .So a user can see the logs and make sure that the DBconnection exists and every thing would be normal and we dont lose any messages in the queue.
But looking into another scenario:The messages I would be getting in the queue are from a 3rd party according a standard schema.The schema would remain same and there is no change in that.But i have seen some where i get some format exceptions and since its not committed the message is back to the queue.At this point this message would be a bottle neck for me as the same messages is picked up again and tries to process the message.Every time the service would pick up the same message and gets the same exception.So this loops infinitely unless that message is removed or put that message last in the queue.
Looking at removing the message:As of now if i go based on the format exception...then i might be wrong since i might encounter some other exceptions in the future .
Is there a way i can put this messages back to the queue last in the list instead beginning of the queue.
Need some advice on how to proceed further.
Note:Queue is Transactional .
As far as I'm aware, MSMQ doesn't automatically dump messages to fail queues. Either way you handle it, it's only a few lines of code (Bill, Michael, and I recommend a fail queue). As far as a fail queue goes, you could simple create one named .\private$\queuename_fail.
Surviving poison messages in MSMQ is a a decent article over this exact topic, which has an example app and source code at the end.
private readonly MessageQueue _failQueue;
private readonly MessageQueue _messageQueue;
/* Other code here (cursor, peek action, run method, initialization etc) */
private void dumpToFailQueue(Message message)
{
var oldId = message.Id;
_failQueue.Send(message, MessageQueueTransactionType.Single);
// Remove the poisoned message
_messageQueue.ReceiveById(oldId);
}
private void moveToEnd(Message message)
{
var oldId = message.Id;
_messageQueue.Send(message, MessageQueueTransactionType.Single);
// Remove the poisoned message
_messageQueue.ReceiveById(oldId);
}