Develop Reference

How to trigger retry/deadletter specific events in a batch of events in Azure EventGrid

I've got an Azure app service with a REST API controller that can be called from Azure Event Grid. As you may know, Event Grid can send multiple events in a batch. I don't know under which circumstances this actually happens, because so far the event payload has always been an array consisting of only one single element, but the documentation makes it clear that there can be multiple events in one go.
Let's say I receive five events, and event #3 is somehow incorrect. How do I let Event Grid know that I've accepted four of the five events, and that it should retry or deadletter the third event?
[Route("api/[controller]")]
[ApiController]
public class EventGridController : ControllerBase
{
[HttpPost("CustomerUpdated")]
public async Task<IActionResult> CustomerUpdated()
{
// ... manage subscription validation here
using var reader = new StreamReader(Request.Body, Encoding.UTF8);
var eventStr = await reader.ReadToEndAsync();
var gridEvents = JsonConvert.DeserializeObject<IEnumerable<GridEvent>>(eventStr);
// ... let's say one of five received events is somehow incorrect here
// ... how do I tell event grid which events failed and which events were accepted
}
Currently, I mark the entire batch as failed by returning BadRequest() even if some of the events actually succeeded. It's an okayish "better safe than sorry" trade-off solution because I'm currently the lone ranger on this project and can as such make sure that my code is idempotent. But I want to make sure that some other developer in the future can't make the codebase non-idempotent and have data inconsistencies popping up all over the place because my code tells EventGrid that the entire batch was failed even though some of the events were actually processed successfully. What can I do?

The Azure Event Grid output batching policy is designed as All or None, see more details here.
In the case like you described, the subscriber needs to handle a logic for skipping already successful processed events and the return status code should be for retrying delivery (e.g. 503). Note, that the 400 Bad Request will immediately process of the deadlettering if it is enabled.

How to listen for Pub/Sub messages in an ASP.NET Core app continuously?

I would like to implement an ASP.NET Core API, which is not responding to HTTP requests, but upon startup starts listening to Google Cloud Pub/Sub messages, and it keeps listening indefinitely throughout its lifetime.
What is the preferred way to implement this with the official Pub/Sub SDK?
I can think of two ways:
Approach 1: Just use a SimpleSubscriber, and in the Startup.Configure start listening to messages:
public void Configure(IApplicationBuilder app)
{
var simpleSubscriber = await SimpleSubscriber.CreateAsync(subscriptionName);
var receivedMessages = new List<PubsubMessage>();
simpleSubscriber.StartAsync((msg, cancellationToken) =>
{
// Process the message here.
return Task.FromResult(SimpleSubscriber.Reply.Ack);
});
...
}
Approach 2: Use a library specifically created to periodically run a job, for example Quartz, Hangfire or FluentScheduler, and every time the job is triggered, pull the new messages with a SubscriberClient.
Which one is the preferred approach? The first one seems simpler, but I'm not sure if it's really reliable.

The first approach is definitely how this is intended to be used.
However, see the docs for StartAsync:
Starts receiving messages. The returned Task completes when either
StopAsync(CancellationToken) is called or if an unrecoverable
fault occurs. This method cannot be called more than once per
SubscriberClient instance.
So you do need to handle unexpected StartAsync shutdown on unrecoverable error. The simplest thing to do would be be use an outer loop, although given these errors are considered unrecoverable it is likely something about the call needs to be changed before it can succeed.
The code might look like this:
while (true)
{
// Each SubscriberClientinstance must only be used once.
var subscriberClient = await SubscriberClient.CreateAsync(subscriptionName);
try
{
await subscriberClient.StartAsync((msg, cancellationToken) =>
{
// Process the message here.
return Task.FromResult(SimpleSubscriber.Reply.Ack);
});
}
catch (Exception e)
{
// Handle the unrecoverable error somehow...
}
}
If this doesn't work as expected, please let us know.
Edit: SimpleSubscriber was renamed to SubscriberClient in the library so the answer has been edited accordingly.

How to reprocess a deadlocked action from http request

The thing is that SQL Server sometimes chooses a session as its deadlock victim when 2 processes lock each other out. The one process does an update and the other just a read. During read SQL Server creates so called 'shared locks' which does not block other reader but does block updaters. So far the only way to solve this is to reprocess the victimized thread.
Now this is happening in a web application and I would like to have a mechanism that can do the reprocessing (let's say with a maximum of 5 times) when needed.
I've looked at the IHttpModule which has a BeginRequest() and EndRequest() event being called (amongst other events) but that does not give me the ability to reprocess the request.
In fact what I need is something that forces itself between the http handler and the process being called.
I could write something like this:
int maxtries = 5;
while(maxtries > 0)
{
try
{
using(var scope = Session.OpenTransaction())
{
// process
scope.Complete(); // commit
return result;
}
}
catch(DeadlockException dlex)
{
maxtries--;
}
catch(Exception ex)
{
throw;
}
}
but I would have to write that for all requests which is tedious and error prone. I would be nice if I could just configure a kind of reprocessing handler via the Web.Config that is automatically called and does the processing deadlock reprocessing for me.

If your getting deadlocks you've got something wrong in your DB layer. You missing indices or something similar, or you are doing out of sequence updates within transactions that are locking dependant entities.
Regardless using HTTP as a mechanism to handle this error is not the way to go.
If you truly need to retry a deadlock, then you should wrap the attempt in your own function and retry almost exactly as you describe above.
BUT I would strongly suggest that you identify the cause of the deadlock and resolve it.
Hope that does not sound too dismissive of your problem, but fix the cause of the problem not the symptoms.

Since you're using MVC and assuming it is safe to rerun your entire action on DB failure, you can simply write a common base controller class from which all of your controllers will inherit (if you already don't have one), and in it override OnActionExecuting and trap specific exception(s) and retry. This way you'll have the code only in one place, but, again, assuming it is safe to rerun the entire action in such case.
Example:
public abstract class MyBaseController : Controller
{
protected override void OnActionExecuting(
ActionExecutingContext filterContext
)
{
int maxtries = 5;
while(maxtries > 0)
{
try
{
return base.OnActionExecuting(filtercontext);
}
catch(DeadlockException dlex)
{
maxtries--;
}
catch(Exception ex)
{
throw;
}
}
throw new Exception("Persistent DB locking - max retries reached.");
}
}
... and then simply update every relevant controller to inherit from this controller (again, if you don't already have a common controller).
EDIT: B/w, Bigtoe's answer is correct - deadlock is the cause and should be dealt with accordingly. The above solution is really a workaround if DB layer cannot be reliably fixed. The first attempt should be on reviewing and (re-)structuring queries so as to avoid deadlock in the first place. Only if that is not practical should the above workaround be employed.

Rebus pub/sub: async way to reply messages from subscribers?

I have been trying to adapt Rebus in one of my applications. easy to configure, works well everything. Have to implement PUB/SUB communication to achieve responses from multiple sources.
so what I made is,
Saga(Publisher)
SearchProductSaga : Saga<ProductSagaData>, IAmInitiatedBy<SearchProduct>, IHandleMessages<SearchStarted>, IHandleMessages<SearchProductResponse>, IHandleMessages<SearchCompleted>
Input Queue for Saga is - ProductSaga.Queue
Subscriber 1
contains following sequence of execution:
public class ProductHanderl_1 : IHandleMessage<SearchProduct>
{
public void Handle(FullTextSearchProductRequest message)
{
Bus.Reply(SearchStarted);
//Some business logic to find products
Bus.Reply(AcutalProductResponse);
Bus.Reply(SearchCompleted);
}
}
Subscriber 2
contains same sequence of execution but a different business logic:
public class ProductHanderl_2 : IHandleMessage<SearchProduct>
{
public void Handle(FullTextSearchProductRequest message)
{
Bus.Reply(SearchStarted);
//Some business logic to find products
Bus.Reply(AcutalProductResponse);
Bus.Reply(SearchCompleted);
}
}
Now, after this implementation, what I was expecting is:
I should be able to calculate number of executing subscribers right now, by receiving SearchStarted messages to SearchProductSaga;
and once subscribers done with business logic, would send SearchCompleted message to indicate saga - we are done. And execute MarkAsComplete(); on saga.
But result I'm getting is quite disappointed. What I found is, from handler if you are replying multiple times(like execution sequence in my subscriber logic), all messages are sent together to publisher queue, once handler execution scope ends.
Correct if I'm wrong, and suggest any solution if anyone has. I could achieve same with threading. But I don't want to manage it myself, so is there any asynchronous way to push messages to queue as and when replied from code.

What you're experiencing is a consequence of the fact that a message is handled in a queue transaction in which all outgoing messages are sent as well.
This means that all sent messages, even though they may have been delivered to whichever queueing system you're using, will not be delivered to anyone until the transaction is committed.
This also means that you'd have to divide your saga actions into multiple discrete steps in order to achieve what you're after.
Does that make sense?

Exceptions in validation

I currently have some code that delibratly throws an exception if the user sends me data that fails validation (see below). I like it because im sure any errors in the application are caught and handled. A am however worried the code being slow as throwing exceptions takes a lot of memory. Im also worried it might be "bad code". Whats your advice? Thanks
public class BTAmendAppointmentRequest
{
public DataLayer.WebserviceMessage AddBTAmendAppointmentRequest(DataLayer.BTAmendAppointmentRequest req)
{
DataLayer.WebserviceMessage rsp = new DataLayer.WebserviceMessage();
try
{
if (!String.IsNullOrEmpty(req.AppointmentReference))
req.AppointmentReference = req.AppointmentReference.Trim();
if (req.OrderRequestID < 1 || string.IsNullOrEmpty(req.AppointmentReference))
{
throw new Exception("Amend appointment failed, you must supply a valid appointment reference and order reference");
}
...Do other stuff
}
catch (Exception ex)
{
rsp = new Service.WebserviceErrorMessage(ex);
}
return rsp;
}
}

If you are expecting these errors, you should return error messages to the user, not throw exceptions.
Reserve exceptions to exceptional situations.
Apart from being expensive, the meaning of an exception, the semantics are that of something exceptional having happened. Validation failing is not exceptional, it is expected.
Having said that, seeing as you are on a web service, an exception is a reasonable thing to do, assuming you also validate before the service call. It is reasonable since a web service can be called by anything - validation may not have happened, and such errors should be exceptional. Additionally, at least with .NET web services, web exceptions are probably the best way to communicate such things back to the client.

Exceptions should be considered as last resort error trap. They should be "exceptional". Data input errors are not exceptions - they are very common, expected events. You shoudl handle validation issues with validation controls or processes, that handle them - display an error message and do not let the processing continue.
Your other problem is that you cannot easily do full form validation if the first error you encounter throws an exception. If I was filling out a form where each error was separately highlighted, I would give up very quickly. You need to be able to validate and display ALL errors on a page, and not permit progress without validation succeeding.

I tend to agree with Oded in that exceptions should only be used for stuff you aren't expecting. The other way to look at it is with using an errors collection, you are able to validate a larger batch instead of throwing an exception on the first problem. This can be more usable for the person consuming your service.
In the case of web services, I would package the entire response in a custom response object, which features a return code. This allows you to have a return code of error, and then encapsulate an errors collection in the response object.