I have successfully created an Azure application that sends DbTransactions to a ServiceBus Queue, and then, enqueues a 'notifying message' to a ServiceBus Topic for other clients to monitor (...so they can receive the updates automatically).
Now, I want to use SignalR to monitor & receive the SubscriptionClient messages...and I have test-code that works just fine on its' own.
I have found many examples for sending messages to an Azure Queue (that is easy). And, I have the code to receive a BrokeredMessage from a SubscriptionClient. However, I cannot get SignalR to continuously monitor my Distribute method.
How do I get SignalR to monitor the Topic?
CODE BEHIND: (updated)
public void Dequeue()
{
SubscriptionClient subscription = GetTopicSubscriptionClient(TOPIC_NAME, SUBSCRIPTION_NAME);
subscription.Receive();
BrokeredMessage message = subscription.Receive();
if (message != null)
{
try
{
var body = message.GetBody<string>();
var contextXml = message.Properties[PROPERTIES_CONTEXT_XML].ToString();
var transaction = message.Properties[PROPERTIES_TRANSACTION_TYPE].ToString();
Console.WriteLine("Body: " + body);
Console.WriteLine("MessageID: " + message.MessageId);
Console.WriteLine("Custom Property [Transaction]: " + transaction);
var context = XmlSerializer.Deserialize<Person>(contextXml);
message.Complete();
Clients.All.distribute(context, transaction);
}
catch (Exception ex)
{
// Manage later
}
}
}
CLIENT-SIDE CODE:
// TEST: Hub - GridUpdaterHub
var hubConnection = $.hubConnection();
var gridUpdaterHubProxy = hubConnection.createHubProxy('gridUpdaterHub');
gridUpdaterHubProxy.on('hello', function (message) {
console.log(message);
});
// I want this automated
gridUpdaterHubProxy.on('distribute', function (context, transaction) {
console.log('It is working');
});
connection.start().done(function () {
// This is successful
gridUpdaterHubProxy.invoke('hello', "Hello");
});
I would not do it like that. Your code is consuming and retaining ASP.NET thread pool's threads for each incoming connection, so if you have many clients you are not scaling well at all. I do not know the internals of SignalR that deep, but I'd guess that your never-ending method is preventing SignalR to let the client call your callbacks because that needs the server method to end properly. Just try to change while(true) with something exiting after, let's say, 3 messages in the queue, you should be called back 3 times and probably those calls will happen all together when your method exits.
If that is right, then you can move to something different, like dedicating a specific thread to consuming the queue and having callbacks called from there usning GlobalHost.ConnectionManager.GetHubContext. Probably better, you could try a different process consuming the queue and doing HTTP POST to your web app, which in turns broadcasts to the clients.
Related
I have two Azure Functions. One is HTTP triggered, let's call it the API and the other one ServiceBusQueue triggered, and let's call this one the Listener.
The first one (the API) puts an HTTP request into a queue and the second one (the Listener) picks that up and processes that. The functions SDK version is: 3.0.7.
I have two projects in my solution for this. One which contains the Azure Functions and the other one which has the services. The API once triggered, calls a service from the other project that puts the message into the queue. And the Listener once received a message, calls a service from the service project to process the message.
Any long-running process?
The Listener actually performs a lightweight workflow and it all happens very quickly considering the amount of work it executes. The average time of execution is 90 seconds.
What's the queue specs?
The queue that the Listener listens to and is hosted in an Azure ServiceBus namespace has the following properties set:
Max Delivery Count: 1
Message time to live: 1 day
Auto-delete: Never
Duplicate detection window: 10 min
Message lock duration: 5 min
And here a screenshot for it:
The API puts the HTTP request into the queue using the following method:
public async Task ProduceAsync(string queueName, string jsonMessage)
{
jsonMessage.NotNull();
queueName.NotNull();
IQueueClient client = new QueueClient(Environment.GetEnvironmentVariable("ServiceBusConnectionString"), queueName, ReceiveMode.PeekLock)
{
OperationTimeout = TimeSpan.FromMinutes(5)
};
await client.SendAsync(new Message(Encoding.UTF8.GetBytes(jsonMessage)));
if (!client.IsClosedOrClosing)
{
await client.CloseAsync();
}
}
And the Listener (the service bus queue triggered azure function), has the following code to process the message:
[FunctionName(nameof(UpdateBookingCalendarListenerFunction))]
public async Task Run([ServiceBusTrigger(ServiceBusConstants.UpdateBookingQueue, Connection = ServiceBusConstants.ConnectionStringKey)] string message)
{
var data = JsonConvert.DeserializeObject<UpdateBookingCalendarRequest>(message);
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarListenerFunction)} picked up a message at {DateTime.Now}. Data: {data}");
await _workflowHandler.HandleAsync(data);
}
The Problem
The Listener function processes the same message 3 times! And I have no idea why! I've Googled and read through a few of StackOverFlow threads such as this one. And it looks like that everybody advising to ensure lock duration is long enough for the process to get executed completely. Although, I've put in 5 minutes for the lock, yet, the problem keeps coming. I'd really appreciate any help on this.
Just adding this in here so might be helpful for some others.
After some more investigations I've realized that in my particular case, the issue was regardless of the Azure Functions and Service Bus. In my workflow handler that the UpdateBookingCalendarListenerFunction sends messages to, I was trying to call some external APIs in a parallel approach, but, for some unknown reasons (to me) the handler code was calling off the external APIs one additional time, regardless of how many records it iterates over. The below code shows how I had implemented the parallel API calls and the other code shows how I've done it one by one that eventually led to a resolution for the issue I had.
My original code - calling APIs in parallel
public async Task<IEnumerable<StaffMemberGraphApiResponse>> AddAdminsAsync(IEnumerable<UpdateStaffMember> admins, string bookingId)
{
var apiResults = new List<StaffMemberGraphApiResponse>();
var adminsToAdd = admins.Where(ad => ad.Action == "add");
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarWorkflowDetailHandler)} Recognized {adminsToAdd.Count()} admins to add to booking with id: {bookingId}");
var addAdminsTasks = adminsToAdd.Select(admin => _addStaffGraphApiHandler.HandleAsync(new AddStaffToBookingGraphApiRequest
{
BookingId = bookingId,
DisplayName = admin.DisplayName,
EmailAddress = admin.EmailAddress,
Role = StaffMemberAllowedRoles.Admin
}));
if (addAdminsTasks.Any())
{
var addAdminsTasksResults = await Task.WhenAll(addAdminsTasks);
apiResults = _populateUpdateStaffMemberResponse.Populate(addAdminsTasksResults, StaffMemberAllowedRoles.Admin).ToList();
}
return apiResults;
}
And my new code without aggregating the API calls into the addAdminsTasks object and hence with no await Task.WhenAll(addAdminsTasks):
public async Task<IEnumerable<StaffMemberGraphApiResponse>> AddStaffMembersAsync(IEnumerable<UpdateStaffMember> members, string bookingId, string targetRole)
{
var apiResults = new List<StaffMemberGraphApiResponse>();
foreach (var item in members.Where(v => v.Action == "add"))
{
_telemetryClient.TrackTrace($"{nameof(UpdateBookingCalendarWorkflowDetailHandler)} Adding {targetRole} to booking: {bookingId}. data: {JsonConvert.SerializeObject(item)}");
apiResults.Add(_populateUpdateStaffMemberResponse.PopulateAsSingleItem(await _addStaffGraphApiHandler.HandleAsync(new AddStaffToBookingGraphApiRequest
{
BookingId = bookingId,
DisplayName = item.DisplayName,
EmailAddress = item.EmailAddress,
Role = targetRole
}), targetRole));
}
return apiResults;
}
I've investigated the first approach and the numbers of tasks were exact match of the number of the IEnumerable input, yet, the API was called one additional time. And within the _addStaffGraphApiHandler.HandleAsync, there is literally nothing than an HttpClient object that raises a POSTrequest. Anyway, using the second code has resolved the issue.
I have created a solution based on Azure Functions and Azure Service Bus, where clients can retrieve information from multiple back-end systems using a single API. The API is implemented in Azure Functions, and based on the payload of the request it is relayed to a Service Bus Queue, picked up by a client application running somewhere on-premise, and the answer sent back by the client to another Service Bus Queue, the "reply-" queue. Meanwhile, the Azure Function is waiting for a message in the reply-queue, and when it finds the message that belongs to it, it sends the payload back to the caller.
The Azure Function Activity Root Id is attached to the Service Bus Message as the CorrelationId. This way each running function knows which message contains the response to the callers request.
My question is about the way I am currently retrieving the messages from the reply queue. Since multiple instances can be running at the same time, each Azure Function instance needs to get it's response from the client without blocking other instances. Besides that, a time out needs to be observed. The client is expected to respond within 20 seconds. While waiting, the Azure Function should not be blocking other instances.
This is the code I have so far:
internal static async Task<(string, bool)> WaitForMessageAsync(string queueName, string operationId, TimeSpan timeout, ILogger log)
{
log.LogInformation("Connecting to service bus queue {QueueName} to wait for reply...", queueName);
var receiver = new MessageReceiver(_connectionString, queueName, ReceiveMode.PeekLock);
try
{
var sw = Stopwatch.StartNew();
while (sw.Elapsed < timeout)
{
var message = await receiver.ReceiveAsync(timeout.Subtract(sw.Elapsed));
if (message != null)
{
if (message.CorrelationId == operationId)
{
log.LogInformation("Reply received for operation {OperationId}", message.CorrelationId);
var reply = Encoding.UTF8.GetString(message.Body);
var error = message.UserProperties.ContainsKey("ErrorCode");
await receiver.CompleteAsync(message.SystemProperties.LockToken);
return (reply, error);
}
else
{
log.LogInformation("Ignoring message for operation {OperationId}", message.CorrelationId);
}
}
}
return (null, false);
}
finally
{
await receiver.CloseAsync();
}
}
The code is based on a few assumptions. I am having a hard time trying to find any documentation to verify my assumptions are correct:
I expect subsequent calls to ReceiveAsync not to fetch messages I have previously fetched and not explicitly abandoned.
I expect new messages that arrive on the queue to be received by ReceiveAsync, even though they may have arrived after my first call to ReceiveAsync and even though there might still be other messages in the queue that I haven't received yet either. E.g. there are 10 messages in the queue, I start receiving the first few message, meanwhile new messages arrive, and after I have read the 10 pre-existing messages, I get the new messages too.
I expect that when I call ReceiveAsync for a second time, that the lock is released from the message I received with the first call, although I did not explicitly Abandon that first message.
Could anyone tell me if my assumptions are correct?
Note: please don't suggest that Durable Functions where designed specifically for this, because they simply do not fill the requirements. Most notably, Durable Functions are invoked by a process that polls a queue with a sliding interval, so after not having any requests for a few minutes, the first new request can take a minute to start, which is not acceptable for my use case.
I would consider session enabled topics or queues for this.
The Message sessions documentation explains this in detail but the essential bit is that a session receiver is created by a client accepting a session. When the session is accepted and held by a client, the client holds an exclusive lock on all messages with that session's session ID in the queue or subscription. It will also hold exclusive locks on all messages with the session ID that will arrive later.
This makes it perfect for facilitating the request/reply pattern.
When sending the message to the queue that the on-premises handlers receive messages on, set the ReplyToSessionId property on the message to your operationId.
Then, the on-premises handlers need to set the SessionId property of the messages they send to the reply queue to the value of the ReplyToSessionId property of the message they processed.
Then finally you can update your code to use a SessionClient and then use the 'AcceptMessageSessionAsync()' method on that to start listening for messages on that session.
Something like the following should work:
internal static async Task<(string?, bool)> WaitForMessageAsync(string queueName, string operationId, TimeSpan timeout, ILogger log)
{
log.LogInformation("Connecting to service bus queue {QueueName} to wait for reply...", queueName);
var sessionClient = new SessionClient(_connectionString, queueName, ReceiveMode.PeekLock);
try
{
var receiver = await sessionClient.AcceptMessageSessionAsync(operationId);
// message will be null if the timeout is reached
var message = await receiver.ReceiveAsync(timeout);
if (message != null)
{
log.LogInformation("Reply received for operation {OperationId}", message.CorrelationId);
var reply = Encoding.UTF8.GetString(message.Body);
var error = message.UserProperties.ContainsKey("ErrorCode");
await receiver.CompleteAsync(message.SystemProperties.LockToken);
return (reply, error);
}
return (null, false);
}
finally
{
await sessionClient.CloseAsync();
}
}
Note: For all this to work, the reply queue will need Sessions enabled. This will require the Standard or Premium tier of Azure Service Bus.
Both queues and topic subscriptions support enabling sessions. The topic subscriptions allow you to mix and match session enabled scenarios as your needs arise. You could have some subscriptions with it enabled, and some without.
The queue used to send the message to the on-premises handlers does not need Sessions enabled.
Finally, when Sessions are enabled on a queue or a topic subscription, the client applications can no longer send or receive regular messages. All messages must be sent as part of a session (by setting the SessionId) and received by accepting the session.
It seems that the feature can not be achieved now.
You can give your voice here where if others have same demand, they will vote up your idea.
First let me explain what I have. I have myself an Azure Service Bus with an Azure Function App. The Service Bus is setup to use SQL Filters to push specific message types into specific topics. Then using my Azure Function App these will get the newest message and then process it.
A basic example
1: I send a request to my EmailAPI
2: EmailAPI then pushing a new message into the Service Bus with a type of "Email"
3: The SQL Filter then sees the type is of "Email" and is placed into the email Topic in the Service Bux
4: The EmailListener Azure Function monitors the Service bus and notices a new message
5: Gather the Service Bus message and process it (basically just send the email using the information provided)
Now let's say for some reason the SMTP server connection is a little broken and some times we get a TimeOutException when attempting to send the email (EmailListener). What happens now when an exception is thrown, the Function App EmailListener will attempt to send it again instantly, no wait, it will just attempt to send it again. It will do this for a total of 10 times and then inform the Service Bus to place the message in the Dead Letter queue.
What I am attempting to do is when an exception is thrown (such as TimeOutException), we wait X amount of time before attempting to process the same message again. I have looked around at many different posts talking about the host.json and attempting to set those settings, but these have not worked. I have found a solution, however the solution requires your to create a clone of the message and push it back into the Service Bus and give it a delayed process time. I would prefer not to implement my own manual delay system, if Azure Service Bus / Function App can deal with retries itself.
The biggest issue I am having (which is probably down to my understanding) is who is at fault? Is it the Service Bus settings to handle the Retry Policy or is it the Azure Function App to deal with attempting to retry after X time.
I have provided a some code, but I feel code isn't really going to help explain my question.
// Pseudo code
public static class EmailListenerTrigger
{
[FunctionName("EmailListenerTrigger")]
public static void Run([ServiceBusTrigger("messages", "email", Connection = "ConnectionString")]string mySbMsg, TraceWriter log)
{
var emailLauncher = new EmailLauncher("SmtpAddress", "SmtpPort", "FromAddress");
try
{
emailLauncher.SendServiceBusMessage(mySbMsg);
}
catch(Exception ex)
{
log.Info($"Audit Log: {mySbMsg}, Excpetion: {ex.message}");
}
}
}
reference one: https://blog.kloud.com.au/2017/05/22/message-retry-patterns-in-azure-functions/ (Thread.Sleep doesn't seem like a good idea)
reference two: https://github.com/Azure/azure-functions-host/issues/2192 (Manually implemented retry)
reference three: https://www.feval.ca/posts/function-queue-retry/ (This seems to refer to queues when I am using topics)
reference four: Can the Azure Service Bus be delayed before retrying a message? (Talks about Defering the message, but then you need to manually get it back out the queue/topic.)
You might be able to solve your issue with the use of Durable Functions. There is for example a built-in method CallActivityWithRetryAsync() that can retry when the activity functions throws an exception.
https://learn.microsoft.com/en-us/sandbox/functions-recipes/durable-diagnostics#calling-activity-functions-with-retry
Your flow would probably something like this:
Service Bus triggered Function. This one starts an Orchestrator Function
The orchestrator calls your activity function (using the aforementioned method)
Your email sending is implemented in an Activity Function and can throw exceptions as needed
While there is no native support for what you want to do, it is still doable without having to do a lot of custom development. You can basically add a service bus output binding to your Azure function, that is connected to the same queue your function consumes messages from. Then, use a custom property to track the number of retries. The following is an example:
private static TimeSpan[] BackoffDurationsBetweenFailures = new[] { }; // add delays here
[FunctionName("retrying-poc")]
public async Task Run(
[ServiceBusTrigger("myQueue")] Message rawRequest,
IDictionary<string, object> userProperties,
[ServiceBus("myQueue")] IAsyncCollector<Message> collector)
{
var request = GetRequest(rawRequest);
var retryCount = GetRetryCount(userProperties);
var shouldRetry = false;
try
{
await _unreliableService.Call(request);
}
catch (Exception ex)
{
// I don't retry if it is a timeout, but that's my own choice.
shouldRetry = !(ex is TimeoutException) && retryCount < BackoffDurationsBetweenFailures.Length;
}
if (shouldRetry)
{
var retryMessage = new Message(rawRequest.Body);
retryMessage.UserProperties.Add("RetryCount", retryCount + 1);
retryMessage.ScheduledEnqueueTimeUtc = DateTime.UtcNow.Add(BackoffDurationsBetweenFailures[retryCount]);
await collector.AddAsync(retryMessage);
}
}
private MyBusinessObject GetRequest(Message rawRequest)
=> JsonConvert.DeserializeObject<MyBusinessObject>(Encoding.UTF8.GetString(rawRequest.Body));
private int GetRetryCount(IDictionary<string, object> properties)
=> properties.TryGetValue("RetryCount", out var value) && int.TryParse(value.ToString(), out var retryCount)
? retryCount
: 0;
Is there a way to invoke a .netcore web api method whenever a message is added to an Azure Servicebus queue ? I would like to implement this without any sort of timer based polling.
I can manually call an api endpoint to process the queue like this:
[HttpGet]
public async Task<IActionResult> ProcessQue()
{
List<string> reList = new List<string>();
try
{
// Register a OnMessage callback
queueClient.RegisterMessageHandler(
async (message, token) =>
{
// Process the message
reList.Add($"Received message: SequenceNumber:{message.SequenceNumber} Body:{message.GetBody<string>()}");
// Complete the message so that it is not received again.
// This can be done only if the queueClient is opened in ReceiveMode.PeekLock mode.
await queueClient.CompleteAsync(message.LockToken);
},
new RegisterHandlerOptions() {MaxConcurrentCalls = 1, AutoComplete = false});
}
catch (Exception exception)
{
Console.WriteLine($"{DateTime.Now} > Exception: {exception.Message}");
}
return Ok(reList);
}
I am looking for a way that this method will fire automatically when a message is added to a queue. Azure functions is probably the right way to do this but I havent been able to connect an azure function to a servicebus queue.
Any suggestions, advise, pocket lint, anything is much appreciated.
You can use Azure Functions for invoking the web api. You need to set the Azure Servicebus queue as the binding. You can get more information on how to use Azure Functions Service Bus bindings on MSDN.
I'm writing a test application with signal r server and a web client and I wanted to know if there is a way to determine or have the server know which transport method the client is establishing with the server.
In regards to websockets which has a persistent two-way connection between the client and server or long polling which keeps polling the server until the server responds and then closes up the connection would there be any downside that I have to be aware of regarding the transport method not being web sockets outside of the persistent two-way connection especially if there are going to be many long running requests being made one after another?
I've noticed that making multiple requests from a client will be handled by the hub and returned when done, example I send a request to wait 10 seconds then a another request to wait 1 second. The Hub will respond to the 1 second wait request first then the 10 second delay, I am curious as to whether there is a thread per request created which is attached to the client via the same persistent duplex connection.
here is my example code.
class Startup
{
public void Configuration(IAppBuilder app)
{
app.UseCors(CorsOptions.AllowAll);
app.MapSignalR();
}
}
public class RunningHub : Hub
{
public void SendLongRunning(string name, string waitFor)
{
Clients.All.addMessage(name, "just requested a long running request I'll get back to you when im done");
LongRunning(waitFor);
Clients.All.addMessage(name, "I'm done with the long running request. which took " + waitFor + " ms");
}
private void LongRunning(string waitFor)
{
int waitTime = int.Parse(waitFor);
Thread.Sleep(waitTime);
}
}
JQuery Sample.
$(function () {
//Set the hubs URL for the connection
$.connection.hub.url = "http://localhost:9090/signalr";
// Declare a proxy to reference the hub.
var signalHub = $.connection.runningHub;
$('#url').append('<strong> Working With Port: ' + $.connection.hub.url + '</strong>');
// Create a function that the hub can call to broadcast messages.
signalHub.client.addMessage = function (name, message) {
//handles the response the message here
};
// Start the connection.
$.connection.hub.start().done(function () {
$('#sendlongrequest').click(function() {
signalHub.server.sendLongRunning($('#displayname').val(), $('#waitTime').val());
});
});
});
For ASP.NET Core;
var transportType = Context.Features.Get<IHttpTransportFeature>()?.TransportType;
Regarding the transport method:
You can inspect HubCallerContext.QueryString param transport:
public void SendLongRunning(string name, string waitFor)
{
var transport = Context.QueryString.First(p => p.Key == "transport").Value;
}
Regarding threading & long-running tasks:
Each request will be handled on a separate thread and the hub pipeline resolves the client-side promise when the hub method completes. This means that you can easily block your connection because of the connection limit in browsers (typically 6 connections at a time).
E.g.: if you use long-polling and you make six requests to the server, each triggering (or directly executing) a long-running operation, then you'll have six pending AJAX requests which only get resolved once the hub method is done, and you won't be able to make any further requests to the server until then. So you should use separate tasks for the long-running code and you should also not await those so the hub dispatcher can send its response without a delay.
If the client needs to know when the long-running task is done, then you should do a push notification from the server instead of relying on the .done() callback.