Is it possible to trigger the same azure function from multiple service bus triggers?
So the scenario I have is that I have two separate service buses that are for all intents and purposes sending the same message just from different sources. I want to know if it is possible to have a single azure function subcribe to both buses.
Some akin to:
public class TheFunction
{
public static async Task Run(
[ServiceBusTrigger("%TopicNameA%", "%SubscriptionNameA%", Connection = "Default")]
[ServiceBusTrigger("%TopicNameB%", "%SubscriptionNameB%", Connection = "Default")] SomeEventData event, ILogger log)
{
// do some shenanigans
}
}
Now I am aware that above isn't possible, but I hope it conveys the general idea of what I would like to achieve.
My fallback is just to have two separate but identical functions where each subscribes to one of the buses and just throws the event to some shared service that does the work.
NOTE: I am unable to send messages from different sources to the same bus. Not impossible, just not feasible to re-jig things within the given timeframe.
Is it possible to trigger the same azure function from multiple
service bus triggers?
So the scenario I have is that I have two separate service buses that
are for all intents and purposes sending the same message just from
different sources. I want to know if it is possible to have a single
azure function subcribe to both buses. Some akin to:
First of all, it is impossible to have multiple triggers in one function of azure function.
Just provide an idea, use the event grid based on the service bus to monitor the information inside the service bus.
Since the azure function can only use the trigger of the service bus but not the inputbinding of the service bus, you need to read the information in the function body.
View the documents below:
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-to-event-grid-integration-concept#azure-portal-instructions
Related
I have an azure function deployed that listens on service bus messages.
Whilst I was testing I realised that the the same azure function instance picked up the service bus message as data wrote to a file I was not expecting.
What I thought would happen is that a new instance of an azure function would launch when each new message is added to the service bus. Am I wrong in this understanding? or can this be achieved and I need to set some type of flag so once my function dequeues a message it stops listening, so any future service bus entries get picked up on a new azure function instance
Thanks
EDIT:
This is the function declaration
[FunctionName("ServiceBusShopifyTrigger")]
public async Task ServiceBusTrigger([ServiceBusTrigger("%serviceBusShopifyExtractQueue%", Connection = "serviceBusExtractConnectionString")] string msg, int deliveryCount, DateTime enqueuedTimeUtc, string messageId)
{
I have placed the following in the start of the function:
string instanceid = Environment.GetEnvironmentVariable("WEBSITE_INSTANCE_ID");
_logger.LogInformation($"****** Orch Shopify - Azure Function Instance ID: {instanceid}");
As I understand this should be unique for each azure instance that spins up. However, I am seeing this as the same each time the function exectues
I also have a HTTP Trigger function and again the Instance ID is the same
Looking at the monitoring I am seeing the following for HTTP Trigger (note the operation ID on closely triggered functions)
These 2 instances were triggered from 2 seperate Http calls
For the service bus monitoring I am seeing different operation ids for closely triggered functions.
Something that I have just noticed in the following:
The calls at 13:26 share the same WEBSITE_INSTANCE_ID
The call at 13:40 share the same WEBSITE_INSTANCE_ID as the calls at 13:26
The call at 15:14 has a different WEBSITE_INSTACE_ID
I have a web application that publishes messages to a topic then several Windows services that subscribe to those topics, some with multiple instances. If the services are running when the messages are published everything works correctly but if they are not then the messages are retained on the queue(s) subscribing to that topic but aren't read when the services start back up.
The desired behavior-
When a message is published to the topic string MyTopic, it is read
from the MyTopicQueue only once. I use some wildcard topics so each message is sent to multiple queues, but multiple instances of a services subscribe to the same topic string and each message should be read by only of those instances
If the subscribers to the MyTopic topic aren't online when the message is published then the messages are retained on MyTopicQueue.
When the Windows services subscribing
to a particularly topic come back on line each retained message is
read from MyTopicQueue by only a single subscriber.
I've found some [typically for IBM] spotty documentation about the MQSUBRQ and MQSO_PUBLICATIONS_ON_REQUEST options but I'm not sure how I should set them. Can someone please help figure out what I need to do to get my desired behavior? [Other than switching back to RabbitMQ which I can't do though I'd prefer it.]
My options:
private readonly int _openOptions = MQC.MQSO_CREATE | MQC.MQSO_FAIL_IF_QUIESCING | MQC.MQSO_MANAGED;
private readonly MQGetMessageOptions _messageOptions = new MQGetMessageOptions()
Code to open the Topic:
_topic = _queueManager.AccessTopic(_settings.TopicString, null,
MQC.MQTOPIC_OPEN_AS_SUBSCRIPTION, _openOptions);
The line of code that reads from the topic (taken from a loop):
_topic.Get(mqMessage, _messageOptions);
If you want the messages to accumulate while you are not connected you need to make the subscription durable by adding MQC.MQSO_DURABLE. In order to be able to resume an existing subscription add MQC.MQSO_RESUME in addition to MQC.MQSO_CREATE.
Be careful with terminology, what you are describing as retained messages is a durable subscription.
Retained publications are something else were MQ can retain one most recently published message on each topic and this message will be retrieved by new subscribers by default unless they use MQSO_NEW_PUBLICATIONS_ONLY to skip receiving the retained publication.
MQSO_PUBLICATIONS_ON_REQUEST allows a subscriber to only receive retained publications on request, it will not receive non-retained publications.
If you want multiple consumers to work together on a single subscription you have two options:
Look at shared subscribers in XMS.NET, look at the CLONESUPP property.
Create a one time durable subscription to a queue on the topics you want consumed, then have your consumers directly consume from the queue not a topic.
Does the durable function awake until activity invoked?
I'm about the implement scheduler, and instead use other library such Hangfire or Quartz. i want to implement durable function that will serve as a scheduler.
And my missing piece is, what happen in the function? does the function got shout until next activity invocation? each one is called execution?
[FunctionName("SchedulerRouter")]
public static async Task<HttpResponseMessage> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")]HttpRequestMessage req,
[OrchestrationClient]DurableOrchestrationClient starter, ILogger log)
{
var data = await req.Content.ReadAsAsync<JObject>();
var instanceId = await starter.StartNewAsync(FunctionsConsts.MAIN_DURABLE_SCHEDULER_NAME, data);
return starter.CreateCheckStatusResponse(req, instanceId);
}
Looks like you are confusing execution time with Max inactivity time is Azure functions:
Durable function is just related to the maximum execution time of a single call. For "out of the box" functions, that timeout is 10min, for durable functions this limitation gets removed. It also introduces support for stateful executions, which means following calls to the same function can share local variables and static members. This is an extension of the "out of the box" functions patterns which needs some additional boiler plate code to make everything working as expected. More details here: https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview
Durable functions and normal functions share the same billing pattern, so cold starts will happen on durable functions as well especially when running in a consumption plan.
Azure functions running in a consumption plan will shutdown during a period of inactivity , and then reallocated and restarted when a new request arrives, this is called: Cold Start. You can mitigate this, building a timer trigger function which awakes your function every 5 to 10 min. But, you will still incurr in cold starts from time to time if your host gets up or down scaled automatically by Azure.
If you want to completely remove the chance of cold starts you will have to move to an App service plan. As a side note, Function apps in Azure are stateless by design, and you should implement your logic with this requirement in mind.
Did you looked into time triggers for AZ Functions? Maybe it is more soutable for you use case. Basically a CRON time tigger that invokes the function according the CRON setting.
The portal example for time trigger
I have a ServiceBusQueue(SBQ), which gets a lots of message payloads.
I have a ServiceBusTrigger(SBT) with accessRights(manage) which continuously polling a message from SBQ.
The problem i am facing is:
My SBT(16 instances at once) pick messages(16 messages individually) at one time and create a request to another server(suppose S1).
If SBT continuously creates 500-600 requests then the server S1 stops to respond.
I am expecting:
I could throttle/restrict to pick the message at once from SBQ so that I indirectly restrict to send the request.
Please share your thoughts, what design should i follow.I couldn't googled the exact solution.
Restrict the maximum concurrent calls of Service Bus Trigger.
In host.json, add configuration to throttle concurrency(i.e. by default 16 messages at once you have seen). Take an example of v2 function.
{
"version": "2.0",
"extensions": {
"serviceBus": {
"messageHandlerOptions": {
"maxConcurrentCalls": 8
}
}
}
}
Restrict Function host instances count. When the host scales out, each instance has one Service Bus trigger which reads multiple messages concurrently as set above.
If the trigger is on dedicated App service plan, scale in the instance counts to some small value. For functions on Consumption plan, add App setting WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT
with reasonable value(<=5). Of course we can set the count to 1 in order to control the behavior strictly.
If we have control over how the messages are sent, schedule the incoming messages to help decrease the request rate.
Use static clients to reuse connection with the Server S1.
I've created an Azure Function that is using a Service Bus queue trigger to run. It's also using a BrokeredMessage as the queue item parameter because I need to get some data out of the custom User Properties of the message.
public async static Task Run([ServiceBusTrigger("myQueue", AccessRights.Manage, Connection = "ConnString")]BrokeredMessage queueItem, TraceWriter log)
{
string myProperty = queueItem.Properties["MyProperty"].ToString();
... // do stuff
}
This all works great and I've deployed it to Azure, but I'd like to test it through the portal. They provide a way to test your function:
But it just asks for the request body. Is there a way to also add User Properties as well?
No, you would have to use some custom code or a tool like ServiceBus Explorer to send a message with metadata properties.