Hey I'm using a ServiceBusTrigger azure function to get the messages received in a queue, and then send them to my webapi which is going to do some stuff with that content
[FunctionName("MyAzureFunction")]
public async void Run(
[ServiceBusTrigger("<MyQueue>", Connection = "<MyConnectionString>")] Message myQueueItem, ILogger log)
{
log.LogInformation($"C# ServiceBus queue trigger function processed message: {myQueueItem.ToString()}");
var client = new HttpClient();
// Retrieving the string content from the message
var bodyMessage = Encoding.UTF8.GetString(myQueueItem.Body);
// Calling my API to do something based on the message content
var response = await client.PostAsync("<MyAPIUrl>", new StringContent(bodyMessage, Encoding.UTF8, "application/json"));
// doing something based on the response
}
I've been reading about azure functions and in order to it gets cheaper I read about durable functions, I'm looking forward how to use them so I can take decisions based on my response and I can get it working with this ServiceBusTrigger
Needs to be changed the current ServiceBusTrigger function so it calls another Function that will actually do the job:
[FunctionName("MyAzureFunction")] public async void Run(
[ServiceBusTrigger("<MyQueue>", Connection = "<MyConnectionString>")] Message myQueueItem,
[DurableClient] IDurableOrchestrationClient orchestratorClient,
ILogger log) {
log.LogInformation($"C# ServiceBus queue trigger function processed message: {myQueueItem.ToString()}");
// Here is where you need to specify in the first parameter the name of the function to be called
// and the last parameter are the params you'll send to that one
var instanceId = await orchestratorClient.StartNewAsync("MyPostFunction", null, myQueueItem);
log.LogInformation($"C# ServiceBus queue trigger function created an async instance of 'MyPostFunction' with the ID: {instanceId}");
}
Then is needed to create another function that will be OrchestrationTrigger type, that will look like this:
[FunctionName("MyPostFunction")] public async void RunOrchestrator([OrchestrationTrigger] IDurableOrchestrationContext context) {
// using the context can be retrieved the parammeters passed in the function above
// in this case I just specify the type of that one and that's it
var myQueueItem = context.GetInput<Message>();
var bodyMessage = Encoding.UTF8.GetString(myQueueItem.Body);
// Create a URI of your API url
var postUri = new Uri($"<MyAPIUrl>");
// depending on your WebAPI you'll need to specify the content type in the headers
var headers = new Dictionary<string, StringValues>() { { "Content-Type", "application/json" } };
// creating durable http request
var request = new DurableHttpRequest(HttpMethod.Post, postUri, headers, bodyMessage);
// Doing the http call async, in this context you'll save money since your function will not be completely waiting for a response
// this one will keep just checking to see if there's a response available or not
var response = await context.CallHttpAsync(request);
// do your stuffs depending in the response
}
In my case I had to specicify the headers in the request, otherwise I used to get 415 Unsupported Media Type can be done in that way, or just creating the request without specifying any header at the begining and then adding those like this:
var request = new DurableHttpRequest(HttpMethod.Post, postUri, null, bodyMessage);
request.Headers.Add("Content-Type", "application/json");
Both options work
Related
I'm using Polly in combination with Microsoft.Extensions.Http.Polly to handle communication with an external API which has rate-limiting (N requests / second).I'm also using .NET 6.
The policy itself works fine for most requests, however it doesn't work properly for sending (stream) data. The API Client requires the usage of MemoryStream. When the Polly policy handles the requests and retries it, the stream data is not sent.
I verified this behavior stems from .NET itself with this minimal example:
using var fileStream = File.OpenRead(#"C:\myfile.pdf");
using var memoryStream = new MemoryStream();
await fileStream.CopyToAsync(memoryStream);
var response = await httpClient.SendAsync(
new HttpRequestMessage
{
// The endpoint will fail the request on the first request
RequestUri = new Uri("https://localhost:7186/api/test"),
Content = new StreamContent(memoryStream),
Method = HttpMethod.Post
}
);
Inspecting the request I see that Request.ContentLength is the length of the file on the first try. On the second try it's 0.
However if I change the example to use the FileStream directly it works:
using var fileStream = File.OpenRead(#"C:\myfile.pdf");
var response = await httpClient.SendAsync(
new HttpRequestMessage
{
// The endpoint will fail the request on the first request
RequestUri = new Uri("https://localhost:7186/api/test"),
Content = new StreamContent(fileStream ),
Method = HttpMethod.Post
}
);
And this is my Polly policy that I add to the chain of AddHttpClient.
public static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
return Policy
.HandleResult<HttpResponseMessage>(response =>
{
return response.StatusCode == System.Net.HttpStatusCode.Forbidden;
})
.WaitAndRetryAsync(4, (retry) => TimeSpan.FromSeconds(1));
}
My question:
How do I properly retry requests where StreamContent with a stream of type MemoryStream is involved, similar to the behavior of FileStream?
Edit for clarification:
I'm using an external API Client library (Egnyte) which accepts an instance of HttpClient
public class EgnyteClient {
public EgnyteClient(string apiKey, string domain, HttpClient? httpClient = null){
...
}
}
I pass an instance which I injected via the HttpContextFactory pattern. This instance uses the retry policy from above.
This is my method for writing a file using EgnyteClient
public async Task UploadFile(string path, MemoryStream stream){
// _egnyteClient is assigned in the constructor
await _egnyteClient.Files.CreateOrUpdateFile(path, stream);
}
This method call works (doesn't throw an exception) even when the API sometimes returns a 403 statucode because the internal HttpClient uses the Polly retry policy. HOWEVER the data isn't always properly transferred since it just works if it was the first attempt.
The root cause of your problem could be the following: once you have sent out a request then the MemoryStream's Position is at the end of the stream. So, any further requests needs to rewind the stream to be able to copy it again into the StreamContent (memoryStream.Position = 0;).
Here is how you can do that with retry:
private StreamContent GetContent(MemoryStream ms)
{
ms.Position = 0;
return new StreamContent(ms);
}
var response = await httpClient.SendAsync(
new HttpRequestMessage
{
RequestUri = new Uri("https://localhost:7186/api/test"),
Content = GetContent(memoryStream),
Method = HttpMethod.Post
}
);
This ensures that the memoryStream has been rewinded for each each retry attempt.
UPDATE #1
After receiving some clarification and digging in the source code of the Egnyte I think I know understand the problem scope better.
A 3rd party library receives an HttpClient instance which is decorated with a retry policy (related source code)
A MemoryStream is passed to a library which is passed forward as a StreamContent as a part of an HttpRequestMessage (related source code)
HRM is passed directly to the HttpClient and the response is wrapped into a ServiceResponse (related source code)
Based on the source code you can receive one of the followings:
An HttpRequestException thrown by the HttpClient
An EgnyteApiException or QPSLimitExceededException or RateLimitExceededException thrown by the ExceptionHelper
An EgnyteApiException thrown by the SendRequestAsync if there was a problem related to the deserialization
A ServiceResponse from SendRequestAsync
As far as I can see you can access the StatusCode only if you receive an HttpRequestException or an EgnyteApiException.
Because you can't rewind the MemoryStream whenever an HttpClient performs a retry I would suggest to decorate the UploadFile with retry. Inside the method you can always set the stream parameter's Position to 0.
public async Task UploadFile(string path, MemoryStream stream){
stream.Position = 0;
await _egnyteClient.Files.CreateOrUpdateFile(path, stream);
}
So rather than decorating the entire HttpClient you should decorate your UploadFile method with retry. Because of this you need to alter the policy definition to something like this:
public static IAsyncPolicy GetRetryPolicy()
=> Policy
.Handle<EgnyteApiException>(ex => ex.StatusCode == HttpStatusCode.Forbidden)
.Or<HttpRequestException>(ex => ex.StatusCode == HttpStatusCode.Forbidden)
.WaitAndRetryAsync(4, _ => TimeSpan.FromSeconds(1));
Maybe the Or builder clause is not needed because I haven't seen any EnsureSuccessStatusCode call anywhere, but for safety I would build the policy like that.
I am trying to create an azure function that will receive XML files and process them accordingly. I managed to achieve this on my local environment and everything works as intended but when I deployed it to Azure and testing, it comes back with the following 500 internal server error:
Data at the root level is invalid. Line 1, position 1.
Is the exact same xml that works on the local environment, I checked it on the online validators and on visual studio, there is no problem with the xml. Hoovering over the text brings:
Only content-type of application/json is accepted.
I have tried different approaches in code to go around reading the string but with no success:
public class Function1
{
[FunctionName("Function1")]
public static async Task<IActionResult> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
ILogger log)
{
try
{
log.LogInformation("C# HTTP trigger function processed a request.");
//string msg = await new StreamReader(req.Body).ReadToEndAsync();
XmlDocument xmlDocumnet = new XmlDocument();
xmlDocumnet.LoadXml(await new StreamReader(req.Body).ReadToEndAsync());
and
string msg = await new StreamReader(req.Body).ReadToEndAsync();
using (var stream = new MemoryStream())
{
var writer = new StreamWriter(stream);
writer.Write(msg);
writer.Flush();
stream.Position = 0;
xmlDocumnet.Load(stream);
}
I tried to add a header to specify content-type : text/xml but it comes back with a 500:
Only content-type of application/json is accepted.
What would be the best approach in this case? I was thinking to create an intermediary storage which I then would load the file in the function without passing it into the body but that would increase the processing time.
I am trying to upload images to blob from my android app. My current approach is uploading images using azureblobsdk which allows me to upload images directly to the blob but now what I am trying is creating an azure function that would accept data in stream or byte array and will store it to my blob.
while uploading the image from my app I used to send some metadata in the string to my app.
var blobClient = containerClient.GetBlobClient(fileName);
Dictionary<string, string> metadataProperties = new Dictionary<string, string>();
metadataProperties.Add(key, value);
await blobClient.SetMetadataAsync(metadataProperties);
here is what I am doing right now from the app
now I am trying this same thing to do from azure by sending those parameters to the azure function but the problem is I am unable to understand how would I send those streams and metadata to the azure function so that I can process them in my azure function
here is till now what I have done since I am new to azure function so need help to move forward with some approach
[FunctionName("UploadProductImages")]
public async Task<HttpResponseMessage> UploadProductImages([HttpTrigger(AuthorizationLevel.Function, "post", Route = "product/uploadimages")] HttpRequestMessage req, Microsoft.Extensions.Logging.ILogger log)
{
}
now with this function in place how do I access the data sent from my app which would be a list of images with metadata I am even confused about what should I send a stream or byte array
my final target is to get the list of images with some metadata of those images and upload to blob azure blob storage
If I understand the problem correctly, you can do something like this
Create a Model for you data and send it like this
HttpClient apiClient = new HttpClient();
using (var message = new HttpRequestMessage(HttpMethod.Post, functionAppLink))
{
message.Content = new StringContent(JsonConvert.SerializeObject(model),Encoding.UTF8, "application/json");
var response = await apiClient.SendAsync(message);
}
On Function app side just retrieve and deserialize
[FunctionName("UploadProductImages")]
public async Task<HttpResponseMessage> UploadProductImages([HttpTrigger(AuthorizationLevel.Function, "post", Route = "product/uploadimages")] HttpRequestMessage req, Microsoft.Extensions.Logging.ILogger log)
{
using var sr = new StreamReader(req.Body);
var input = await sr.ReadToEndAsync();
var model = JsonConvert.Deserialize<Model>(input);
// Do your thing
}
Very, very new to Azure Functions and getting very frustrated.
All I want to do is execute on a 'get' request from a HttpTriggerFunction and return stream content from the CloudBlobContainer.
I really don't see why this is so hard. Just trying to host a SPA using Azure Functions.
Something like this
public static class UIHandler
{
[FunctionName("UIHandler")]
public static async Task<HttpResponseMessage> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", Route = null)]HttpRequest req,
TraceWriter log,
CloudBlobContainer container)
{
log.Info("C# HTTP trigger function processed a request.");
var stream = await container.GetBlockBlobReference({Infer file name from request here}).OpenReadAsync();
return new HttpResponseMessage()
{
StatusCode = HttpStatusCode.OK,
Content = new StreamContent(stream)
};
}
}
When I try to run this I get the following error.
Run: Microsoft.Azure.WebJobs.Host: Error indexing method
'UIHandler.Run'. Microsoft.Azure.WebJobs.Host: Cannot bind parameter
'container' to type CloudBlobContainer. Make sure the parameter Type
is supported by the binding. If you're using binding extensions (e.g.
ServiceBus, Timers, etc.) make sure you've called the registration
method for the extension(s) in your startup code (e.g.
config.UseServiceBus(), config.UseTimers(), etc.).
I'm using Azure Functions 2. I can't see from the web how to setup the browsing extensions for this. Iv'e also looked into Input and Output bindings. I don't understand what makes a parameter input or output bound when your using C# that only seems to exist in the JSON.
Do I need to corresponding JSON file ? If so what is it called where does it go.
Thanks in Advance
Have a look at Blob Storage Input Binding. The very first sample there shows how to read blob stream, just replace Queue Trigger with HTTP trigger, e.g.
[FunctionName("UIHandler")]
public static async Task<HttpResponseMessage> Run(
[HttpTrigger(AuthorizationLevel.Function, "get", Route = "{name}")] HttpRequest req,
string name,
TraceWriter log,
[Blob("samples-workitems/{name}", FileAccess.Read)] Stream stream)
{
log.Info($"C# HTTP trigger function processed a request for {name}.");
return new HttpResponseMessage()
{
StatusCode = HttpStatusCode.OK,
Content = new StreamContent(stream)
};
}
We are limiting access to an enterprise system by forcing the client to make their CRUD calls through our application, and then our application will forward that very same request to its destination, saving the header information.
Client makes a request to an ApiController
We pass the request to the service layer
The service layer forwards the request its intended enterprise system destination.
To elaborate on the points above:
The client issues a request against this:
[HttpGet]
[Route("opportunities({id:guid})")]
[Route("opportunities")]
public async Task<HttpResponseMessage> GetOpportunity()
{
var query = Request.RequestUri.AbsolutePath.Split('/').Last() + Request.RequestUri.Query;
var response = await _opportunityService.GetOpportunity(query);
return response;
}
The service method GetOpportunity is defined as:
public async Task<HttpResponseMessage> GetOpportunity(string query)
{//at the line below is where i want to send the same headers that were passed in originally at step 1
var response = Client.Instance.GetAsync(Client.Instance.BaseAddress + query); //this is just using HttpClient to make this call
var responseType = response.Result.StatusCode;
if (responseType == HttpStatusCode.NotFound)
return new HttpResponseMessage
{
StatusCode = responseType
};
return await response;
}
How do we save the header information from Step 1?
By using the following middleware I have been able to grab ALL header information; however, I am not sure on how to cache them or make them available to the service layer:
public class HeaderAuthenticationAttribute : ActionFilterAttribute
{
public override void OnActionExecuting(HttpActionContext actionContext)
{
var headers = actionContext.Request.Headers;
}
}
You are basically acting as a proxy. I see a few possible options.
One would be to pass the Original Request into the service as an explicit dependency
[HttpGet]
[Route("opportunities({id:guid})")]
[Route("opportunities")]
public async Task<HttpResponseMessage> GetOpportunity() {
var response = await _opportunityService.GetOpportunity(this.Request);
return response;
}
and extract the information there
public async Task<HttpResponseMessage> GetOpportunity(HttpRequestMessage Request) {
//at the line below is where i want to send the same headers that were passed in originally at step 1
var query = Request.RequestUri.AbsolutePath.Split('/').Last() + Request.RequestUri.Query;
var headers = Request.Headers;
var url = Client.Instance.BaseAddress + query;
//create new request and copy headers
var proxy = new HttpRequestMessage(HttpMethod.Get, url);
foreach (var header in headers) {
proxy.Headers.Add(header.Key, header.Value);
}
var response = await Client.Instance.SendAsync(proxy);//This is an assumption.
var responseType = response.StatusCode; //Do not mix blocking calls. It can deadlock
if (responseType == HttpStatusCode.NotFound)
return new HttpResponseMessage {
StatusCode = responseType
};
return response;
}
If you do not want to mix the layers and concerns you can extract the needed information into your own model and pass that to the service in order to recreate the needed request.