Develop Reference

Stream large file from web api

We have a web API that produces large files (up to 10 GB).
I am building an endpoint that will provide a file to the client.
There is a cloud-front server between the API and the client.
My current implementation has several issues I need to solve.
We are using .NET Core 3.1.
The service is hosted in IIS.
The code in the controller is:
return File(
new FileStream(path, FileMode.Open, FileAccess.Read),
ContentType.ApplicationOctetStream,
filename);
Getting the 504 response from the cloud-front server. The configured timeout is 60 seconds.
Getting out-of-memory exception on the server.
Questions:
Is there anything I need to add to the headers to make it come through the cloud-front server?
Should I use a different result type? I tried PhysicalFile() with the same results.
Are there any settings I should check on the cloud-front side?
Can be the problem on the client side? I have tested that via swagger and postman with the same result.
Is there a way I can limit the amount of memory the endpoint can use? The host machine is very limited in resources.

ClientConnectionFailure at forward-request

I have an Angular Web Application, that is backed by a C# Web Api, which facilitates speaking to an Azure Function App.
An rough example flow is like the following:
Angular Web App (press download with selected parameters) -> send GET request to API Management Service
API Management Service makes call to a C# Web Api
C# Web Api then responds back to the APIM, which in turn calls an Azure Function App to further process
data from an external source
Once a csv is ready, the data payload is downloaded in the browser where the Web App is open
For larger payloads, the download request fails with the following error in Application Insights:
"ClientConnectionFailure at forward-request"
This error occurs at exactly 2 minutes, every time, unless the payload is sufficiently small.
This lead me to believe that the Function App, which I understand as the client in this situation, is timing out, and cancelling the request.
But testing a GET with the exact same parameters through a local instance of the Azure Function App using Postman, the payload is successfully retrieved.
So the issue isn't the Azure Function App, because it did not time out in Postman as when using the WebApp.
This leads me to three different possibilities:
The C# WebApi is timing out and cancelling the request before the APIM can respond in full
The WebApp itself is timing out.
The internet browser (Chrome), is timing out. (Chrome has a hard unchangeable timeout of 5 minutes, so unlikely)
#1. To tackle the the first option, I upgraded the timeout of the HttpClient created in the relevant download action:
public aync Task<HttpResponseMessage> DownloadIt(blah)
{
HttpClient client = getHttpClient();
client.Timeout = TimeSpan.FromMilliseconds(Convert.ToDouble(600000)); // 10 minutes
var request = new HttpRequestMessage(HttpMethod.Get, buildQueryString(blah, client.BaseAddress));
return await client.SendAsync(request);
}
private HttpClient getHttpClient()
{
return _httpClientFactory.CreateClient("blah");
}
This had no effect as the same error was observed.
#2. There are a couple of Timeout properties in the protractor.conf.js, like allScriptsTimeout and defaultTimeoutInterval.
Increasing these had no effect.
** There is a last possibility that the APIM itself is timing out, but looking into the APIM policy for the relevant API, there is no forward-request property, with a timeout, meaning by default according to Microsoft, there is no timeout for the APIM.
https://learn.microsoft.com/en-us/azure/api-management/api-management-advanced-policies
I've tried a few different strategies but to no avail.

Indeed there's a timeout, as ClientConnectionFailure indicates that the client closes the connection with API Management (APIM) while APIM is yet to return a response to it (the client), in this case while it was forwarding the request to the backend(forward-request)
To debug this kind of issues, the best approach is to collect APIM inspector trace to inspect request processing inside APIM pipeline, paying attention to the time spent on each section of the request - Inbound, Backend, Outbound. The section where the most time is spent is probably the culprit (or it's dependencies). Hopefully, this helps you track down the problem.

You can explicitly set a forward-request on the entire function app or a single endpoint such as:
<backend>
<forward-request timeout="1800" />
</backend>
where the time is in seconds (1800*60 = 60 minutes here)
To do this in APIM,
go to your APIM
APIs
Select your function app
Click on the Code icon </> under Inbound Processing
Alternatively, if you want to do this for just a single operation/endpoint, before performing step 4., click on an individual operation/endpoint.

After testing each component of the solution locally (outside Azure), web app (front end), web api, function app (backend), it is clear that the issue was caused by Azure itself, namely the default 4 minutes for Idle Timeout at the Azure Load Balancer.
I double checked by timing the requests that failed and always got 4 minutes.
The way the code in the backend is sending requests is all together, for larger data sets this caused it to hit the load balancer's timeout.
It looks like the load balancer timeout is configurable, but this doesn't look like something I will be able to change.
So solution: Write more efficiet/better code in the backend.

ASP.NET MVC High latency - Delay between ajax request and action invocation because of large file write

Update 2019-04-24
Problem TL;DR one controller call was causing the next few calls to have a ~15s delay before they even reached the controllers.
I've narrowed the cause to a large file write in the request that causes further delay File.WriteAllText(htmlFilePath, reportHTML);
Originial post
The Problem
I have a long running controller request for generating a report. After the report in generated, additional http requests are made to fetch resultant images.
However, the http calls for the images take about 15s between the ajax call in the browser and when the controller action is invoked. After that, the method runs speedily.
Evidence so far
Previously, we used WCF to run the report generation on a separate machine and there was no such delay.
I've tried running both the report generation and image retrieval methods as async calls on their own threads (but on the same machine). However, that still has the delay.
The delay also only happens on the first image request after generating the report. Afterwards, there is no delay.
There is also no session state and disabling session state has no effect
The Ask
Does anyone know what might cause this delay? How can I get better insights into blocks in ASP.NET code or IIS processes?
Other details:
Using CoreHtmlToImage for report generation and azure storage emulator for image storage.
ASP.NET MVC is version 5.2.3 (not core)

Turns out that writing to the website's directory causes the webserver to restart the site
Source: Creating temporary files in wwroot folder ASP.Net MVC3
So if you write files using the assembly working directory like
var uriAssemblyPath = System.Reflection.Assembly.GetExecutingAssembly().CodeBase;
var assemblyPath = new Uri(uriAssemblyPath).LocalPath;
var baseDirectory = System.IO.Path.GetDirectoryName(assemblyPath);
You'll run into this issue.
If you need a consistent directory outside the webroot, you can use the designated temp directory
Path.GetTempPath()
Source: Where can I write a temp file from ASP.NET?

Microsoft Graph API for OneDrive responds with service not available

I am trying out the one drive graph api to upload folder to my one drive folder.
Using the regular upload works fine.
I'm also testing the resumable upload, which is used for large files. But this is where I'm getting a strange response.
I'm following this link for how to do it: https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_createuploadsession.
First i get a create an upload session using "https://graph.microsoft.com/v1.0/me/drive/items/xxxxxxxxxx:/filename.txt:/createUploadSession".
This gives me back an uploadUrl value, something like "https://api.onedrive.com/rup/xxxxxxxxxxxxx"
I then make a PUT request to that URL with the correct headers.
The response I receive is a 400 (bad request) with the following text (including the HTML):
<h2>Our services aren't available right now</h2><p>We're working to restore all services as soon as possible. Please check back soon.</p>Ref A: 235A863C95DC45BE98688D905A7DB3C1 Ref B: BUH01EDGE0107 Ref C: 2018-08-28T18:56:52Z
I have been getting this for 3 days now and I can't seem to get hold of any support from Microsoft. According to this website, everything is running: https://portal.office.com/servicestatus
Does anyone know why I'm getting this error?

I found the cause for the error.
I received the error because I provided the authentication token in the header.
For small file uploads it is required, but for large file uploads it is not required.
I was using the same code for PUT, POST and GET requests, where I only pass in the URL and HTTP Content and i would always add the auth headers. But for large file uploads it is not required.
But still a very strange error response to receive for adding unrequired headers.

ASP.NET Web API StreamContent - make browser show download progress

From an ASP.NET Web Api 2.x controller I'm am serving files using an instance of the StreamContent type. When a file is requested, its blob is located in the database, and a blob stream is opened. The blob stream is then used as input to a StreamContent instance.
Boiled down, my controller action looks similar to this:
[HttpGet]
[Route("{blobId}")]
public HttpResponseMessage DownloadBlob(int blobId)
{
// ... find the blob in DB and open the 'myBlobStream' based on the given id
var result = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StreamContent(myBlobStream)
};
result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
result.Content.Headers.ContentLength = myBlobStream.Length;
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
{
FileName = "foo.txt",
Size = myBlobStream.Length
};
return result;
}
When I hit the endpoint in Chrome (v. 35) it says that it is resolving the host (localhost) and when the file has downloaded it then appears in the download bar. However, I am wondering what is needed to enable Chrome (or any other browser) to show the download progress?
I thought this would be fixed by included the header information like content-type, content-length, and content-disposition, but from what I have tried, that does not make any difference.

Turned out that my implementation was correct. I closed fiddler and everything worked as expected. Don't know if fiddler somehow waits for the entire response to complete before it sends it through its proxy - at least, that would explain why the browser stays in the "resolving host" state until the entire file has been downloaded.

The Web API doesn't "push" information so, unless you have a background thread on your client polling the server for the download status every few seconds or so, this is a bad idea. For a number of reasons in fact:
Increased load on the server to serve multiple requests (imagine if many clients did that at the same time)
Increased data communication from your client (would be important if you were doing this on a mobile phone contract)
etc. (I'm sure I can think of more but it's late)
You might want to consider SignalR for this, although I'm no expert on it. According to the summary in the page I linked:
ASP.NET SignalR is a new library for ASP.NET developers that makes developing real-time web functionality easy. SignalR allows bi-directional communication between server and client. Servers can now push content to connected clients instantly as it becomes available. SignalR supports Web Sockets, and falls back to other compatible techniques for older browsers. SignalR includes APIs for connection management (for instance, connect and disconnect events), grouping connections, and authorization.
If your Web API can allow it, I suppose a potential alternative would be to first send a quick GET request to receive the size of the file you're about to download and store it in your client. In fact, you could utilise the Content-Length header here to avoid the extra GET. Then do your file download and, while it's happening, your client can report the download progress by comparing how much of the file it has received against the full size of the file it got from the server.