We have a web API that produces large files (up to 10 GB).
I am building an endpoint that will provide a file to the client.
There is a cloud-front server between the API and the client.
My current implementation has several issues I need to solve.
We are using .NET Core 3.1.
The service is hosted in IIS.
The code in the controller is:
return File(
new FileStream(path, FileMode.Open, FileAccess.Read),
ContentType.ApplicationOctetStream,
filename);
Getting the 504 response from the cloud-front server. The configured timeout is 60 seconds.
Getting out-of-memory exception on the server.
Questions:
Is there anything I need to add to the headers to make it come through the cloud-front server?
Should I use a different result type? I tried PhysicalFile() with the same results.
Are there any settings I should check on the cloud-front side?
Can be the problem on the client side? I have tested that via swagger and postman with the same result.
Is there a way I can limit the amount of memory the endpoint can use? The host machine is very limited in resources.
Related
I have a problem when I upload the large file on Azure. I am working on an ASP.NET Core 5.0 API project.
I have implemented functionality regarding Microsoft recommendation. Moreover, I added a pooling mechanism so the frontend application has another endpoint to check upload status.
Everything works fine when I run locally but I have a problem with a large file on Azure. My API is using Azure App Service Premium P1v3. It returns a 502 bad gateway for large files (above 1GB).
I made a tests and 98 % time consuming is reading stream. From Microsft docs it is:
if (MultipartRequestHelper
.HasFileContentDisposition(contentDisposition))
{
untrustedFileNameForStorage = contentDisposition.FileName.Value;
// Don't trust the file name sent by the client. To display
// the file name, HTML-encode the value.
trustedFileNameForDisplay = WebUtility.HtmlEncode(
contentDisposition.FileName.Value);
streamedFileContent =
await FileHelpers.ProcessStreamedFile(section, contentDisposition,
ModelState, _permittedExtensions, _fileSizeLimit);
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
}
I know there is a load balancer timeout of 230 seconds on Azure App Service but when I test it using postman in most cases 502 is being returned after 30 seconds.
Maybe I need to set some configuration feature on Azure App Service? Always on is enabled.
I would like to stay with Azure App Service, but I was thinking about migrating to Azure App service or allow the Frontend application to upload files directly to Azure Blob Storage.
Do you have any idea how to solve it?
Newset
Uploading and Downloading large files in ASP.NET Core 3.1?
The previous answers are based on only using app services, but it is not recommended to store large files in app services. The first is that future updates will become slower and slower, and the second is that the disk space will soon be used up.
So it is recommended to use azure storage. If you use azure storage, suggestion 2 is recommended for larger files to upload large files in chunks.
Preview
Please confirm whether the large file can be transferred successfully even if the error message returns a 500 error.
I have studied this phenomenon before, and each browser is different, and the 500 error time is roughly between 230s-300s. But looking through the log, the program continues to run.
Related Post：
The request timed out. The web server failed to respond within the specified time
So there are two suggestions I give, you can refer to:
Suggestion 1:
It is recommended to create an http interface (assuming the name is getStatus) in your program to receive file upload progress, similar to processbar. When the file starts to transfer, monitor the upload progress, upload the file interface, return HttpCode 201 accept, then the status value is obtained through getStatus, when it reaches 100%, it returns success.
Suggestion 2:
Use MultipartRequestHelper to cut/slice large file. Your usage maybe wrong. Please refer below post.
Dealing with large file uploads on ASP.NET Core 1.0
The version of .net core is inconsistent, but the idea is the same.
Facing similar issue on uploading document of larger size(up to 100MB) through as.net core api hosted as azure app gateway and have set timeout to 10min and applied these attributes on action
[RequestFormLimits(MultipartBodyLengthLimit = 209715200)]
[RequestSizeLimit(209715200)]
Even kestrel has configured to accept 200MB
UseKestrel(options =>
{
options.Limits.MaxRequestBodySize = 209715200;
options.Limits.KeepAliveTimeout = TimeSpan.FromMinutes(10);
});
The file content is in base64 format in request object.
Appreciate if any help on this problem.
I have an Angular Web Application, that is backed by a C# Web Api, which facilitates speaking to an Azure Function App.
An rough example flow is like the following:
Angular Web App (press download with selected parameters) -> send GET request to API Management Service
API Management Service makes call to a C# Web Api
C# Web Api then responds back to the APIM, which in turn calls an Azure Function App to further process
data from an external source
Once a csv is ready, the data payload is downloaded in the browser where the Web App is open
For larger payloads, the download request fails with the following error in Application Insights:
"ClientConnectionFailure at forward-request"
This error occurs at exactly 2 minutes, every time, unless the payload is sufficiently small.
This lead me to believe that the Function App, which I understand as the client in this situation, is timing out, and cancelling the request.
But testing a GET with the exact same parameters through a local instance of the Azure Function App using Postman, the payload is successfully retrieved.
So the issue isn't the Azure Function App, because it did not time out in Postman as when using the WebApp.
This leads me to three different possibilities:
The C# WebApi is timing out and cancelling the request before the APIM can respond in full
The WebApp itself is timing out.
The internet browser (Chrome), is timing out. (Chrome has a hard unchangeable timeout of 5 minutes, so unlikely)
#1. To tackle the the first option, I upgraded the timeout of the HttpClient created in the relevant download action:
public aync Task<HttpResponseMessage> DownloadIt(blah)
{
HttpClient client = getHttpClient();
client.Timeout = TimeSpan.FromMilliseconds(Convert.ToDouble(600000)); // 10 minutes
var request = new HttpRequestMessage(HttpMethod.Get, buildQueryString(blah, client.BaseAddress));
return await client.SendAsync(request);
}
private HttpClient getHttpClient()
{
return _httpClientFactory.CreateClient("blah");
}
This had no effect as the same error was observed.
#2. There are a couple of Timeout properties in the protractor.conf.js, like allScriptsTimeout and defaultTimeoutInterval.
Increasing these had no effect.
** There is a last possibility that the APIM itself is timing out, but looking into the APIM policy for the relevant API, there is no forward-request property, with a timeout, meaning by default according to Microsoft, there is no timeout for the APIM.
https://learn.microsoft.com/en-us/azure/api-management/api-management-advanced-policies
I've tried a few different strategies but to no avail.
Indeed there's a timeout, as ClientConnectionFailure indicates that the client closes the connection with API Management (APIM) while APIM is yet to return a response to it (the client), in this case while it was forwarding the request to the backend(forward-request)
To debug this kind of issues, the best approach is to collect APIM inspector trace to inspect request processing inside APIM pipeline, paying attention to the time spent on each section of the request - Inbound, Backend, Outbound. The section where the most time is spent is probably the culprit (or it's dependencies). Hopefully, this helps you track down the problem.
You can explicitly set a forward-request on the entire function app or a single endpoint such as:
<backend>
<forward-request timeout="1800" />
</backend>
where the time is in seconds (1800*60 = 60 minutes here)
To do this in APIM,
go to your APIM
APIs
Select your function app
Click on the Code icon </> under Inbound Processing
Alternatively, if you want to do this for just a single operation/endpoint, before performing step 4., click on an individual operation/endpoint.
After testing each component of the solution locally (outside Azure), web app (front end), web api, function app (backend), it is clear that the issue was caused by Azure itself, namely the default 4 minutes for Idle Timeout at the Azure Load Balancer.
I double checked by timing the requests that failed and always got 4 minutes.
The way the code in the backend is sending requests is all together, for larger data sets this caused it to hit the load balancer's timeout.
It looks like the load balancer timeout is configurable, but this doesn't look like something I will be able to change.
So solution: Write more efficiet/better code in the backend.
I have an ASP.NET Web API deployed on Azure App service.
I am experiencing following error: For one specific business object my Web API's GET method is returning Internal server error, while for other business objects the same GET method is working fine.
When I debugged my Web API it turned out, that valid business object is returned, but… GET method was triggered multiple times (and on client side I see that it is called only once)
This is an excerpt where Web API is called from client code
// Create HTTP transport objects
HttpRequestMessage httpRequest = new HttpRequestMessage();
httpRequest.Method = HttpMethod.Get;
httpRequest.RequestUri = new Uri(url);
// Set Credentials
if (this.Credentials != null)
{
cancellationToken.ThrowIfCancellationRequested();
await this.Credentials.ProcessHttpRequestAsync(httpRequest, cancellationToken).ConfigureAwait(false);
}
HttpResponseMessage httpResponse = await this.HttpClient.SendAsync(httpRequest, cancellationToken).ConfigureAwait(false);
Besides - if I try to open that same url from browser (e.g.: https://myservice.com/api/businessObject/xxx) - request is performed only once (as it should) and correct results (Json) is displayed in browser.
Any suggestions what to try to figure why call from client side (for specific object) results in multiple Web API service executions and how to fix this?
My Web API service is deriving from System.Web.Http.ApiController
I got some information from exception, but it doesn't seem to be very helpful
Exception thrown: 'Microsoft.Rest.TransientFaultHandling.HttpRequestWithStatusException' in Microsoft.Rest.ClientRuntime.dll The thread 0x27fc has exited with code 0 (0x0)
EDIT
I got some information from Azure Log stream, but that information does not seam to make sense… because this problem happens for one specific business object (and only when requested from my application - not failing from web browser), other business objects are working fine so I don't see how this could be related to access / web.config file...
IIS was not able to access the web.config file for the Web site or application. This can occur if the NTFS permissions are set incorrectly.
IIS was not able to process configuration for the Web site or application.
The authenticated user does not have permission to use this DLL.
..
Ensure that the NTFS permissions for the web.config file are correct and allow access to the Web servers machine account.
Check the event logs to see if any additional information was logged.
Verify the permissions for the DLL.
Install the .NET Extensibility feature if the request is mapped to a managed handler.
Create a tracing rule to track failed requests for this HTTP status code
I have a minimum ASP.NET Handler (.ashx) that returns a PDF file:
public void ProcessRequest(HttpContext context)
{
context.Response.ContentType = "application/pdf";
context.Response.BinaryWrite(File.ReadAllBytes(context.Server.MapPath("~/files/GettingStarted.pdf")));
}
public bool IsReusable
{
get
{
return false;
}
}
}
When I run my web application on IIS Express, the app is hosted at localhost:45050. If I browse to localhost:45050/handler1.ashx on my main development machine, the PDF is downloaded as expected. If I use the DHC Chrome extension (an HTTP client) to perform an HTTP GET on localhost:45050/handler1.ashx, an HTTP 200 OK response code is returned along with the binary data:
HOWEVER, if I run the exact same ASP.NET project on a different machine, I run into bizarre issues. With the project running locally on localhost:45050, I'm still able to browse to localhost:45050/handler1.ashx in Chrome/Firefox/IE to download the file. But, when I use the DHC extension to perform an HTTP GET on localhost:45050/handler1.ashx, there is no response!
I'm able the resolve localhost:45050 (the home page of the site) via DHC on this alternate machine. The server responds with 200 OK and yields the landing page.
But when dealing with the handler that returns binary content, I cannot get any response back from the server with any HTTP client aside from the browser's URL bar. How are browsers able to resolve the HTTP response when standalone HTTP clients cannot? Does anyone have any idea what may be happening here? What would cause behavior to change across machines? I'm trying to handle the response in a JavaScript client, but I'm not getting any data back.
Any help would be greatly appreciated. Thanks!
The top answer here...
Best way to stream files in ASP.NET
...resolved the problem. It seems that writing large files in a single call is a no-no on certain servers. You have to chunk the response manually.
From an ASP.NET Web Api 2.x controller I'm am serving files using an instance of the StreamContent type. When a file is requested, its blob is located in the database, and a blob stream is opened. The blob stream is then used as input to a StreamContent instance.
Boiled down, my controller action looks similar to this:
[HttpGet]
[Route("{blobId}")]
public HttpResponseMessage DownloadBlob(int blobId)
{
// ... find the blob in DB and open the 'myBlobStream' based on the given id
var result = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = new StreamContent(myBlobStream)
};
result.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
result.Content.Headers.ContentLength = myBlobStream.Length;
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
{
FileName = "foo.txt",
Size = myBlobStream.Length
};
return result;
}
When I hit the endpoint in Chrome (v. 35) it says that it is resolving the host (localhost) and when the file has downloaded it then appears in the download bar. However, I am wondering what is needed to enable Chrome (or any other browser) to show the download progress?
I thought this would be fixed by included the header information like content-type, content-length, and content-disposition, but from what I have tried, that does not make any difference.
Turned out that my implementation was correct. I closed fiddler and everything worked as expected. Don't know if fiddler somehow waits for the entire response to complete before it sends it through its proxy - at least, that would explain why the browser stays in the "resolving host" state until the entire file has been downloaded.
The Web API doesn't "push" information so, unless you have a background thread on your client polling the server for the download status every few seconds or so, this is a bad idea. For a number of reasons in fact:
Increased load on the server to serve multiple requests (imagine if many clients did that at the same time)
Increased data communication from your client (would be important if you were doing this on a mobile phone contract)
etc. (I'm sure I can think of more but it's late)
You might want to consider SignalR for this, although I'm no expert on it. According to the summary in the page I linked:
ASP.NET SignalR is a new library for ASP.NET developers that makes developing real-time web functionality easy. SignalR allows bi-directional communication between server and client. Servers can now push content to connected clients instantly as it becomes available. SignalR supports Web Sockets, and falls back to other compatible techniques for older browsers. SignalR includes APIs for connection management (for instance, connect and disconnect events), grouping connections, and authorization.
If your Web API can allow it, I suppose a potential alternative would be to first send a quick GET request to receive the size of the file you're about to download and store it in your client. In fact, you could utilise the Content-Length header here to avoid the extra GET. Then do your file download and, while it's happening, your client can report the download progress by comparing how much of the file it has received against the full size of the file it got from the server.