Develop Reference

Trying to stream direct to AWS.S3 .NET Core

Context: I've been trying to find a "proper" solution to upload a file throw an API endpoint and upload the stream from request.Body without using the "IFormFile" so that I will have no data binding or load on the memory(if possible).
I tried https://trailheadtechnology.com/dos-and-donts-for-streaming-file-uploads-to-azure-blob-storage-with-net-mvc/ and this https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-6.0#upload-large-files-with-streaming
but I got stuck on issues like this "Multipart body length limit 16384 exceeded" or MultipartReader with currentStream = 0.
Next thing I want to try is to somehow get from request.Body the stream and only send the stream without boundary and those headers.
My question is: What would be a better solution for streaming file upload to AWS.S3 of big files without having big load on memory?

Uploading a file through ASP MVC asyncronously?

Is it possible to upload a file asynchronously to a website using ASP MVC 4?
In my current solution, when the user uploads a file, the post-back checks the number of bytes in the file stream - if it's greater than 8024 bytes, we put an error in TempData and return back to the same page indicating that a file is too large. If it is under 8024, then I create a new Task that will funnel bytes from the FileStream into a new destination file, somewhere on the server.
However, my boss correctly pointed out that at that point it may be too late to warn the user that their file is too large.
We were wondering whether it was possible to check the length of the file on the server side before reading any bytes from the user (and thus without the user having to wait for the file to fully upload before finding out that their file is too large - imagine trying to upload an 8125kb file on a 3MBit connection only to find out that actually the file you gave was 1 byte too large, sorry, try again.
The reason I asked if we could upload a file asynchronously is because this seems very much like asynchronous streaming to me - immediately give a response, regardless of the actual progress of the upload.
We're aware you can use IIS (through the httpHandler attribute in the configuration) to prevent requests over X bytes, but we want to handle error handling ourselves.

How can i store Blob data on Redis witj .NET 4.5 async and streaming?

I need to cache blob data (images) to Redis, and get them with the new Task based API and stream the content. Ensuring that the web server does NOT hold the whole image in memory (as a byte[]).
Any way to get both async and to stream the data back, not getting all data as a byte[]?

You can use ReadAsync/WriteAsync to asynchronously read and write the data in chunks. The byte[] that you will write will not be the whole image but just a buffer to contain the chunks.
Check out http://redis.io/clients for a list of available redis clients. https://github.com/ctstone/csredis looks promising as it already uses Tasks in it's Async interface.

How can i upload file size 1.25GB in ASP.NET MVC

My client is uploading file more then 1 GB through application. I know i can only upload only 100mb using asp.net MVC application.
public static byte[] ReadStream(Stream st)
{
st.Position = 0;
byte[] data = new byte[st.Length];
.
.
.
.
}
i am getting error at byte[] data = new byte[st.Length]; because st.Length=1330768612
Error - "Exception of type 'System.OutOfMemoryException' was thrown."
Is there any way i can upload more then 1gb file?
Why we can define maxRequestLength= 0 - 2097151 in webconfig,

IMO you need to use the right tool for the job. Http was simply not intended to transfer large files like this. Why dont you use ftp instead, and maybe you could then build a web interface around that.

The error shown to you suggests the server has not enough memory to process the file in memory. Validate if your server has enough memory to allocate such a big array/file.
You could also try to process chuncks of the stream. The fact that you get an out of memory suggests that the file is sent to the server, but the server cannot process the file.
I really think it has to do with the size of the array you allocate. It just won't fit in the memory of you machine (of in the memory assigned to .NET).

The error says that you run out of memory while trying to allocate a 1GB byte array in memory. This is not related to MVC. You should also note that the memory limit for 32bit processes is 2GB. If your server runs a 32bit OS and you allocate 1GB of that for a single upload you will quickly deplete the available memory.
Instead of trying to read the entire stream in memory, use Stream.Read to read the data in chuncks using a reasonably sized buffer and store the chuncks to a file stream with a Write call. Not only will you avoid OutOfMemoryExceptions, your code will also run much faster, because you won't have to wait to load the entire 1GB before storing it to a file.
The code can be as simple as this:
public static void SaveStream(Stream st,string targetFile)
{
byte[] inBuffer = new byte[10000];
using(FileStream outStream=File.Create(targetFile,20000))
using (BinaryWriter wr = new BinaryWriter(outStream))
{
st.Read(inBuffer, 0, inBuffer.Length);
wr.Write(inBuffer);
}
}
You can tweak the buffer sizes to balance throughput (how quickly you upload and save) vs scalability (how many clients you can handle).

I remember on an older project we had to work out a way to allow the user to upload a 2-4gb file for our ASP.NET web application.
If I recall correctly, we used the 'File Upload Control' and edited the web.config to allow for a greater file size:
<system.web>
<httpRuntime executionTimeout="2400" maxRequestLength="40960" />
</system.web>
Another option would be to use the HttpModule:
http://dotnetslackers.com/Community/blogs/haissam/archive/2008/09/12/upload-large-files-in-asp-net-using-httpmodule.aspx

I would suggest using FTP or write a small desktop application.
HTTP was never intended to send such large files.

Here's a Microsoft Knowledge base answer for you

C# Response.WriteFile vs Response.TransmitFile filesize issues

I have a 5Mb pdf on the server dowloading this file using a writeFile gives me a 15Mb download, where as the transmitfile gives the correct 5Mb filesize...
Is this due to some sort of uncompression into memory on the server for the writeFile? Just wonder if anyone had seen the same thing happening...
(ps only noticed it since we went to iis7??)
code being...
if (File.Exists(filepath))
{
HttpContext.Current.Response.Clear();
HttpContext.Current.Response.ContentType = "application/octet-stream";
HttpContext.Current.Response.AddHeader("content-disposition","attachment;filename=\""+Path.GetFileName(filepath)+"\"");
HttpContext.Current.Response.AddHeader("content-length", new FileInfo(filepath).Length.ToString());
//HttpContext.Current.Response.WriteFile(filepath);
HttpContext.Current.Response.TransmitFile(filepath);
HttpContext.Current.Response.Flush();
HttpContext.Current.Response.Close();
}

TransmitFile - Writes the specified file directly to an HTTP response output stream without buffering it in memory.
WriteFile - Writes the specified file directly to an HTTP response output stream.
I would say the difference occurs because Transmit file doesn't buffer it. Write file is using buffering (Afiak), basically temporarily holding the data before transmitting it, as such it cannot guess the accurate file size because its writing it in chunks.

You can understand by following definition.
Response.TransmitFile VS Response.WriteFile:
TransmitFile: This method sends the file to the client without loading it to the Application memory on the server. It is the ideal way to use it if the file size being download is large.
WriteFile: This method loads the file being download to the server's memory before sending it to the client. If the file size is large, you might the ASPNET worker process might get restarted.*
Reference :- TransmitFile VS WriteFile