I'm trying to stream a Video, that is saved as an attachment in a Ravendb-Database, through an ASP.NET MVC 5 Action to a WebBrowser. It is working with the following Code, but the Video gets fully downloaded before the Video starts. I don't get what I'm doing wrong.
I found some ways to do Streaming in MVC, but they seem to expect a Seekable Stream - but the stream I receive from Ravendb is not seekable; it even does not provide a length. So the only way of doing it would be to copy the ravendb-stream to a memorystream and provide a PartialContent or similar from there.
Does anybody have a better solution? I cannot be the only one that wants to stream a Video from a database without loading the full Video into Memory before sending it.
I'm fetching the attachment from ravendb like this:
public async Task<System.IO.Stream> GetAttachmentAsync(IAttachmentPossible attachedObject, string key)
{
using (var ds = InitDatabase())
{
using (var session = ds.OpenAsyncSession())
{
try
{
var result = await session.Advanced.Attachments.GetAsync(attachedObject.Id, key);
return result.Stream;
}
catch (Exception ex)
{
return null;
}
}
}
}
After that I send the stream to the browser like this:
var file = await _database.GetAttachmentAsync(entry, attachmentId);
HttpResponseMessage msg = new HttpResponseMessage(HttpStatusCode.OK);
msg.Content = new StreamContent(file);
msg.Content.Headers.ContentType = new MediaTypeHeaderValue("video/mp4");
return msg;
Any ideas? Thank you very much!
I think that it is correct to copy the stream into a memory stream. With the answer you linked (https://stackoverflow.com/a/39247028/10291808) you can do streaming.
Maybe could be an idea to think about the multiple calls that will done to retrieve the file from the db (maybe you could cache the file for a limited time to improve performance).
Related
I have a large (around 1 GB) database object in a Postgres database.
I can save and load to this database from memory or files BUT:
How do I serve this file to an endpoint a user can request via HttpGet?
When I use await to load from memory it will take a while until I get a response
I use EF Core with .NET 6.0 and steeltoe 3.1.3
Relevant code:
API resource
[HttpGet]
[Route("download")]
public async Task<ActionResult<MemoryStream>> Download(uint oid)
{
MemoryStream zipFileMemoryStream = default(MemoryStream);
try
{
zipFileMemoryStream = await _databaseService.LoadLargeObjectAsync(oid);
}
catch (Exception)
{
return new StatusCodeResult((int)HttpStatusCode.InternalServerError);
}
return zipFileMemoryStream;
}
database load object method:
public async Task<MemoryStream> LoadLargeObjectAsync(uint oid)
{
NpgsqlConnection connection = new NpgsqlConnection(_context.Database.GetConnectionString());
connection.Open();
NpgsqlLargeObjectManager manager = new NpgsqlLargeObjectManager(connection);
MemoryStream memoryStream = new MemoryStream();
using(NpgsqlTransaction transaction = connection.BeginTransaction())
{
using (NpgsqlLargeObjectStream objectStream = await manager.OpenReadAsync(oid))
{
await objectStream.CopyToAsync(memoryStream);
}
}
memoryStream.Position = 0;
return memoryStream;
}
one issue is that when I try downloading the file with postman it takes a while until the file is loaded in memory (of course) and until then, postman will show the 'Sending Request' animation
right up until the return zipFileMemoryStream; statement (again, of course)
where postman finally starts downloading the file and then crashes (I assume postman can't handle 1gb of files per http get).
Do I need to have the object in memory or can I start streaming the content before it is fully loaded/stream the content as I load it?
Which optimizations can I make?
For the 1 GB file my memory spikes up to 2.6 GB, I assume that I make a copy somewhere but I can't put my finger on it.
Thanks in advance
I'm trying to effectively proxy a file upload via an ASP.NET Core 5 MVC controller to another API:
[DisableFormValueModelBinding]
[HttpPost]
public async Task<IActionResult> Upload()
{
var reader = new MultipartReader(Request.GetMultipartBoundary(), Request.Body);
MultipartSection section;
while ((section = await reader.ReadNextSectionAsync().ConfigureAwait(false)) != null)
{
if (section.ContentType == "application/json")
{
await SendFile(section.Body);
}
}
return View("Upload");
}
private async Task SendFile(Stream stream)
{
var request = new HttpRequestMessage(HttpMethod.Post, "http://blah/upload");
request.Content = new StreamContent(stream);
var response = await httpClient.SendAsync(request);
}
However, the receiving API always gets an empty stream.
I can confirm the SendFile method works as the following test works from within the controller method:
using (var fs = new FileStream("test.json", FileMode.Open))
{
await SendFile(fs);
}
And I can see the uploaded file if I try to read it in the controller:
var buf = new char[256];
using (var sr = new StreamReader(section.Body))
{
var x = await sr.ReadAsync(buf, 0, buf.Length);
while (x > 0)
{
log.Debug(new string(buf));
x = await sr.ReadAsync(buf, 0, buf.Length);
}
}
So both ends seem to work, just not together.
I have EnableBuffering set:
app.Use(next => context =>
{
context.Request.EnableBuffering();
return next(context);
});
And I'm disabling binding of the uploaded files to the model using the DisableFormValueModelBindingAttribute example from Upload files in ASP.NET Core
I've also tried rewinding the stream manually using Seek, but it doesn't make a difference.
It works if I copy it through a MemoryStream:
using (var ms = new MemoryStream())
{
await section.Body.CopyToAsync(ms);
await ms.FlushAsync();
ms.Seek(0, SeekOrigin.Begin);
await SendFile(ms);
}
However, this buffers the file in memory which is not suitable for large files.
It also works if I read the uploaded file first, rewind and then try:
var buf = new char[256];
using (var sr = new StreamReader(section.Body))
{
var x = await sr.ReadAsync(buf, 0, buf.Length);
while (x > 0)
{
log.Debug(new string(buf));
x = await sr.ReadAsync(buf, 0, buf.Length);
}
}
section.Body.Seek(0, SeekOrigin.Begin);
// this works now:
await SendFile(section.Body);
Again, this is not suitable for large files.
It seems the stream is not in the correct state to be consumed by my SendFile method but I cannot see why.
UPDATE
Based on comments from Jeremy Lakeman I took a closer look at what was happening with the stream length.
I discovered that removing EnableBuffering makes it work as expected, so the issue is sort of resolved by that.
However, I came across this aspnetcore Github comment where a contributor states that:
We don't support flowing the Request Body through as a stream to HttpClient.
That and the other comments in that issue support Jeremy's comments about CanSeek and the stream length, and it's unclear (to me) whether this should actually work and whether it's just a coincidence that it now does (i.e. will I get hit with another gotcha later).
In this specific scenario with MIME multipart, where we don't know the stream length without buffering/counting the whole file, is there an alternative to StreamContent or a different way to handle the file upload?
The Microsoft docs page Upload files in ASP.NET Core advises only to use an alternative approach. It talks about streaming uploads, however, it stops short of properly consuming the stream and just buffers the file into a MemoryStream (completely defeating the purpose of streaming)
I have a client written in Angular 2, it hits an API that returns a large set of objects. I want to split it up so I can present some data for the user to work with whilst loading the rest in the background.
From looking online at the ways to stream data back in a request the PushStreamContent class seemed the most appropriate (open to better suggestions). I have been reading through this tutorial http://blog.guvweb.co.uk/2014/07/02/streaming-web-api/
I have tried implemented a basic example that will stream a list of numbers so I could test it out.
At the minute I am just hitting the API directly from the browser to see if it returns any data. I get this error message.
Can someone tell me what I am doing wrong?
My Controller Class
[Route("numbers")]
[HttpGet]
public IHttpActionResult StreamData()
{
// Generates a list of numbers 1 --> 1,000,000
List<int> numbers = _dataAccess.GenerateNumbers();
var response = Request.CreateResponse();
response.Content = new PushStreamContent( (stream, Content, context) =>
{
try
{
foreach (var num in numbers)
{
var serializer = new JsonSerializer();
using (var writer = new StreamWriter(stream))
{
serializer.Serialize(writer, num);
stream.Flush();
}
}
}
finally
{
stream.Close();
}
});
return Ok(response);
}
Doesn't mater I found the issue, my own common sense was the culprit!
My browser was downloading the files and I didn't see them, the API stream works just fine!
I have ASP.NET Web API project where a user can download some stuff from a database.
My Download controller fetches data from the database instance. Every single result has a blob field which is some kind of data (1).
I want add each result to a ZIP file (2). After all I send the HTTP response adding my stream content.
List<Result> results = m_Repository.GetResultsForResultId(given_id_by_request);
// 1
foreach (Result result in results)
{
string fileName = String.Format("{0}-{1}.bin", id >> 16, result.Id);
zipFile.AddEntry(fileName, result.Value);
}
// 2
PushStreamContent pushStreamContent = new PushStreamContent((stream, content, context) =>
{
zipFile.Save(stream);
stream.Close();
}
response = new HttpResponseMessage(HttpStatusCode.OK) { Content = pushStreamContent };
It works nice! But on big download requests this exhausts my memory. I need to find a way to put a stream into a zip archive bufferless. Can someone please help me?!
As far as I can see from the code you posted, you are not disposing the streams you create after usage. This can add to a great amount of memory being reserved by your app which might cause your problems.
I am using the ZipArchive to put multiple files into a zip file in my web application. The code Looks somewhat like that:
using (var compressedFileStream = new MemoryStream())
{
using (var zipArchive = new ZipArchive(compressedFileStream, ZipArchiveMode.Update, false))
{
foreach (Result result in results)
{
string fileName = String.Format("{0}-{1}.bin", id >> 16, result.Id);
var zipEntry = zipArchive.CreateEntry(fileName);
using (var originalFileStream = new MemoryStream(result.Value))
{
using (var zipEntryStream = zipEntry.Open())
{
originalFileStream.CopyTo(zipEntryStream);
}
}
}
}
return File(compressedFileStream.ToArray(), "application/zip", string.Format("Download_{0:ddMMyyy_hhmm}.zip", DateTime.Now));
}
I am using that code snippet inside an MVC Controller method so you have to adapt the return part for your situation.
The above code works fine in my application for up to 300 entries or 50MB volume (those are the limits set by the requirements for my app).
Hope that helps you.
EDIT: Forgot the closing bracket of the first using block. the return Statement has to be inside this using-block, else the stream will be disposed.
I'm testing how to upload to AWS using SDK with a sample .txt file from a web app. The file uploads to the Bucket, but the downloaded file from the bucket is just an empty Notepad document without the text from the original uploaded file. I'm new to working with streams, so I'm not sure what could be wrong here. Does anyone see why the data wouldn't be sent in the transfer request? Thanks in advance!
using (var client = new AmazonS3Client(Amazon.RegionEndpoint.USWest1))
{
//Save File to Bucket
using (FileStream txtFileStream = (FileStream)UploadedHttpFileBase.InputStream)
{
try
{
TransferUtility fileTransferUtility = new TransferUtility();
fileTransferUtility.Upload(txtFileStream, bucketLocation,
UploadedHttpFileBase.FileName);
}
catch (Exception e)
{
e.Message.ToString();
}
}
}
EDIT:
Both TransferUtility and PutObjectRequest/PutObjectResponse/AmazonS3Client.PutObject saved a blank text file. Then, after having some trouble instantiating a new FileStream, a MemoryStream used after resetting the starting position to zero still saved a blank text file. Any ideas?
New Code:
using (var client = new AmazonS3Client(Amazon.RegionEndpoint.USWest1))
{
Stream saveableStream = new MemoryStream();
using (Stream source = (Stream)UploadedHttpFileBase.InputStream)
{
source.Position = 0;
source.CopyTo(saveableStream);
}
//Save File to Bucket
try
{
PutObjectRequest request = new PutObjectRequest
{
BucketName = bucketLocation,
Key = UploadedHttpFileBase.FileName,
InputStream = saveableStream
};
PutObjectResponse response = client.PutObject(request);
}
catch (Exception e)
{
e.Message.ToString();
}
}
Most probably that TransferUtility doesn't work good with temporary upload files. Try to copy your input stream somewhere (e.g. into other not-so-temporary file, or even MemoryStream if you're sure it would not give you OutOfMemory at some point). Another thing is to get rid of TransferUtility and use low-level AmazonS3Client.PutObject with which you get finer control over Stream lifetime (do not forget that you'll need to implement some retrying as S3 API is prone to returning random temporary errors).
The answer had something to do with nesting, which is still a little beyond my understanding, and not because the code posted here was inherently wrong. This code came after an initial StreamReader which checked the first line of the text file to determine whether or not to save the file. After moving the code out from the while loop doing the ReadLines, the upload worked. Everything works as it's supposed to now that the validation is reorganized so that there's no need for the nested Stream or MemoryStream.