I have an action which retrieves a file from a SQL database, creates a temporary physical file, and depending on the type of the file either attempts to extract a thumbnail using the ShellFile API, or if it's a PDF, convert the 1st page to a jpg using ImageMagick.
This action is called using an Ajax GET request which receives a base64 representation of the image which is displayed to the user.
Currently, if lets say 5 Ajax requests were made, the server will process each request one at a time. Instead of this behavior, I'd prefer if it processed multiple requests at a time so the user can receive all of the information faster.
So far I've tried making the function use async/await, but it still processes the requests one at a time. Since the APIs I'm using are older and don't support async IO operations, I was following this guide from Microsoft which explains how you can wrap synchronous functions around async functions. The code I have so far is:
public ApiResult<string> RunTask(string fileName, string guid)
{
using (MagickImage image = new MagickImage())
{
using (MemoryStream stream = new MemoryStream())
{
DbFile file = SharedData.Files.GetFileData(Guid.Parse(guid));
file.ShouldNotBeNull();
string path = Server.MapPath($"~/Temp/{fileName}");
System.IO.File.WriteAllBytes(path, file.FileData);
switch (Path.GetExtension(fileName))
{
case ".pdf":
image.Read(path);
break;
default:
ShellFile shellFile = ShellFile.FromFilePath(path);
Bitmap shellThumb = shellFile.Thumbnail.ExtraLargeBitmap;
shellThumb.MakeTransparent(shellThumb.GetPixel(0, 0));
ImageConverter converter = new ImageConverter();
image.Read((byte[])converter.ConvertTo(shellThumb, typeof(byte[])));
break;
}
image.Format = MagickFormat.Png;
image.Write(stream, MagickFormat.Png);
System.IO.File.Delete(path);
return Ok<string>($"data:image/png;base64,{Convert.ToBase64String(stream.ToArray())}");
}
}
}
[HttpGet]
public async Task<ApiResult<string>> GenerateThumbnail(string fileName, string guid)
{
return await Task.Run(() => RunTask(fileName, guid));
}
But when debugging it and having breakpoints print when the start and end of the function are reached, I receive:
START: "file1.pdf"
END: "file1.pdf"
START: "file2.jpg"
END: "file2.jpg"
...
When I'd expect something more akin to:
START: "file1.pdf"
START: "file2.jpg"
...
END: "file1.pdf"
END: "file2.jpg"
How would I fix my code to have the desired behavior? If the function is wrapped by an async one, shouldn't it be running multiple calls together? This mostly comes down to not being very familiar with C# and ASP.net backends. Considering the article from Microsoft's docs that I linked says you generally shouldn't wrap synchronous functions around async ones, I have a feeling there is a very different approach I should take.
(I've also verified the Ajax calls are going off together, and it isn't the source of the bottleneck)
Related
I'm using a MinIO S3 file server to store user data of my blazor server side application. Since users are working with datasets for some ML applications, quite a lot of files need to be updated.
Initial Issue
I'm stating my initial issue, just for completeness and to avoid creating a y/x problem post (However, I'm curious about my attempted solution in any case):
I'm using a hidden <InputFile/> and some javascript to create a drop area for the files the user wants to upload. Problem is that if more then ~200-250 files are dropped (i.e. OnChange event with those files is triggered), the blazor circuit breaks.
Attempted Solution
My solution was rather simple: Force the user to upload zipped files with the completed dataset, if the file count is >150.
New Problem
For long running upload operations I need some user feedback. For single files the upload is fast enough to have some GUI changes after each file, but the MinIO .NET client doesn't seem to have any way to know the progress of a single file upload.
Here is the current code for a single file upload:
public async Task StoreImage(ImageDatabaseModel image, Stream dataStream, string contentType, long fileSize)
{
await CreateBucketIfNotExists(image.UserID);
PutObjectArgs args = new PutObjectArgs()
.WithBucket(image.UserID)
.WithObject(GetMinioImageName(image))
.WithStreamData(dataStream)
.WithContentType(contentType)
.WithObjectSize(fileSize);
await _minio.PutObjectAsync(args);
}
My idea is was now to use the stream to get the progress information. Here is to code of an initial test:
public async Task StoreZibFile(ImageDatabaseModel image, Stream dataStream, string contentType, long fileSize)
{
// never mind I'm still uploading an image, this is only for testing anyway
await CreateBucketIfNotExists(image.UserID);
PutObjectArgs args = new PutObjectArgs()
.WithBucket(image.UserID)
.WithObject(GetMinioImageName(image))
.WithStreamData(dataStream)
.WithContentType(contentType)
.WithObjectSize(fileSize);
var t1 = _minio.PutObjectAsync(args);
var t2 = Task.Run(async () =>
{
while (dataStream.Position < fileSize)
{
Console.WriteLine(dataStream.Position);
await Task.Delay(500);
}
});
await Task.WhenAll(t1, t2);
}
This gives the expected output of the dataStream.Position value rising during the upload.
The question is: Is this approach suitable for my usecase? Are there any downsides I'm unaware of?
While this particular question is about embeddedIO and FileReponseAsync but it probably has more to do with the handling of streams and async in Tasks in C#.
I am using EmbeddedIO (I needed a quick and dirty web server, and so far it has worked like a charm -- however the lack of documentation is a bit frustrating), but I am attempting to return a file with the following code:
var file = new FileInfo(Path.Combine(FileLocations.TemplatePath, templateFile.FilePath));
string fileExtension = Path.GetExtension(templateFile.FilePath);
return this.FileResponseAsync(file, MimeTypes.DefaultMimeTypes.Value.ContainsKey(fileExtension) ?
MimeTypes.DefaultMimeTypes.Value[fileExtension] : "application/octet-stream");
I get the following error:
Message
Failing module name: Web API Module
Cannot access a closed file.
Stack Trace
at System.IO.__Error.FileNotOpen()
Which makes sense, since in the EmbedIO code FileResponseAsync looks like:
using (FileStream fileStream = file.OpenRead())
return context.BinaryResponseAsync(fileStream, ct, useGzip);
and the filestream will be disposed as soon as the BinaryReponse returns. I've solved the problem by changing my code to not dispose of the filestream:
var fileStream = file.OpenRead();
return this.BinaryResponseAsync(fileStream);
While this works, it seems wrong to rely on Garbage Collection to dispose of these files at a later date. How are resources like this (not only in EmbeddedIO but in this modern async world) supposed to be handled?
I am trying to send a continuous stream, from a C# application, to an ASP Core REST API.
I define a continuous stream as for example someone talking into a microphone and the sound being sent directly, without being saved to a local file) to the Rest API to be saved to file.
I have been searching a lot on Google for something like that and so far could not find anything really useful.
I have been trying to emulate it by sending a large file (297MB).
This is what I have so far for the client side:
string TARGETURL = "http://localhost:58000/api/file/";
string filePath = #"G:\Voice\Samples\The Monkey's Paw.wav";
byte[] fileContent = File.ReadAllBytes(filePath);
var dummyStream = new MemoryStream(fileContent);
var inputData = new StreamContent(dummyStream);
HttpResponseMessage response = this._httpClient.PostAsync(TARGETURL, inputData).Result;
HttpContent result = response.Content;
if (response.IsSuccessStatusCode)
{
string contents = result.ReadAsStringAsync().Result;
}
else
{
// do something
}
And for the server side:
[Route("")]
[HttpPost]
public async Task<JsonResult> Post()
{
Dictionary<string, object> rv = new Dictionary<string, object>();
try
{
string file = Path.Combine(#"G:\Voice\Samples\dummy.txt");
using (FileStream fs = new FileStream(file, FileMode.Create, FileAccess.Write,
FileShare.None, 4096, useAsync: true))
{
await Request.Body.CopyToAsync(fs);
}
// complete the transaction
rv.Add("success", true);
rv.Add("error", "");
}
catch(Exception ex)
{
}
return Json(rv);
}
When I am sending the file, the server throw the following exception:
The request's Content-Length 304137380 is larger than the request body size limit 30000000.
I know that I could increase the body size limit, but that's not a longer term solution as the stream length could get longer that any limit I set.
That's why I am trying to find a solution that send the stream by chunks for the server to rebuild and write to a file.
What you probably want to do is use a different network stack. A web application is always going to try and fit everything into HTTP. This is a very specific kind of way to communicate. And REST is built on top of these ideas as well. Things are generally though of as a document with references on the Internet, and REST is an extension to this idea.
It does however sit on top of some other great technologies that might suit your need better.
There's nothing to stop you using the internet, but maybe you need to look at possibly a UDP or TCP level implementation. Be aware that you will still be sending information in packets. There is no such thing as a constant stream of bits on the internet. A sound wave in the real world is an infinite thing, but computers are rubbish at that.
Maybe start by taking a look at using sockets and a library like NAudio.
I am looking into PdfReport.Core and have been asked to let our .NET CORE 2.0 WEB-API return a PDF to the calling client. The client would be any https caller like a ajax or mvc client.
Below is a bit of the code I am using. I am using swashbuckle to test the api, which looks like it is returning the report but when I try to open in a PDF viewer it says it is curropted. I am thinking I am not actually outputting the actual PDF to the stream, suggestions?
[HttpGet]
[Route("api/v1/pdf")]
public FileResult GetPDF()
{
var outputStream = new MemoryStream();
InMemoryPdfReport.CreateStreamingPdfReport(_hostingEnvironment.WebRootPath, outputStream);
outputStream.Position = 0;
return new FileStreamResult(outputStream, "application/pdf")
{
FileDownloadName = "report.pdf"
};
}
I'm not familiar with that particular library, but generally speaking with streams, file corruption is a result of either 1) the write not being flushed or 2) incorrect positioning within the stream.
Since, you've set the position back to zero, I'm guessing the problem is that your write isn't being flushed correctly. Essentially, when you write to a stream, the data is not necessarily "complete" in the stream. Sometimes writes are queued to more efficiently write in batches. Sometimes, there's cleanup tasks a particular stream writer needs to complete to "finalize" everything. For example, with a format like PDF, end matter may need to be appended to the bytes, particular to the format. A stream writer that is writing PDF would take care of this in a flush operation, since it cannot be completed until all writing is done.
Long and short, review the documentation of the library. In particular, look for any method/process that deals with "flushing". That's most likely what your missing.
I'm using the AWS SDK package from Nuget to download files from S3. This involves creating a GetObject request. Amazon has an example of how to do this in their documentation, although I'm actually using the async version of the method.
My code to download a file looks something like this:
using (var client = new AmazonS3Client(accessKey, secretAccessKey, RegionEndpoint.USEast1))
{
var request = new GetObjectRequest
{
BucketName = "my-bucket",
Key = "file.exe"
};
using (var response = await client.GetObjectAsync(request))
{
response.WriteResponseStreamToFile(#"C:\Downloads\file.exe");
}
}
This works; it downloads the file successfully. However, it seems like a little bit of a black box, in that I never really know how long it's going to take to download the file. What I'm hoping to do is get some sort of Progress event so that I can display a nice WPF ProgressBar and watch the download progress. This means I would need to know the size of the file and the number of bytes downloaded, and I'm not sure if there's a way to do that with the AWS SDK.
You can do:
using (var response = client.GetObject(request))
{
response.WriteObjectProgressEvent += Response_WriteObjectProgressEvent;
response.WriteResponseStreamToFile(#"C:\Downloads\file.exe");
}
private static void Response_WriteObjectProgressEvent(object sender, WriteObjectProgressArgs e)
{
Debug.WriteLine($"Tansfered: {e.TransferredBytes}/{e.TotalBytes} - Progress: {e.PercentDone}%");
}
Can you hook in to the WriteObjectProgressEvent object? If you subscribe to events from this object your function will be called multiple times during the download. It will receive the number of bytes that are downloaded/remaining so you can build a progress indicator.