Develop Reference

C# HttpClient Abruptly stops download

I have this C# Script that is supposed to download a zip archive from GitHub, unpack it and put it in a specific folder:
using (var client = new HttpClient())
{
var filePath = Path.GetFullPath("runtime");#"https://github.com/BlackBirdTV/tank/releases/latest/download/runtime.zip?raw=true";
ConsoleUtilities.UpdateProgress("Downloading Runtime...", 0);
var request = await client.GetStreamAsync(url);
var buffer = new byte[(int)bufferSize];
var totalBytesRead = 0;
int bytes = 0;
while ((bytes = await request.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
totalBytesRead += bytes;
ConsoleUtilities.UpdateProgress($"Downloading Runtime... ({totalBytesRead} of {bufferSize} bytes read) ", (int)(totalBytesRead / bufferSize * 100));
}
}
Decompress(buffer, filePath);
When I now run this, the download starts and it seems like it finishes, yet at a sporadic place it just stops. Somehow, It downloads the bytes as my Console shows, but they are zeroed out. It seems like either my computer receives zeros (which I doubt) or the bytes don't get written to the buffer.
Weirdly enough, downloading the file over the browser works just fine.
Any help is greatly appreciated

As I state in the comments, your problem is that each iteration of your while loop is overwriting your buffer, and you are not accumulating the data anywhere. So your last iteration doesn't completely fill the buffer and all you're left with is whatever data you got in the last iteration.
You could fix that bug by storing the accumulated buffer somewhere, but a far better solution is to not fuss with buffers and such and just use the built-in CopyToAsync method of Stream:
using var client = new HttpClient();
using var stream = await client.GetStreamAsync("https://github.com/BlackBirdTV/tank/releases/latest/download/runtime.zip?raw=true");
using var file = new FileStream(#"c:\temp\runtime.zip", FileMode.Create);
await stream.CopyToAsync(file);
Here I'm saving it to a local file at c:\temp\runtime.zip, but obviously change that to suit your needs. I suppose you're avoiding this method so you can track progress, which is fair enough. So if that's really important to you, read on for a fix to your original solution.
For completeness, here's your original code fixed up to work by writing the buffer to a FileStream:
var bufferSize = 1024 * 10;
var url = #"https://github.com/BlackBirdTV/tank/releases/latest/download/runtime.zip?raw=true";
using var client = new HttpClient();
using var stream = await client.GetStreamAsync(url);
using var file = new FileStream(#"c:\temp\runtime.zip", FileMode.Create);
var filePath = Path.GetFullPath("runtime");
var request = await client.GetStreamAsync(url);
var buffer = new byte[(int)bufferSize];
var totalBytesRead = 0;
int bytes = 0;
while ((bytes = await request.ReadAsync(buffer, 0, buffer.Length)) != 0)
{
await file.WriteAsync(buffer, 0, bytes);
totalBytesRead += bytes;
ConsoleUtilities.UpdateProgress($"Downloading Runtime... ({totalBytesRead} of {bufferSize} bytes read) ", (int)(totalBytesRead / bufferSize * 100));
}

System.ObjectDisposedException UWP

i've an issue in my application. I'm trying to send an image into FTP server. I'm able to connect with server, authenticate, and other. But when i try to send file, if the file is big (usually bigger than 40-50kb) i got System.ObjectDisposedException.
This is the code that i use to send:
public async Task <byte[]> GetResultingBuffer(IRandomAccessStreamWithContentType readStream, IBuffer buffer)
{
var resultingBuffer = new byte[0];
while (true)
{
IBuffer readBuffer = await readStream.ReadAsync(buffer, 1024, InputStreamOptions.Partial);
if (readBuffer.Length == 0) break;
resultingBuffer = resultingBuffer.Concat(readBuffer.ToArray()).ToArray();
}
return resultingBuffer;
}
public async Task UploadFileAsync(StorageFile file, string destination)
{
using (var stream = await OpenWriteAsync(destination))
{
//
// A more efficient way, maybe a DataReader can be used here
using (var readStream = await file.OpenReadAsync())
{
var buffer = new byte[1024].AsBuffer();
var resultingBuffer = new byte[0];
resultingBuffer = await GetResultingBuffer(readStream, buffer);
await stream.WriteAsync(resultingBuffer.AsBuffer());
await stream.FlushAsync();
}
}
}
I tried to edit it, before editing GetResultingBuffer was not a task, but a cycle inside UploadFileAsnc. How can i prevent to dispose the buffer? Is there another solution? Thanks!

I solved it in this way
using (var readStream = await file.OpenReadAsync())
{
var buffer = new byte[3000000].AsBuffer();
//var resultingBuffer = new byte[10000000];
Debug.Write("-------");
//while (true)
//{
IBuffer readBuffer = await readStream.ReadAsync(buffer, 3000000, InputStreamOptions.Partial);
//if (readBuffer.Length == 0) break;
//resultingBuffer = resultingBuffer.Concat(readBuffer.ToArray()).ToArray();
//}
// await stream.WriteAsync(resultingBuffer.AsBuffer());
var resultingBuffer = new byte[readBuffer.Length];
readBuffer.CopyTo(resultingBuffer);
await stream.WriteAsync(resultingBuffer.AsBuffer());
}
Debug.Write("-------");
await stream.FlushAsync();

Dealing with large file uploads on ASP.NET Core 1.0

When I'm uploading large files to my web api in ASP.NET Core, the runtime will load the file into memory before my function for processing and storing the upload is fired. With large uploads this becomes an issue as it is both slow and requires more memory. For previous versions of ASP.NET there are some articles on how to disable the buffering of requests, but I'm not able to find any information on how to do this with ASP.NET Core. Is it possible to disable the buffering of requests so I don't run out of memory on my server all the time?

Use the Microsoft.AspNetCore.WebUtilities.MultipartReader because it...
can parse any stream [with] minimal buffering. It gives you the headers and body of each section one at a time and then you do what you want with the body of that section (buffer, discard, write to disk, etc.).
Here is a middleware example.
app.Use(async (context, next) =>
{
if (!IsMultipartContentType(context.Request.ContentType))
{
await next();
return;
}
var boundary = GetBoundary(context.Request.ContentType);
var reader = new MultipartReader(boundary, context.Request.Body);
var section = await reader.ReadNextSectionAsync();
while (section != null)
{
// process each image
const int chunkSize = 1024;
var buffer = new byte[chunkSize];
var bytesRead = 0;
var fileName = GetFileName(section.ContentDisposition);
using (var stream = new FileStream(fileName, FileMode.Append))
{
do
{
bytesRead = await section.Body.ReadAsync(buffer, 0, buffer.Length);
stream.Write(buffer, 0, bytesRead);
} while (bytesRead > 0);
}
section = await reader.ReadNextSectionAsync();
}
context.Response.WriteAsync("Done.");
});
Here are the helpers.
private static bool IsMultipartContentType(string contentType)
{
return
!string.IsNullOrEmpty(contentType) &&
contentType.IndexOf("multipart/", StringComparison.OrdinalIgnoreCase) >= 0;
}
private static string GetBoundary(string contentType)
{
var elements = contentType.Split(' ');
var element = elements.Where(entry => entry.StartsWith("boundary=")).First();
var boundary = element.Substring("boundary=".Length);
// Remove quotes
if (boundary.Length >= 2 && boundary[0] == '"' &&
boundary[boundary.Length - 1] == '"')
{
boundary = boundary.Substring(1, boundary.Length - 2);
}
return boundary;
}
private string GetFileName(string contentDisposition)
{
return contentDisposition
.Split(';')
.SingleOrDefault(part => part.Contains("filename"))
.Split('=')
.Last()
.Trim('"');
}
External References
https://github.com/aspnet/HttpAbstractions/pull/146
https://github.com/aspnet/HttpAbstractions

Shaun Luttin's answer is great, and now much of the work he's demonstrated is provided by ASP.Net Core 2.2.
Get the boundary:
// Microsoft.AspNetCore.Http.Extensions.HttpRequestMultipartExtensions
var boundary = Request.GetMultipartBoundary();
if (string.IsNullOrWhiteSpace(boundary))
return BadRequest();
You still get a section as follows:
var reader = new MultipartReader(boundary, Request.Body);
var section = await reader.ReadNextSectionAsync();
Check the disposition and convert to FileMultipartSection:
if (section.GetContentDispositionHeader())
{
var fileSection = section.AsFileSection();
var fileName = fileSection.FileName;
using (var stream = new FileStream(fileName, FileMode.Append))
await fileSection.FileStream.CopyToAsync(stream);
}

In your Controller you can simply use Request.Form.Files to access the files:
[HttpPost("upload")]
public async Task<IActionResult> UploadAsync(CancellationToken cancellationToken)
{
if (!Request.HasFormContentType)
return BadRequest();
var form = Request.Form;
foreach(var formFile in form.Files)
{
using(var readStream = formFile.OpenReadStream())
{
// Do something with the uploaded file
}
}
return Ok();
}

How to save data from stream to file in WinRT?

I would like to have a method which takes as an input a System.IO.Stream and use it to write data from it to file. So far I have the following:
public async Task SaveStreamToFileX(Stream stream, string filePath, IProgress<long> progress)
{
var folder = await StorageFolder.GetFolderFromPathAsync(Path.GetDirectoryName(filePath));
var fileName = Path.GetFileName(filePath);
StorageFile file = await folder.CreateFileAsync(fileName, CreationCollisionOption.OpenIfExists);
var istream = stream.AsInputStream();
var canRead = stream.CanRead; //this returns true
using (var reader = new DataReader(istream))
{
using (IRandomAccessStream fileStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
using (IOutputStream outputStream = fileStream.GetOutputStreamAt(0))
{
using (DataWriter writer = new DataWriter(outputStream))
{
IBuffer buffer;
long readBytes = 0;
const int bufferSize = 8192;
while ((buffer = reader.ReadBuffer(bufferSize)).Length > 0) //exception System.Exception with message: Out of range ...
{
writer.WriteBuffer(buffer);
readBytes += bufferSize;
progress.Report(readBytes);
}
}
}
}
}
}
Problem is, that an exception (Out of range) is thrown when I try to read data in while cycle (first read). Stream should have data. I am not sure if so long code is neccesarry, if somebody has better solution it would be great.
Side note:
If I try await reader.LoadAsync(50) it returns 50. I am not sure what LoadAsync should do. Maybe I have to call it before read to prepare data for the read? I will investigate this further ...
Also, the Stream.CanRead returns true.

The problem was not with converting streams as I orginally thought. It was just lack of the knowlidge how the work with files is done in WinRT (the docs from microsoft are really terrible in my opinion).
In the end with help of my collegue with tried several ways and end up with the following:
public async Task SaveStreamToFile(Stream stream, string filePath, IProgress<long> progress )
{
var folder = await StorageFolder.GetFolderFromPathAsync(Path.GetDirectoryName(filePath));
var fileName = Path.GetFileName(filePath);
StorageFile file = await folder.CreateFileAsync(fileName, CreationCollisionOption.OpenIfExists);
var istream = stream.AsInputStream();
using (var reader = new DataReader(istream))
{
using (IRandomAccessStream fileStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
using (IOutputStream outputStream = fileStream.GetOutputStreamAt(0))
{
using (DataWriter writer = new DataWriter(outputStream))
{
long writtenBytes = 0;
const int bufferSize = 8192;
uint loadedBytes = 0;
while ((loadedBytes = (await reader.LoadAsync(bufferSize))) > 0) //!!!
{
IBuffer buffer = reader.ReadBuffer(loadedBytes);
writer.WriteBuffer(buffer);
uint tmpWritten = await writer.StoreAsync(); //!!!
writtenBytes += tmpWritten;
progress.Report(writtenBytes);
}
}
}
}
}
}
I would like to see some simpler implementation, but this works. The problems were that LoadAsync was missing (which seems to be necessary to call) and during write operation the StoreAsync must be called in order to commit the data (flushing was not sufficient).
I hope this help somebody.

I would advise against that kind of code and instead take advantage of Windows Runtime Interop extension methods. That would produce a neater and more readable code, ex :
private async Task CopyToTempFile(Stream stream, string temporaryFileName) {
var file = await ApplicationData.Current.TemporaryFolder.CreateFileAsync(temporaryFileName, CreationCollisionOption.ReplaceExisting);
using (var outputstream = await file.OpenStreamForWriteAsync()) {
await stream.CopyToAsync(outputstream);
}
}

How to write a stream to a file in Windows Runtime?

There are lots of questions around, but mostly are about writing strings to a file. I'm a bit confused!
What is the proper way to write a stream to a file?
what I did:
Stream stream = await GetStream(source);
var file = await ApplicationData.Current.TemporaryFolder.CreateFileAsync("TempUsersFile", CreationCollisionOption.ReplaceExisting);
var fileStream = await file.OpenAsync(FileAccessMode.ReadWrite);
using (var dataWriter = new Windows.Storage.Streams.DataWriter(fileStream))
{
// How can I write to buffer and write to the file
}

There are many ways of writing a stream to file and they also depend on your needs. The method below performs asynchronous operation with a specified buffer:
public async Task SaveStreamToFile(Stream streamToSave, string fileName, CancellationToken cancelToken)
{
StorageFile file = await ApplicationData.Current.TemporaryFolder.CreateFileAsync(fileName, CreationCollisionOption.ReplaceExisting);
using (Stream fileStram = await file.OpenStreamForWriteAsync())
{
const int BUFFER_SIZE = 1024;
byte[] buf = new byte[BUFFER_SIZE];
int bytesread = 0;
while ((bytesread = await streamToSave.ReadAsync(buf, 0, BUFFER_SIZE)) > 0)
{
await fileStram.WriteAsync(buf, 0, bytesread);
cancelToken.ThrowIfCancellationRequested();
}
}
}
I've implemented also a CancellationToken in case there will be a need to cancel the long running Task.

I would recommend using the StreamReader and StreamWriter classes:
StreamWriter sw = new StreamWriter(outstream);
StreamReader sr = new StreamReader(instream);
while (!sr.EndOfStream)
{
int ch = sr.Read();
sw.Write(ch);
}