I have very little understanding of the c# streams. I'm trying to upload brotli compressed json into azure storage.
private async Task UploadJSONAsync(BlobClient blob, object serializeObject, CancellationToken cancellationToken)
{
var json = JsonConvert.SerializeObject(serializeObject);
using (var sourceStream = new MemoryStream(Encoding.UTF8.GetBytes(json)))
using (var destStream = new MemoryStream())
using (var brotliStreamCompressor = new BrotliStream(destStream, CompressionLevel.Optimal, false))
{
sourceStream.CopyTo(brotliStreamCompressor);
//brotliStreamCompressor.Close(); // Closes the stream, can't read from a closed stream.
await blob.DeleteIfExistsAsync();
await blob.UploadAsync(destStream, cancellationToken);
//brotliStreamCompressor.Close(); // destStream has zero bytes
}
}
}
I'm sure my lack of stream knowledge is preventing this from working.
In order to read the stream I had to set it position back to zero.
private async Task UploadJSONAsync(BlobClient blob, object serializeObject, CancellationToken cancellationToken)
{
var json = JsonConvert.SerializeObject(serializeObject);
using (var sourceStream = new MemoryStream(Encoding.UTF8.GetBytes(json)))
using (var destStream = new MemoryStream())
using (var brotliStreamCompressor = new BrotliStream(destStream, CompressionLevel.Optimal, false))
{
sourceStream.CopyTo(brotliStreamCompressor);
brotliStreamCompressor.Close();
destStream.Position = 0;
await blob.DeleteIfExistsAsync();
await blob.UploadAsync(destStream, cancellationToken);
}
}
}
Related
All,
I have a instance where I need to accept a gzip file from an HTTP POST in an aspnet core webapi. I have tried getting it from HTTPContext.Form.Files which obviously didn't work since it is content-type application/gzip. I have also tried:
var stream = new MemoryStream();
await HttpContext.Request.Body.CopyToAsync(stream);
await Decompress(stream, filename);
private async Task Decompress(MemoryStream compressedFileStream, string filename)
{
var stream = new MemoryStream();
using (var decompressor = new GZipStream(compressedFileStream, CompressionMode.Decompress))
{
await decompressor.CopyToAsync(stream);
}
using (var streamReader = new StreamReader(stream))
{
var content = await streamReader.ReadToEndAsync();
var doc = new XmlDocument();
doc.LoadXml(content);
}
}
content is "" so it's not getting the content of what is in the gzip file.
All help would be greatly appreciated.
Here is my latest attempt:
var stream = new MemoryStream();
await HttpContext.Request.Body.CopyToAsync(stream);
await using (var decompressor = new GZipStream(stream, CompressionMode.Decompress))
{
var stream1 = new MemoryStream();
await decompressor.CopyToAsync(stream1);
using (var streamReader = new StreamReader(stream1))
{
var content = await streamReader.ReadToEndAsync();
var doc = new XmlDocument();
doc.LoadXml(content);
}
}
I think the issue is I am not getting the request body correctly. Any thoughts on that?
Thank you!
I will try to keep it short and precise.
Requirement:
Download large (400mb) xml response from 3rd party and store as ZipArchive on disk.
Current solution:
using (var memoryStream = new MemoryStream())
{
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
var file = archive.CreateEntry($"{deliveryDate:yyyyMMdd}.xml");
using(var entryStream = file.Open())
{
using (var payload = new MemoryStream())
{
using var response = await _httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
await response.Content.CopyToAsync(payload);
payload.Seek(0, SeekOrigin.Begin);
await payload.CopyToAsync(entryStream);
}
}
}
using (var fileStream = new FileStream(Path.Combine(filePath), FileMode.Create, FileAccess.Write, FileShare.None))
{
memoryStream.Seek(0, SeekOrigin.Begin);
await memoryStream.CopyToAsync(fileStream);
}
}
Additional Information:
I can compress a 400mb file to approx. 20mb in about 40 seconds. 1/4 is download 3/4 is compression.
The httpClient is re-used.
The code runs in a long lived application hosted as a k8 linux pod.
Issues with current solution:
I fail to understand if this implementation will clean up after itself. I would be thankful for pointers towards potential leaks.
may be writing more directly to the filestream would be faster / cleaner
and the response should be disposed:
using System.IO.Compression;
string url = "https://stackoverflow.com/questions/70605408/better-way-to-process-large-httpclient-response-400mb-to-ziparchive";
string filePath = "test.zip";
using(HttpClient client = new HttpClient())
using (var fileStream = new FileStream(Path.Combine(filePath), FileMode.Create, FileAccess.Write, FileShare.None))
using (var archive = new ZipArchive(fileStream, ZipArchiveMode.Create, true))
{
var file = archive.CreateEntry($"test.xml");
using (var entryStream = file.Open())
using (var response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
{
response.EnsureSuccessStatusCode();
var stream = await response.Content.ReadAsStreamAsync();
await stream.CopyToAsync(entryStream);
}
}
I would like to be able to return MemoryStream from my function but I think that when returning the stream its also automatically closed.
using (var httpStream = await httpClient.GetStreamAsync(link))
{
using (var stream = new MemoryStream())
{
await httpStream.CopyToAsync(stream);
return stream;
}
}
Is there any way to maybe override this so I can return stream so I can use it elsewhere.
Here's how I'm trying to use it in another method:
using (var fielStream = new FileStream(path, FileMode.Create))
{
Stream stream = await GetStreamAsync(videoid, type, quality);
await stream.CopyToAsync(fileStream);
}
Drop inner using: it is the caller (who calls for creation) should Dispose the stream:
using (var httpStream = await httpClient.GetStreamAsync(link)) {
var stream = new MemoryStream();
try {
await httpStream.CopyToAsync(stream);
return stream;
}
catch {
stream.Dispose();
throw;
}
}
It is the caller who should put using, like this:
// We create MemoryStream with the code above (GetStreamAsync)
/ and then Dispose the stream
using (var stream = await GetStreamAsync(...)) {
...
}
I am trying to write the stream resulting from compressing several files in a ZIP but I can't. The ZIP file does, but when I want to write the resulting stream, the copyto method does nothing and my http request never ends.
I don't know why my logic doesn't work, I hope you can help me please.
public async Task<HttpResponseMessage> downloadFile3(string filePath, System.Threading.CancellationToken token)
{
var response = new HttpResponseMessage(System.Net.HttpStatusCode.OK)
{
Content = new PushStreamContent(async (streamout, context, transportContext) =>
{
try
{
using (var ms = new MemoryStream())
{
using (var zipArchive = new ZipArchive(ms, ZipArchiveMode.Create, true))
{
var entry = zipArchive.CreateEntry(filePath);
using (var fileStream = File.OpenRead(filePath))
{
using (var entryStream = entry.Open())
{
await fileStream.CopyToAsync(entryStream);
}
}
}
ms.Position = 0;
ms.CopyTo(streamout); //THIS LINE DOESN'T WORK
}
}
catch (Exception ex)
{
}
finally
{
streamout.Close();
}
}, "application/zip"),
};
response.Content.Headers.ContentLength = new FileInfo(filePath).Length;
response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment")
{
Size = new FileInfo(filePath).Length,
FileName = Path.GetFileName(filePath)
};
return response;
}
I'm having strange problem with this piece of code which basically zips files (docs) and uploads them to blob storage.
v11SDK: (docs)
var blockBlobClient = new BlockBlobClient(ConnectionString, ContainerName, "test-blob.zip");
// Saved zip is valid
// using (FileStream zipStream = new FileStream(#"C:\Users\artur\Desktop\test-local.zip", FileMode.OpenOrCreate))
// Uploaded zip is invalid
using (var stream = await blockBlobClient.OpenWriteAsync(true))
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create))
{
var readmeEntry = archive .CreateEntry("Readme.txt");
using (StreamWriter writer = new StreamWriter(readmeEntry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
await stream.FlushAsync();
}
v12SDK: (docs)
var blobClient = new BlobClient(ConnectionString, InputContainerName, "test-blob.zip");
using var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create))
{
var readmeEntry = archive.CreateEntry("Readme.txt");
using StreamWriter writer = new StreamWriter(readmeEntry.Open());
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
await writer.FlushAsync();
}
stream.Position = 0;
await blobClient.UploadAsync(stream, true);
await stream.FlushAsync();
}
Saving zip file locally produces a valid zip (164 bytes). Saving zip to blob storage (using storage emulator) produces invalid zip (102 bytes).
I can't figure out why
Here is the correct code.
The problem was premature disposing of inner stream by ZipArchive. Note in my code below, I have passed leaveInnerStreamOpen as true while creating ZipArchive since we are already disposing stream in the outer using. Also for V11 code, I have switched to MemoryStream instead of OpenWrite of blob stream since did not have control to set stream position to 0 if we use OpenWrite. And you don't need any Flush :)
v11SDK:
var blockBlobClient = new BlockBlobClient(ConnectionString, ContainerName, "test-blob.zip");
using var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create, true))
{
var readmeEntry = archive.CreateEntry("Readme.txt");
using (StreamWriter writer = new StreamWriter(readmeEntry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
}
stream.Position = 0;
await blockBlobClient.UploadAsync(stream);
v12SDK:
var blobClient = new BlobClient(ConnectionString, InputContainerName, "test-blob.zip");
using var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create, true))
{
var readmeEntry = archive.CreateEntry("Readme.txt");
using StreamWriter writer = new StreamWriter(readmeEntry.Open());
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
}
stream.Position = 0;
await blobClient.UploadAsync(stream, true);