All,
I have a instance where I need to accept a gzip file from an HTTP POST in an aspnet core webapi. I have tried getting it from HTTPContext.Form.Files which obviously didn't work since it is content-type application/gzip. I have also tried:
var stream = new MemoryStream();
await HttpContext.Request.Body.CopyToAsync(stream);
await Decompress(stream, filename);
private async Task Decompress(MemoryStream compressedFileStream, string filename)
{
var stream = new MemoryStream();
using (var decompressor = new GZipStream(compressedFileStream, CompressionMode.Decompress))
{
await decompressor.CopyToAsync(stream);
}
using (var streamReader = new StreamReader(stream))
{
var content = await streamReader.ReadToEndAsync();
var doc = new XmlDocument();
doc.LoadXml(content);
}
}
content is "" so it's not getting the content of what is in the gzip file.
All help would be greatly appreciated.
Here is my latest attempt:
var stream = new MemoryStream();
await HttpContext.Request.Body.CopyToAsync(stream);
await using (var decompressor = new GZipStream(stream, CompressionMode.Decompress))
{
var stream1 = new MemoryStream();
await decompressor.CopyToAsync(stream1);
using (var streamReader = new StreamReader(stream1))
{
var content = await streamReader.ReadToEndAsync();
var doc = new XmlDocument();
doc.LoadXml(content);
}
}
I think the issue is I am not getting the request body correctly. Any thoughts on that?
Thank you!
Related
I am attempting to get a file from Ebays API
https://developer.ebay.com/api-docs/sell/feed/resources/task/methods/getResultFile#h2-samples
I am getting data back but it is not decompressing properly or encoding properly. It should be an gziped xml file. The documentation is not very clear on this actually. I am using RestSharp for my http calls (106.15.0).
Exception:
Found invalid data while decoding
My Code:
const string url = "sell/feed/v1/task/task-16-SOMENUMBER/download_result_file";
var restClient = new RestClient(_restApiUrl)
{
Authenticator = new OAuth2AuthorizationRequestHeaderAuthenticator(authentication.AuthorizationToken, "Bearer")
};
var httpRequest = new RestRequest(url, Method.GET);
httpRequest.AddHeader("Accept-Encoding", "application/gzip");
httpRequest.AddHeader("Accept", "*/*");
byte[] myfile = restClient.DownloadData(httpRequest);
var decodedString = Encoding.UTF8.GetString(myfile);
using (var stream = new MemoryStream(myfile))
{
string res;
using (GZipStream zipStream = new GZipStream(stream, CompressionMode.Decompress))
{
using (var sr = new StreamReader(zipStream))
{
res = sr.ReadToEnd(); //ERROR HERE: Found invalid data while decoding
}
}
var result = res;
}
First 30 of returned string (Encoding.Default.GetString(myfile))
PK\u0003\u0004\u0014\0\b\b\b\0\f£†T\0\0\0\0\0\0\0\0\0\0\0\0>\0\0\0ActiveInventoryReport
Hex
50-4B-03-04-14-00-08-08-08-00-0C-A3-86-54-00-00-00-00-00-00-00-00-00-00-00-00-3E-00-00-00-41-63-74-69-76-65-49-6E-76-65-6E-74-6F-72-79-52-65-70-6F-72-74-2D-41-70-72-2D-30-36-2D-32-30-32-32-2D-32-30-3A-32-34-3A-32-30-2D-30-37-30-30-2D-31-33-33-34-39-39-38-35-32-34-2E-78-6D-6C-BD-9D-5B-53-9B-47-B6-86-AF-77-7E-45-CA-F7-32-7D-3E-4C-79-3C-25-09-03-1E-C0-D6-48-38-60-DF-B1-8D-F6-98-0A-01-17-86-4C-3C-BF-7E-F7-E1-
I will try to keep it short and precise.
Requirement:
Download large (400mb) xml response from 3rd party and store as ZipArchive on disk.
Current solution:
using (var memoryStream = new MemoryStream())
{
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
var file = archive.CreateEntry($"{deliveryDate:yyyyMMdd}.xml");
using(var entryStream = file.Open())
{
using (var payload = new MemoryStream())
{
using var response = await _httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
await response.Content.CopyToAsync(payload);
payload.Seek(0, SeekOrigin.Begin);
await payload.CopyToAsync(entryStream);
}
}
}
using (var fileStream = new FileStream(Path.Combine(filePath), FileMode.Create, FileAccess.Write, FileShare.None))
{
memoryStream.Seek(0, SeekOrigin.Begin);
await memoryStream.CopyToAsync(fileStream);
}
}
Additional Information:
I can compress a 400mb file to approx. 20mb in about 40 seconds. 1/4 is download 3/4 is compression.
The httpClient is re-used.
The code runs in a long lived application hosted as a k8 linux pod.
Issues with current solution:
I fail to understand if this implementation will clean up after itself. I would be thankful for pointers towards potential leaks.
may be writing more directly to the filestream would be faster / cleaner
and the response should be disposed:
using System.IO.Compression;
string url = "https://stackoverflow.com/questions/70605408/better-way-to-process-large-httpclient-response-400mb-to-ziparchive";
string filePath = "test.zip";
using(HttpClient client = new HttpClient())
using (var fileStream = new FileStream(Path.Combine(filePath), FileMode.Create, FileAccess.Write, FileShare.None))
using (var archive = new ZipArchive(fileStream, ZipArchiveMode.Create, true))
{
var file = archive.CreateEntry($"test.xml");
using (var entryStream = file.Open())
using (var response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
{
response.EnsureSuccessStatusCode();
var stream = await response.Content.ReadAsStreamAsync();
await stream.CopyToAsync(entryStream);
}
}
I have some old Web API action method which returns CSV file. It worked for long time, but recently stopped. Now it causes ERR_SPDY_PROTOCOL_ERROR.
ERR_SPDY_PROTOCOL_ERROR in Chrome is often associated with Avast security as described here. In my case however it's not caused by Avast, and other web browsers throw exceptions too.
My action method looks as follows:
[HttpGet]
[Route("csv")]
public HttpResponseMessage SomeMethod([FromUri]SomeSearchCriteria sc)
{
using (MemoryStream stream = new MemoryStream())
{
StreamWriter writer = new StreamWriter(stream, Encoding.UTF8);
string content = someLogic.SomeSearchmethod(sc);
writer.Write(content);
writer.Flush();
stream.Position = 0;
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType = new MediaTypeHeaderValue("text/csv");
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") { FileName = "Export.csv" };
return result;
}
}
The method is called by angular front end by simple change of window.location on button click.
Whole action method is executed properly, with no exceptions. Error is shown only by web browser.
Flushing sockets in Chrome as described here does not solve the issue.
I have tried this method in API controller and call through chrome browser, it throws net::ERR_CONNECTION_RESET
There is some issue in response filled with StreamContent, use ByteArrayContent in result content, it works perfectly.
[HttpGet]
[Route("csv")]
public HttpResponseMessage SomeMethod([FromUri]SomeSearchCriteria sc)
{
using (MemoryStream stream = new MemoryStream())
{
StreamWriter writer = new StreamWriter(stream, Encoding.UTF8);
string content = "test";
writer.Write(content);
writer.Flush();
stream.Position = 0;
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
//result.Content = new StreamContent(stream);
result.Content = new ByteArrayContent(stream.ToArray());
result.Content.Headers.ContentType = new MediaTypeHeaderValue("text/csv");
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") { FileName = "Export.csv" };
return result;
}
}
I am trying to save an image from the web to local storage to be manipulated later, but it appears to be corrupt and attempting to open it with an external application fails. Opening the image in the webbrowser works completely normally. Thanks for any help.
var client = new HttpClient();
var clientResponse = await client.GetByteArrayAsync(imageUri);
var temp = ApplicationData.Current.TemporaryFolder;
StorageFile file;
if ((await temp.GetFilesAsync()).Any(f => f.Name == "temp_image.png")) {
file = await temp.GetFileAsync("tempcolorizer.png");
} else {
file = await temp.CreateFileAsync("temp_image.png");
}
using (var fs = await file.OpenReadAsync())
using (var writer = new DataWriter(fs)) {
writer.WriteBytes(clientResponse);
}
You have to call StoreAsync:
using (var fs = await file.OpenReadAsync())
using (var writer = new DataWriter(fs)) {
writer.WriteBytes(clientResponse);
await writer.StoreAsync();
}
I use DotNetZip.
I get error
It is not possible to use PKZIP encryption on a non-seekable input
stream
what to do
var outputStream = new MemoryStream();
using (var zip = new ZipFile())
{
zip.Password = "123456!";
var outputStreamFile = new MemoryStream();
var userId = m_userRepository.GetuserByLogin(this.User.Identity.Name).UserId;
using (var streamWriter = new StreamWriter(outputStreamFile))
{
streamWriter.WriteLine(m_kamikaze2Repository.GetGameById(gameId, userId).Result);
}
zip.AddEntry("result_" + gameId, outputStreamFile);
zip.Save(outputStream);//error
}
The problem is that outpuStreamFile is closed when you add it to the ZipFile. StreamWriter.Dispose will dispose the stream so just move the using statement to ensure that outputStreamFile is available when you call zip.Save.
Also, before adding outputStreamFile to the ZipFile you need to rewind it.
using (var outputStreamFile = new MemoryStream()) {
var userId = m_userRepository.GetuserByLogin(this.User.Identity.Name).UserId;
var streamWriter = new StreamWriter(outputStreamFile);
streamWriter.WriteLine(m_kamikaze2Repository.GetGameById(gameId, userId).Result);
outputStreamFile.Seek(0, SeekOrigin.Begin);
zip.AddEntry("result_" + gameId, outputStreamFile);
zip.Save(outputStream);
}
Try like this:
using (var outputStream = new MemoryStream())
using (var zip = new ZipFile())
{
zip.Password = "123456!";
zip.AddEntry("result_" + gameId, "Some content");
zip.Save(outputStream);
byte[] zipFile = outputStream.ToArray();
// TODO: do something with the zip
}