I have a web application written in ASP.NET. All is working okay, except that I would like to compress the data being returned. The data is basically a List of custom models. Currently I do something like:
string json_string = new JavaScriptSerializer().Serialize(my_models);
using (var output = new MemoryStream())
{
using (var compressor = new Ionic.Zlib.GZipStream(output, Ionic.Zlib.CompressionMode.Compress, Ionic.Zlib.CompressionLevel.BestCompression))
{
compressor.Write(Encoding.ASCII.GetBytes(json_string), 0, json_string.Length);
}
}
I then proceed with the following:
HttpResponseMessage json = Request.CreateResponse(HttpStatusCode.OK, json_string);
json.Content.Headers.Add("content-encoding", "gzip");
This causes an application error. In Chrome (through the console), I see the following message:
ERR_CONTENT_DECODING_FAILED
Where am I going wrong here? Thanks!
So in your code you wrote the compressed content to your MemoryStream but your json_string is still your original json_string which then you added that original string as response but marked it as compressed gzip format.
So in the end Chrome tries to decode a pure string JSON content with gzip and it failed to do so
You would want to write whats inside output into json_string object before sending it off
Or you can config IIS to always compress everything to save yourself some trouble.
Related
I am saving zip files to an AWS S3 bucket. I am now trying to create a C# .NET API that will allow me to download a specified key from the bucket and save it to a HttpResponseMessage in the Content key.
I've referred to the following question to set up my response for zip files: How to send a zip file from Web API 2 HttpGet
I have modified the code in the previous question so that it instead reads from a TransferUtility stream.
Problem is I am coming into an error when trying to extract or view the file that looks like the following:
The response I am getting back from the API looks like:
The relevant code looks like:
[HttpGet, Route("GetFileFromS3Bucket")]
public HttpResponseMessage GetFileFromS3Bucket(string keyName)
{
HttpResponseMessage response = new HttpResponseMessage();
string bucketName = "myBucket";
RegionEndpoint bucketRegion = RegionEndpoint.ARegion;
IAmazonS3 s3Client;
s3Client = new AmazonS3Client(bucketRegion);
try
{
var fileTransferUtility = new TransferUtility(s3Client);
var stream = fileTransferUtility.OpenStream(bucketName, keyName);
response.Content = new StreamContent(stream);
response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
response.Content.Headers.ContentDisposition.FileName = keyName + ".zip";
response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/zip");
response.StatusCode = HttpStatusCode.OK;
}
catch (Exception e)
{
response.Content = new StringContent("Something went wrong, error: " + e.Message);
response.StatusCode = HttpStatusCode.InternalServerError;
}
return response;
}
Results of troubleshooting:
The file from the Web API comes out with nearly double the expected size based on what is in S3. This is consistent across different files
Changing the bucket to be publicly accessible did not help (setting since reverted to not allowing public access)
Changing the file type to XML did not display a nicely formatted error (there was a suggestion that you may receive an XML response if an error was provided from S3)
Saving the S3 stream to a file and then saving directly to a file resulted in the correct file size. Seems safe to say the stream from S3 is not the problem
It appears that there ia a problem with the way the HTTPResponseMessage is handling the zip file. I'm unsure of whether it is actually on the server side, or whether it is on the client to parse the data and Swagger is simply incapable of doing that. Any help would be greatly appreciated.
Update 1
I do not believe this string is Base64 encoded as the result I got from converting the stream to a string is the following:
I've updated the code sample with the two lines showing the conversion from a stream to string.
Update 2
I've confirmed the issue is with how the response is handling the stream, or something in the response itself. Downloading the file stream from S3 and saving to a new file on the local computer resulted in a valid file that opened as expected.
Update 3
Link to GDrive folder with testing files: https://drive.google.com/drive/folders/1q_N3NTHz5E_nebtBQJHor3HfqUZWhGgd?usp=sharing
I unfortunately can't provide access to the original file as it contains sensitive data. The provided files are still causing the same problem however.
Interesting to note that the test file came out looking like:
The underscores on either side of the filename are quite strange.
I am running the following relevant packages:
Update 4
I've found the following UTF8 references in various files:
File: configuration91.svcinfo
I could not find anything that said anything about 'responseEncoding' anywhere in the project.
I am going to throw an answer up, because what's happening to you is unorthodox. I use S3 for many things and have done what you are doing with no problems in the past. To ensure that I am mimicking what you are doing, I duplicated your code:
[HttpGet, Route("GetFileFromS3Bucket/{keyName}")]
public HttpResponseMessage GetFileFromS3Bucket(string keyName)
{
string bucketName = "testzipfilesagain";
string awsAccessKey = "AKIAJ********A3QHOUA";
string awsSecretKey = "IYUJ9Gy2wFCQ************dCq5suFS";
IAmazonS3 client = new AmazonS3Client(awsAccessKey, awsSecretKey, RegionEndpoint.USEast1);
var fileTransferUtility = new TransferUtility(client);
var stream = fileTransferUtility.OpenStream(bucketName, "md5.zip");
var resp = new HttpResponseMessage();
resp.Content = new StreamContent(stream);
resp.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
resp.Content.Headers.ContentDisposition.FileName = keyName + ".zip";
resp.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/zip");
resp.StatusCode = HttpStatusCode.OK;
return resp;
}
These are the packages I have installed:
<ItemGroup>
<PackageReference Include="AWSSDK.S3" Version="3.3.111.37" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.WebApiCompatShim" Version="2.2.0" />
<PackageReference Include="Swashbuckle.AspNetCore" Version="5.5.1" />
</ItemGroup>
Everything runs perfectly well.
Trying to troubleshoot your code is going to be fruitless because it works perfectly fine, but there is something wrong with your environment.
So this isn't an answer to your question, but a answer to how you can try to solve the issue at hand and get past this.
Make sure your nuget packages are up to date
Do you have any middleware injected in your pipeline? If so, what?
Post your startup.cs -- maybe something is out of order in your Configure routine.
Could you start a brand new project and try your code in that?
Can you try a small 5KB zip file and post the original and the corrupt so we can look?
I would love to get to the bottom of this as I really like to solve these types of problems.
EDIT 1
So I looked at the zip files and they have been run through a UTF8 encoding process. So, if you take your original zip file, and run this code on it:
var goodBytes = File.ReadAllBytes("Some test to upload to S3.zip");
var badBytes = File.ReadAllBytes("_Some test to upload to S3.zip.zip_");
File.WriteAllText("Some test to upload to S3.zip.utf8", Encoding.UTF8.GetString(goodBytes));
var utf8EncodedGoodBytes = File.ReadAllBytes("Some test to upload to S3.zip.utf8");
var identical = badBytes.SequenceEqual(utf8EncodedGoodBytes);
It the results are:
I am going to do some research and figure out what could be causing your stream to become UTF-8 encoded. Is there anything in your config that looks like this? Can you search your entire solution for anything that resembles "utf" or "utf8" or "utf-8"?
I'm downloading in image from web to save it locally. It works great with any other image formats but it this method below fails with an argument exception when I try to read a WebP image.
private static Image GetImage(string url)
{
try
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
return Image.FromStream(response.GetResponseStream());
}
catch
{
return null;
}
}
How do you read .webp images in C#?
I found this other question that allows for converting between types but I do not want to do that WebP library for C#
Reason I'm not wanting to do it is because I think it might lose some quality. Besides, I want to know why is this not working.
The base class libraries won't help you to deal with WebP images. However, if you only want to save the received file to the disk, you don't have to even know that you are dealing with a WebP images. You can simply treat the received data as a binary blob and dump it to a file, for example using Stream.CopyTo and a FileStream.
The Content-Type HTTP header will give you the mime type of the file you're downloading, and the Content-Disposition header can provide you with a filename and extension (though you might have to do some parsing). You can access those using HttpWebResponse.ContentType and HttpWebResponse.Headers["Content-Disposition"].
#Trillian nailed it. Here is a code snippet for what I did based on his suggestion. Wanted to add code so not posting this as a comment.
To get just the image file extension, you can do this
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
string fileExt = response.ContentType.Replace("image/", string.Empty);
To get the file name with extension, you can do the following and the do parsing like I did above. It just has some more data in it.
response.Headers["Content-Disposition"];
Once you have you file name you want to save as, create a file stream and copy the response stream into it.
FileStream fs = new FileStream(targetPath + fileName, FileMode.Create);
response.GetResponseStream().CopyTo(fs);
Assuming you app has access to the destination, image should get saved. Make sure to add try catch and handle exceptions properly. Also note that FileMode.Create will overwrite if the file already exists!
My problem is that I dont Know how i can download a File withknowing the file name or the file extension in the url, like this http://findicons.com/icon/download/235456/internet_download/128/png?id=235724
I hope you can help me
You could inspect the Content-Disposition response header using an HTTP request to get the filename. This would be a more general solution, so even if the filename is not contained in the URL, it would work:
var url = "http://findicons.com/icon/download/235456/internet_download/128/png?id=235724";
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
var fn = response.Headers["Content-Disposition"].Split(new string[] { "=" }, StringSplitOptions.None)[1];
string basePath = #"X:\Folder\SubFolder"; // Change accordingly...
var responseStream = response.GetResponseStream();
using (var fileStream = File.Create(Path.Combine(basePath, fn)))
{
responseStream.CopyTo(fileStream);
}
}
The above code uses certain methods and functions, you can find more information here:
HttpWebRequest - Usage example here
Saving a stream to a file - see this answer. Just note that when saving an HTTP response stream, you don't need to seek to the beginning, as it already is at the beginning and doing so will throw an exception. So, to be on the safe side, use it like I have in the code above.
Hope this answer helps you :)
It's possible to get the filename since the server is sending the Content-Disposition header. Here's a code example on how to get the filename using the HttpClient class:
var url = "http://findicons.com/icon/download/235456/internet_download/128/png?id=235724";
using (var client = new HttpClient())
using (var response = await client.GetAsync(url))
{
// make sure our request was successful
response.EnsureSuccessStatusCode();
// read the filename from the Content-Disposition header
var filename = response.Content.Headers.ContentDisposition.FileName;
// read the downloaded file data
var stream = await response.Content.ReadAsStreamAsync();
// Where you want the file to be saved
var destinationFile = Path.Combine("C:\\local\\directory", filename);
// write the steam content into a file
using (var fileStream = File.Create(destinationFile))
{
stream.CopyTo(fileStream);
}
}
I've had a hard time solving this issue myself and have come to a solution solving some issues in relation of obtaining the file name automatically.
Some headers does not include the Content Disposition as suggested in
https://stackoverflow.com/a/37228939/8805908, but the ones that do is still used.
I was wondering how Chrome, firefox etc. could obtain the name of the file though this information is not available through any header entry. I found that the links left without the information could be derived via its URL, from which i am using the code of:
http://codesnippets.fesslersoft.de/how-to-get-the-filename-of-url-in-c-and-vb-net/
My conclusion so far is; check the header for Content Disposition, if this does not contain any information check the URL for any file matches. So far I have not found a download link that I have not been able to retrieve the file name of.
I hope this may solve some issues.
--- Edit 12-06-2018
The solution using these methods are satisfying the following links:
5 test cases
I'm having tough time trying to send large files with HTTP file upload in ASP.NET.
The target is to transfer contents larger than 2GB, compressed to save the bandwidth. Sending 3GB uncompressed works well, all the files are received and saved correctly on the disk. Yet, when I use compression (either gzip or deflate), I get the following error from the receiving API:
Unexpected end of MIME multipart stream. MIME multipart message is not complete.
The thing is that only when sending large requests (approx. 300MB is the upper limit), I get the exception. Uploading 200MB compressed works well.
Here's the upload code:
using (var client = new HttpClient())
{
using (var content = new MultipartFormDataContent())
{
client.BaseAddress = new Uri(fileServerUrl);
client.DefaultRequestHeaders.TransferEncodingChunked = true;
CompressedContent compressedContent = new CompressedContent(content, "gzip");
var request = new HttpRequestMessage(HttpMethod.Post, "/api/fileupload/")
{
Content = compressedContent
};
var uploadResponse = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).Result;
}
}
The CompressedContent class is a class from WebApiContrib set of helpers
Here's the receiving end
// CustomStreamProvider is an implementation to store the files uploaded on the disk
var streamProvider = new CustomStreamProvider(uploadPath);
var res = await Request.Content.ReadAsMultipartAsync(streamProvider);
streamProvider.FileData.Select(file => new FileInfo(file.LocalFileName)).Select(fi => fi.FullName)
Can you provide me with a clue what the problem is? Why is it that larger contents (larger than 300MB) appear to be compressed improperly while smaller are transferred just fine?
Are you compressing the file in chunks? If you are able to send uncompressed files, then clearly, the problem is your compression routine. Using gzip or Deflate out of the box has limitation on file size. Therefore you need to compress in chunks.
Tip:
Debug your compression routine. Try to compress your file and save it to the HDD and see if it is readable.
Take a look at this article
Check what is content-length, HTTP server side routine does not expect you to send different content-length. You are probably sending content-length of original file size, where else you are sending compressed content and MultiPart parser is expecting data equal to content-length.
Instead, you send a zipped file and extract it on server side and let both zipping and unzipping be different at application level instead of plugging inside http protocol.
You will have to make sure, order of compression/decompression and content-length are correct on both side. As far as I am aware, compression on upload is not very common on HTTP protocol.
Try to add a name to your input as below:
<input type="file" id="fileInput" name="fileInput"/>
Or use a custom stream to append the newline that ASP.NET web api is expecting:
Stream reqStream = Request.Content.ReadAsStreamAsync().Result;
MemoryStream tempStream = new MemoryStream();
reqStream.CopyTo(tempStream);
tempStream.Seek(0, SeekOrigin.End);
StreamWriter writer = new StreamWriter(tempStream);
writer.WriteLine();
writer.Flush();
tempStream.Position = 0;
StreamContent streamContent = new StreamContent(tempStream);
foreach(var header in Request.Content.Headers)
{
streamContent.Headers.Add(header.Key, header.Value);
}
// Read the form data and return an async task.
await streamContent.ReadAsMultipartAsync(provider);
I am building a C# client for a CGI service (not sure what its called exactly).
It accepts a bunch of XML and spits out a response. I have tested it straight in Firefox and it works (see below).
Now I am not sure how to do this in C# code though? Does anyone have a helpful snippet of code, I can't imagine it would be that difficult?
http://www.travelcommunications.co.uk/cgi-bin/d3web_gzip.ssh?%3CTCOML%20version=%22NEWFORMAT%22%3E%3CTransferOnly%3E%3CAvailability%3E%3CRequest%3E%3CAgentCode%3ETEST%3C/AgentCode%3E%3CAgentType%3ETA%3C/AgentType%3E%3CDeparturePointCode%3EALC%3C/DeparturePointCode%3E%3CDeparturePointType%3EAIRPORT%3C/DeparturePointType%3E%3CArrivalPointCode%3EBEN%3C/ArrivalPointCode%3E%3CArrivalPointType%3ERESORT%3C/ArrivalPointType%3E%3CSectorType%3ERETURN%3C/SectorType%3E%3CArrDate%3E10.10.10%3C/ArrDate%3E%3CArrTime%3E10:00%3C/ArrTime%3E%3CRetDate%3E17.10.10%3C/RetDate%3E%3CRetTime%3E10:00%3C/RetTime%3E%3CBrochure%3E001%3C/Brochure%3E%3CAdults%3E2%3C/Adults%3E%3CChildren%3E0%3C/Children%3E%3CInfants%3E0%3C/Infants%3E%3CCurrencyCode%3EUKL%3C/CurrencyCode%3E%3C/Request%3E%3C/Availability%3E%3C/TransferOnly%3E%3C/TCOML%3E
You're looking for the WebClient class.
For example: (2nd EDIT: With GZIP; this code is tested and actually works)
string response;
using (var client = new WebClient()) {
byte[] bytes = client.DownloadData(url);
using(var reader = new StreamReader(new GZipStream(new MemoryStream(bytes), CompressionMode.Decompress)))
response = reader.ReadToEnd();
}
However, if the URL returns raw XML, you can also load the XML directly from the URL, like this:
var doc = XDocument.Load(url);