I've looked around for a solution to my issue but no-one seems to being aiming for quite what I'm trying to achieve.
My problem is such, I have Zip files stored in Azure Blob storage, now for security's sake we have an API2 controller action that provisions these zip files, rather than allowing direct downloading. This action will retrieve the blob, and download it to a stream, so that it can be packaged within a HTTPResponseMessage.
All of the above works, however, when I attempt to recreate the zip file, I'm informed it's corrupted. For now I'm just attempting to have the server (running on localhost) create the zip file, whereas the endgame is to have remote Client applications do this (I'm fairly certain the solution to my issue on the server would be the same.
public class FileActionResult : IHttpActionResult
{
private HttpRequestMessage _request;
private ICloudBlob _blob;
public FileActionResult(HttpRequestMessage request, ICloudBlob blob)
{
_request = request;
_blob = blob;
}
public async Task<HttpResponseMessage> ExecuteAsync(System.Threading.CancellationToken cancellationToken)
{
var fileStream = new MemoryStream();
await _blob.DownloadToStreamAsync(fileStream);
var byteTest = new byte[fileStream.Length];
var test = fileStream.Read(byteTest, 0, (int)fileStream.Length);
try
{
File.WriteAllBytes(#"C:\testPlatFiles\test.zip", byteTest);
}
catch(ArgumentException ex)
{
var a = ex;
}
var response = _request.CreateResponse(HttpStatusCode.Accepted);
response.Content = new StreamContent(fileStream);
response.Content.Headers.ContentLength = _blob.Properties.Length;
response.Content.Headers.ContentType = new MediaTypeHeaderValue(_blob.Properties.ContentType);
//set the fileName
response.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment")
{
FileName = _blob.Name,
Size = _blob.Properties.Length
};
return response;
}
}
I've looked into Zip libraries to see if any present the solution for converting the stream of a zip back to a zip file, but all I can find is reading zip files into streams, or the creation in order to provision a file download instead of a filecreate.
Any help would be much appreciated, thank you.
You use DotNetZip. Its ZipFile class has a static factory method that should do what you want: ZipFile.Read( Stream zipStream ) reads the given stream as a zip file and gives you back a ZipFile instance (which you can use for whatever.
However, if your Stream contains the raw zip data and all you want to do is persist it to disk, you should just be able to write the bytes straight to disk.
If you're getting 'zip file corrupted' errors, I'd look at the content encoding used to send the data to Azure and the content encoding it's sent back with. You should be sending it up to Azure with a content type of application/zip or application/octet-stream and possibly adding metadata to the Azure blob entry to send it down the same way.
Edited To Note: DotNetZip used to live at Codeplex. Codeplex has been shut down. The old archive is still available at Codeplex. It looks like the code has migrated to Github:
https://github.com/DinoChiesa/DotNetZip. Looks to be the original author's repo.
https://github.com/haf/DotNetZip.Semverd. This looks to be the currently maintained version. It's also packaged up an available via Nuget at https://www.nuget.org/packages/DotNetZip/
Related
Continuing the saga, here is part I: ContentHash is null in Azure.Storage.Blobs v12.x.x
After a lot of debugging, root cause appears to be that the content hash was not calculated after uploading a blob, therefore the BlobContentInfo or BlobProperties were returning a null content hash and my whole flow is based on receiving the hash from Azure.
What I've discovered is that it depends on which HttpRequest stream method I call and upload to azure:
HttpRequest.GetBufferlessInputStream(), the content hash is not calculated, even if I go into azure storage explorer, the ContentMD5 of the blob is empty.
HttpRequest.InputStream() everything works as expected.
Do you know why this different behavior? And do you know how to make to receive content hash for streams received by GetBufferlessInputStream method.
So the code flow looks like this:
var stream = HttpContext.Current.Request.GetBufferlessInputStream(disableMaxRequestLength: true)
var container = _blobServiceClient.GetBlobContainerClient(containerName);
var blob = container.GetBlockBlobClient(blobPath);
BlobHttpHeaders blobHttpHeaders = null;
if (!string.IsNullOrWhiteSpace(fileContentType))
{
blobHttpHeaders = new BlobHttpHeaders()
{
ContentType = fileContentType,
};
}
// retry already configured of Azure Storage API
await blob.UploadAsync(stream, httpHeaders: blobHttpHeaders);
return await blob.GetPropertiesAsync();
In the code snippet from above ContentHash is NOT calculated, but if I change the way I am getting the stream from the http request with following snippet ContentHash is calculated.
var stream = HttpContext.Current.Request.InputStream
P.S. I think its obvious, but with the old sdk, content hash was calculated for streams received by GetBufferlessInputStream method
P.S2 you can find also an open issue on github: https://github.com/Azure/azure-sdk-for-net/issues/14037
P.S3 added code snipet
Ran into this today. From my digging, it appears this is a symptom of the type of Stream you use to upload, and it's not really a bug. In order to generate a hash for your blob (which is done on the client side before uploading by the looks of it), it needs to read the stream. Which means it would need to reset the position of your stream back to 0 (for the actual upload process) after generating the hash. Doing this requires the ability to perform the Seek operation on the stream. If your stream doesn't support Seek, then it looks like it doesn't generate the hash.
To get around the issue, make sure the stream you provide supports Seek (CanSeek). If it doesn't, then use a different Stream/copy your data to a stream that does (for example MemoryStream). The alternative would be for the internals of the Blob SDK to do this for you.
A workaround is that when get the stream via GetBufferlessInputStream() method, convert it to MemoryStream, then upload the MemoryStream. Then it can generate the contenthash. Sample code like below:
var stream111 = System.Web.HttpContext.Current.Request.GetBufferlessInputStream(disableMaxRequestLength: true);
//convert to memoryStream.
MemoryStream stream = new MemoryStream();
stream111.CopyTo(stream);
stream.Position = 0;
//other code
// retry already configured of Azure Storage API
await blob.UploadAsync(stream, httpHeaders: blobHttpHeaders);
Not sure why, but as per my debug, I can see when using the method GetBufferlessInputStream() in the latest SDK, during upload, it actually calls the Put Block api in the backend. And in this api, MD5 hash is not stored with the blob(Refer to here for details.). Screenshot as below:
However, when using InputStream, it calls the Put Blob api. Screenshot as below:
I am saving zip files to an AWS S3 bucket. I am now trying to create a C# .NET API that will allow me to download a specified key from the bucket and save it to a HttpResponseMessage in the Content key.
I've referred to the following question to set up my response for zip files: How to send a zip file from Web API 2 HttpGet
I have modified the code in the previous question so that it instead reads from a TransferUtility stream.
Problem is I am coming into an error when trying to extract or view the file that looks like the following:
The response I am getting back from the API looks like:
The relevant code looks like:
[HttpGet, Route("GetFileFromS3Bucket")]
public HttpResponseMessage GetFileFromS3Bucket(string keyName)
{
HttpResponseMessage response = new HttpResponseMessage();
string bucketName = "myBucket";
RegionEndpoint bucketRegion = RegionEndpoint.ARegion;
IAmazonS3 s3Client;
s3Client = new AmazonS3Client(bucketRegion);
try
{
var fileTransferUtility = new TransferUtility(s3Client);
var stream = fileTransferUtility.OpenStream(bucketName, keyName);
response.Content = new StreamContent(stream);
response.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
response.Content.Headers.ContentDisposition.FileName = keyName + ".zip";
response.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/zip");
response.StatusCode = HttpStatusCode.OK;
}
catch (Exception e)
{
response.Content = new StringContent("Something went wrong, error: " + e.Message);
response.StatusCode = HttpStatusCode.InternalServerError;
}
return response;
}
Results of troubleshooting:
The file from the Web API comes out with nearly double the expected size based on what is in S3. This is consistent across different files
Changing the bucket to be publicly accessible did not help (setting since reverted to not allowing public access)
Changing the file type to XML did not display a nicely formatted error (there was a suggestion that you may receive an XML response if an error was provided from S3)
Saving the S3 stream to a file and then saving directly to a file resulted in the correct file size. Seems safe to say the stream from S3 is not the problem
It appears that there ia a problem with the way the HTTPResponseMessage is handling the zip file. I'm unsure of whether it is actually on the server side, or whether it is on the client to parse the data and Swagger is simply incapable of doing that. Any help would be greatly appreciated.
Update 1
I do not believe this string is Base64 encoded as the result I got from converting the stream to a string is the following:
I've updated the code sample with the two lines showing the conversion from a stream to string.
Update 2
I've confirmed the issue is with how the response is handling the stream, or something in the response itself. Downloading the file stream from S3 and saving to a new file on the local computer resulted in a valid file that opened as expected.
Update 3
Link to GDrive folder with testing files: https://drive.google.com/drive/folders/1q_N3NTHz5E_nebtBQJHor3HfqUZWhGgd?usp=sharing
I unfortunately can't provide access to the original file as it contains sensitive data. The provided files are still causing the same problem however.
Interesting to note that the test file came out looking like:
The underscores on either side of the filename are quite strange.
I am running the following relevant packages:
Update 4
I've found the following UTF8 references in various files:
File: configuration91.svcinfo
I could not find anything that said anything about 'responseEncoding' anywhere in the project.
I am going to throw an answer up, because what's happening to you is unorthodox. I use S3 for many things and have done what you are doing with no problems in the past. To ensure that I am mimicking what you are doing, I duplicated your code:
[HttpGet, Route("GetFileFromS3Bucket/{keyName}")]
public HttpResponseMessage GetFileFromS3Bucket(string keyName)
{
string bucketName = "testzipfilesagain";
string awsAccessKey = "AKIAJ********A3QHOUA";
string awsSecretKey = "IYUJ9Gy2wFCQ************dCq5suFS";
IAmazonS3 client = new AmazonS3Client(awsAccessKey, awsSecretKey, RegionEndpoint.USEast1);
var fileTransferUtility = new TransferUtility(client);
var stream = fileTransferUtility.OpenStream(bucketName, "md5.zip");
var resp = new HttpResponseMessage();
resp.Content = new StreamContent(stream);
resp.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment");
resp.Content.Headers.ContentDisposition.FileName = keyName + ".zip";
resp.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/zip");
resp.StatusCode = HttpStatusCode.OK;
return resp;
}
These are the packages I have installed:
<ItemGroup>
<PackageReference Include="AWSSDK.S3" Version="3.3.111.37" />
<PackageReference Include="Microsoft.AspNetCore.Mvc.WebApiCompatShim" Version="2.2.0" />
<PackageReference Include="Swashbuckle.AspNetCore" Version="5.5.1" />
</ItemGroup>
Everything runs perfectly well.
Trying to troubleshoot your code is going to be fruitless because it works perfectly fine, but there is something wrong with your environment.
So this isn't an answer to your question, but a answer to how you can try to solve the issue at hand and get past this.
Make sure your nuget packages are up to date
Do you have any middleware injected in your pipeline? If so, what?
Post your startup.cs -- maybe something is out of order in your Configure routine.
Could you start a brand new project and try your code in that?
Can you try a small 5KB zip file and post the original and the corrupt so we can look?
I would love to get to the bottom of this as I really like to solve these types of problems.
EDIT 1
So I looked at the zip files and they have been run through a UTF8 encoding process. So, if you take your original zip file, and run this code on it:
var goodBytes = File.ReadAllBytes("Some test to upload to S3.zip");
var badBytes = File.ReadAllBytes("_Some test to upload to S3.zip.zip_");
File.WriteAllText("Some test to upload to S3.zip.utf8", Encoding.UTF8.GetString(goodBytes));
var utf8EncodedGoodBytes = File.ReadAllBytes("Some test to upload to S3.zip.utf8");
var identical = badBytes.SequenceEqual(utf8EncodedGoodBytes);
It the results are:
I am going to do some research and figure out what could be causing your stream to become UTF-8 encoded. Is there anything in your config that looks like this? Can you search your entire solution for anything that resembles "utf" or "utf8" or "utf-8"?
I am trying to send a continuous stream, from a C# application, to an ASP Core REST API.
I define a continuous stream as for example someone talking into a microphone and the sound being sent directly, without being saved to a local file) to the Rest API to be saved to file.
I have been searching a lot on Google for something like that and so far could not find anything really useful.
I have been trying to emulate it by sending a large file (297MB).
This is what I have so far for the client side:
string TARGETURL = "http://localhost:58000/api/file/";
string filePath = #"G:\Voice\Samples\The Monkey's Paw.wav";
byte[] fileContent = File.ReadAllBytes(filePath);
var dummyStream = new MemoryStream(fileContent);
var inputData = new StreamContent(dummyStream);
HttpResponseMessage response = this._httpClient.PostAsync(TARGETURL, inputData).Result;
HttpContent result = response.Content;
if (response.IsSuccessStatusCode)
{
string contents = result.ReadAsStringAsync().Result;
}
else
{
// do something
}
And for the server side:
[Route("")]
[HttpPost]
public async Task<JsonResult> Post()
{
Dictionary<string, object> rv = new Dictionary<string, object>();
try
{
string file = Path.Combine(#"G:\Voice\Samples\dummy.txt");
using (FileStream fs = new FileStream(file, FileMode.Create, FileAccess.Write,
FileShare.None, 4096, useAsync: true))
{
await Request.Body.CopyToAsync(fs);
}
// complete the transaction
rv.Add("success", true);
rv.Add("error", "");
}
catch(Exception ex)
{
}
return Json(rv);
}
When I am sending the file, the server throw the following exception:
The request's Content-Length 304137380 is larger than the request body size limit 30000000.
I know that I could increase the body size limit, but that's not a longer term solution as the stream length could get longer that any limit I set.
That's why I am trying to find a solution that send the stream by chunks for the server to rebuild and write to a file.
What you probably want to do is use a different network stack. A web application is always going to try and fit everything into HTTP. This is a very specific kind of way to communicate. And REST is built on top of these ideas as well. Things are generally though of as a document with references on the Internet, and REST is an extension to this idea.
It does however sit on top of some other great technologies that might suit your need better.
There's nothing to stop you using the internet, but maybe you need to look at possibly a UDP or TCP level implementation. Be aware that you will still be sending information in packets. There is no such thing as a constant stream of bits on the internet. A sound wave in the real world is an infinite thing, but computers are rubbish at that.
Maybe start by taking a look at using sockets and a library like NAudio.
I'm downloading in image from web to save it locally. It works great with any other image formats but it this method below fails with an argument exception when I try to read a WebP image.
private static Image GetImage(string url)
{
try
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
return Image.FromStream(response.GetResponseStream());
}
catch
{
return null;
}
}
How do you read .webp images in C#?
I found this other question that allows for converting between types but I do not want to do that WebP library for C#
Reason I'm not wanting to do it is because I think it might lose some quality. Besides, I want to know why is this not working.
The base class libraries won't help you to deal with WebP images. However, if you only want to save the received file to the disk, you don't have to even know that you are dealing with a WebP images. You can simply treat the received data as a binary blob and dump it to a file, for example using Stream.CopyTo and a FileStream.
The Content-Type HTTP header will give you the mime type of the file you're downloading, and the Content-Disposition header can provide you with a filename and extension (though you might have to do some parsing). You can access those using HttpWebResponse.ContentType and HttpWebResponse.Headers["Content-Disposition"].
#Trillian nailed it. Here is a code snippet for what I did based on his suggestion. Wanted to add code so not posting this as a comment.
To get just the image file extension, you can do this
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
string fileExt = response.ContentType.Replace("image/", string.Empty);
To get the file name with extension, you can do the following and the do parsing like I did above. It just has some more data in it.
response.Headers["Content-Disposition"];
Once you have you file name you want to save as, create a file stream and copy the response stream into it.
FileStream fs = new FileStream(targetPath + fileName, FileMode.Create);
response.GetResponseStream().CopyTo(fs);
Assuming you app has access to the destination, image should get saved. Make sure to add try catch and handle exceptions properly. Also note that FileMode.Create will overwrite if the file already exists!
I would like to load an excel file directly from an ftp site into a memory stream. Then I want to open the file in the FarPoint Spread control using the OpenExcel(Stream) method. My issue is I'm not sure if it's possible to download a file directly into memory. Anyone know if this is possible?
Yes, you can download a file from FTP to memory.
I think you can even pass the Stream from the FTP server to be processed by FarPoint.
WebRequest request = FtpWebRequest.Create("ftp://asd.com/file");
using (WebResponse response = request.GetResponse())
{
Stream responseStream = response.GetResponseStream();
OpenExcel(responseStream);
}
Using WebClient you can do nearly the same. Generally using WebClient is easier but gives you less configuration options and control (eg.: No timeout setting).
WebClient wc = new WebClient();
using (MemoryStream stream = new MemoryStream(wc.DownloadData("ftp://asd.com/file")))
{
OpenExcel(stream);
}
Take a look at WebClient.DownloadData. You should be able to download the file directory to memory and not write it to a file first.
This is untested, but something like:
var spreadSheetStream
= new MemoryStream(new WebClient().DownloadData(yourFilePath));
I'm not familiar with FarPoint though, to say whether or not the stream can be used directly with the OpenExcel method. Online examples show the method being used with a FileStream, but I'd assume any kind of Stream would be accepted.
Download file from URL to memory.
My answer does not exactly show, how to download a file for use in Excel, but shows how to create a generic-purpose in-memory byte array.
private static byte[] DownloadFile(string url)
{
byte[] result = null;
using (WebClient webClient = new WebClient())
{
result = webClient.DownloadData(url);
}
return result;
}