I want to download files from Azure using C# then stream those into MemoryStream after that return/display to the user in Front-end with a link (Azure URI - which goes to the Azure blob) and the user will be able to see those PDF files in the browser or download them. There are multiple blobs/files in Azure so, I want to loop through each file and download to stream for example: using a foreach.
I'm not sure how can I reference those blobs CloudBlockBlob blockBlob = container.GetBlockBlobReference(fileName); as here I could give a name of the specific file but I've multiple files so not sure what to go here "fileName".
Code:
var files = container.ListBlobs();
foreach (var file in files)
{
using (var memoryStream = new MemoryStream())
{
CloudBlockBlob blockBlob = container.GetBlockBlobReference(fileName);
blockBlob.DownloadToStream(memoryStream);
}
}
I'm not sure if I'm looping correcting right now in the code and downloading every blob?
Also, I tried replacing fileName with file.Uri.Segments.Last() -
I guess which gets the name of blobs.
The problem I'm having is that this foreach is just getting me one PDF file whenever I try to use the links in front-end. So, I need to know how can I properly loop through each file and download them?
So, I need to know how can I properly loop through each file and download them?
We can't download the mutiple files from the memory directly. If zip file is acceptable, you could use a compressed file such as a zip file to transfer multiple files instead. The following is my demo code, it works correctly on my side.
using (var ms = new MemoryStream())
{
using (var zipArchive = new ZipArchive(ms, ZipArchiveMode.Create, true))
{
foreach (var file in files)
{
if (file.GetType() != typeof(CloudBlockBlob)) continue;
var blob = (CloudBlockBlob) file;
var entry = zipArchive.CreateEntry(blob.Name, CompressionLevel.Fastest);
using (var entryStream = entry.Open())
{
CloudBlockBlob blockBlob = container.GetBlockBlobReference(blob.Name);
blockBlob.DownloadToStream(entryStream);
}
}
}
}
Related
I am Using MVC.Net application and I want to download multiple files as zipped. I have written code with memory stream and ZipArchive in my controller action method. With that code I am able to successfully download the files as zipped folder. But when I unzipped those and trying to open them then I am getting the error as below
opening word document with Microsoft word - Word found unreadable content error
opening image file(.png) - dont support file format error
Here is my controller method code to zip the files
if (sendApplicationFiles != null && sendApplicationFiles.Any())
{
using (var compressedFileStream = new MemoryStream())
{
// Create an archive and store the stream in memory.
using (var zipArchive = new ZipArchive(compressedFileStream, ZipArchiveMode.Create, true))
{
foreach (var file in sendApplicationFiles)
{
// Create a zip entry for each attachment
var zipEntry = zipArchive.CreateEntry(file.FileName);
// Get the stream of the attachment
using (var originalFileStream = new MemoryStream(file.FileData))
using (var zipEntryStream = zipEntry.Open())
{
// Copy the attachment stream to the zip entry stream
originalFileStream.CopyTo(zipEntryStream);
}
}
}
return new FileContentResult(compressedFileStream.ToArray(), "application/zip") { FileDownloadName = "Filename.zip" };
}
}
Expecting the document content should load without error
I did not investigate your code, but this link might help. There are few examples.
https://learn.microsoft.com/en-us/dotnet/standard/io/how-to-compress-and-extract-files
I have a zip file(.Exe - Self-extracting zip file) that can be extracted using 7zip. As I want to automate the extraction process, I used the below C# code. It is working for the normal 7z files. But facing this issue 'Cannot access the closed Stream', when I trying to extract the specific self-extracting (.Exe) zip file. Fyi. Manually I ensured the 7zip command line version is unzipping the file.
using (SevenZipExtractor extract = new SevenZipExtractor(zipFileMemoryStream))
{
foreach (ArchiveFileInfo archiveFileInfo in extract.ArchiveFileData)
{
if (!archiveFileInfo.IsDirectory)
{
using (var memory = new MemoryStream())
{
string shortFileName = Path.GetFileName(archiveFileInfo.FileName);
extract.ExtractFile(archiveFileInfo.Index, memory);
byte[] content = memory.ToArray();
file = new MemoryStream(content);
}
}
}
}
The zip file is in Azure blob storage. I dont know how to get the extracted files in the blob storage.
Here is one of the workarounds that has worked for me. Instead of 7Zip I have used ZipArchive.
ZipArchive archive = new ZipArchive(myBlob);
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(destinationStorage);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = blobClient.GetContainerReference(destinationContainer);
foreach(ZipArchiveEntry entry in archive.Entries) {
log.LogInformation($"Now processing {entry.FullName}");
string valideName = Regex.Replace(entry.Name, # "[^a-zA-Z0-9\-]", "-").ToLower();
CloudBlockBlob blockBlob = container.GetBlockBlobReference(valideName);
using(var fileStream = entry.Open()) {
await blockBlob.UploadFromStreamAsync(fileStream);
}
}
REFERENCE:
How to Unzip Automatically your Files with Azure Function v2
I want to upload zip file in small chunks (less than 5 MB) to blob containers in Microsoft Azure Storage. I already configured 4 MB chunk limits in BlobRequestOptions but when I run my code and check the memory usage in Azure Cloud, its not uploading in chunks. I am using C# .NET Core. Because I want to zip files that are already located in Azure Cloud, so first I am downloading the individual files to stream, adding stream to zip archive and then uploading the zip back to the cloud. The following is my code:
if (CloudStorageAccount.TryParse(_Appsettings.GetSection("StorConf").GetSection("StorageConnection").Value, out CloudStorageAccount storageAccount)) {
CloudBlobClient BlobClient = storageAccount.CreateCloudBlobClient();
TimeSpan backOffPeriod = TimeSpan.FromSeconds(2);
int retryCount = 1;
BlobRequestOptions bro = new BlobRequestOptions() {
SingleBlobUploadThresholdInBytes = 4096 * 1024, // 4MB
ParallelOperationThreadCount = 1,
RetryPolicy = new ExponentialRetry(backOffPeriod, retryCount),
// new
ServerTimeout = TimeSpan.MaxValue,
MaximumExecutionTime = TimeSpan.FromHours(3),
//EncryptionPolicy = policy
};
// set blob request option for created blob client
BlobClient.DefaultRequestOptions = bro;
// using specified container which comes via transaction id
CloudBlobContainer container = BlobClient.GetContainerReference(transaction id);
using(var zipArchiveMemoryStream = new MemoryStream()) {
using(var zipArchive = new ZipArchive(zipArchiveMemoryStream, ZipArchiveMode.Create, true)) // new
{
foreach(FilesListModel FileName in filesList) {
if (await container.ExistsAsync()) {
CloudBlob file = container.GetBlobReference(FileName.FileName);
if (await file.ExistsAsync()) {
// zip: get stream and add zip entry
var entry = zipArchive.CreateEntry(FileName.FileName, CompressionLevel.Fastest);
// approach 1
using(var entryStream = entry.Open()) {
await file.DownloadToStreamAsync(entryStream, null, bro, null);
await entryStream.FlushAsync();
entryStream.Close();
}
} else {
downlReady = "false";
}
} else {
// case: Container does not exist
//return BadRequest("Container does not exist");
}
}
}
if (downlReady == "true") {
string zipFileName = "sample.zip";
CloudBlockBlob zipBlockBlob = container.GetBlockBlobReference(zipFileName);
zipArchiveMemoryStream.Position = 0;
//zipArchiveMemoryStream.Seek(0, SeekOrigin.Begin);
// new
zipBlockBlob.Properties.ContentType = "application/x-zip-compressed";
await zipArchiveMemoryStream.FlushAsync();
await zipBlockBlob.UploadFromStreamAsync(zipArchiveMemoryStream, zipArchiveMemoryStream.Length, null, bro, null);
}
zipArchiveMemoryStream.Close();
}
}
The following is a snapshot of the memory usage (see private_Memory) in azure cloud kudu process explorer:
memory usage
Any suggestions would be really helpful. Thank you.
UPDATE 1:
To make it more clear. I have files which are already located in Azure blob storage. Now I want to read the files from the container, create a ZIP which contains all of my files. The major challenge here is that my code is obviously loading all files into memory to create the zip. If and how it is possible to read files from a container and write the ZIP file back into the same container in parallel/pieces, so that my Azure web app does NOT need to load the whole files into memory? Ideally I read the files in pieces and also start writing the zip already so that my Azure web app consumes less memory.
I have found the solution by referring to this stackoverflow article:
How can I dynamically add files to a zip archive stored in Azure blob storage?
The way to do is to simultaneously write to the zip memory stream while reading / downloading the input files.
Below is my code snippet:
using (var zipArchiveMemoryStream = await zipBlockBlob.OpenWriteAsync(null, bro, null))
using (var zipArchive = new ZipArchive(zipArchiveMemoryStream, ZipArchiveMode.Create))
{
foreach (FilesListModel FileName in filesList)
{
if (await container.ExistsAsync())
{
CloudBlob file = container.GetBlobReference(FileName.FileName);
if (await file.ExistsAsync())
{
// zip: get stream and add zip entry
var entry = zipArchive.CreateEntry(FileName.FileName, CompressionLevel.Fastest);
// approach 1
using (var entryStream = entry.Open())
{
await file.DownloadToStreamAsync(entryStream, null, bro, null);
entryStream.Close();
}
}
}
}
zipArchiveMemoryStream.Close();
}
I am making a GET request using HttpClient to download a zip file from the internet.
I want to extract all the files contained in the zip file without saving the zip file to disk.
Currently, I am able to download and save the zip file to disk, extract its contents and then delete the zip file from disk. This perfectly fine. However, I want to optimize the process.
I found a way to extract the contents directly from the downloaded zip stream but I have to specify the filenames and extensions.
I am not sure how to extract the contents while preserving their original filenames and extensions without me specifying them.
Current Approach:
string requestUri = "https://www.nuget.org/api/v2/package/" + PackageName + "/" + PackageVersion;
HttpResponseMessage response = await client.GetAsync(requestUri);
response.EnsureSuccessStatusCode();
using Stream PackageStream = await response.Content.ReadAsStreamAsync();
SaveStream($"{DownloadPath}.zip", PackageStream);
ZipFile.ExtractToDirectory($"{DownloadPath}.zip", ExtractPath);
File.Delete($"{DownloadPath}.zip");
// Directly extract Zip contents without saving file and without losing filename and extension
using (ZipArchive archive = new ZipArchive(await response.Content.ReadAsStreamAsync()))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
using (Stream stream = entry.Open())
{
using (FileStream file = new FileStream("file.txt", FileMode.Create, FileAccess.Write))
{
stream.CopyTo(file);
}
}
}
}
.NET 4.8
.NET Core 3.1
C# 8.0
Any help in this regards would be appreciated.
Please feel free to comment on alternative approaches or suggestions.
Thank you in advance.
ZipArchiveEntry has a Name and FullName property that can be used to get the names of the files within the archive while preserving their original filenames and extensions
The FullName property contains the relative path, including the subdirectory hierarchy, of an entry in a zip archive. (In contrast, the Name property contains only the name of the entry and does not include the subdirectory hierarchy.)
For example
using (ZipArchive archive = new ZipArchive(await response.Content.ReadAsStreamAsync())) {
foreach (ZipArchiveEntry entry in archive.Entries) {
using (Stream stream = entry.Open()) {
string destination = Path.GetFullPath(Path.Combine(downloadPath, entry.FullName));
var directory = Path.GetDirectoryName(destination);
if (!Directory.Exists(directory))
Directory.CreateDirectory(directory);
using (FileStream file = new FileStream(destination, FileMode.Create, FileAccess.Write)) {
await stream.CopyToAsync(file);
}
}
}
}
will extract the files in the same subdirectory hierarchy as they were stored in the archive while if entry.Name was used, all the files would be extracted to the same location.
I am working on a project where I need to get Zip files from S3 Bucket.
I was able to copy these files one at a time to my EC2 instance using
foreach (S3Object o in response.S3Objects)
{
GetObjectRequest requests = new GetObjectRequest();
requests.BucketName = "mybucket";
requests.Key = o.Key;
GetObjectResponse responses = client.GetObject(requests);
responses.WriteResponseStreamToFile(#"D:\myfile.zip");
Console.WriteLine("{0}\t{1}\t{2}", o.Key, o.Size, o.LastModified);
}
but I would like to unzip these files on the fly to a specific location instead of copy them locally.
I tried the following but it did not work
using (ZipArchive archive = ZipFile.OpenRead(responses.ResponseStream.ToString())) //unzip file
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
archive.ExtractToDirectory(myPath);
}
}
}
Thanks
ZipFile.OpenRead() takes a file name of a local file so that won't work. You can, however, unzip an archive in a MemoryStream (see this answer and this answer).