What I'm looking for is zip/compress S3 files without having them first downloaded to EFS or on a file system and then upload the zip file back to S3. Is there a C# way to achieve the same? I found the following post, but not sure its C# equivalent
https://www.antstack.io/blog/create-zip-using-lambda-with-files-streamed-from-s3/
I've written following code to zip files from a MemoryStream
public static void CreateZip(string zipFileName, List<FileInfo> filesToZip)
{
//zipFileName is the final zip file name
LambdaLogger.Log($"Zipping in progress for: {zipFileName}");
using (MemoryStream zipMS = new MemoryStream())
{
using (ZipArchive zipArchive = new ZipArchive(zipMS, ZipArchiveMode.Create, true))
{
//loop through files to add
foreach (var fileToZip in filesToZip)
{
//read the file bytes
byte[] fileToZipBytes = File.ReadAllBytes(fileToZip.FullName);
ZipArchiveEntry zipFileEntry = zipArchive.CreateEntry(fileToZip.Name);
//add the file contents
using (Stream zipEntryStream = zipFileEntry.Open())
using (BinaryWriter zipFileBinary = new BinaryWriter(zipEntryStream))
{
zipFileBinary.Write(fileToZipBytes);
}
}
}
using (FileStream finalZipFileStream = new FileStream(zipFileName, FileMode.Create))
{
zipMS.Seek(0, SeekOrigin.Begin);
zipMS.CopyTo(finalZipFileStream);
}
}
}
But problem is how to make it read file directly from S3 and upload the compressed file.
public static async Task CreateZipFile(List<List<KeyVersion>> keyVersions)
{
using MemoryStream zipMS = new MemoryStream();
using (ZipArchive zipArchive = new ZipArchive(zipMS, ZipArchiveMode.Create, true))
{
foreach (var key in keyVersions)
{
foreach (var fileToZip in key)
{
GetObjectRequest request = new GetObjectRequest
{
BucketName = "dev-s3-zip-bucket",
Key = fileToZip.Key
};
using GetObjectResponse response = await s3client.GetObjectAsync(request);
using Stream responseStream = response.ResponseStream;
ZipArchiveEntry zipFileEntry = zipArchive.CreateEntry(fileToZip.Key);
//add the file contents
using Stream zipEntryStream = zipFileEntry.Open();
await responseStream.CopyToAsync(zipEntryStream);
}
}
zipArchive.Dispose();
}
zipMS.Seek(0, SeekOrigin.Begin);
var fileTxfr = new TransferUtility(s3client);
await fileTxfr.UploadAsync(zipMS, "dev-s3-zip-bucket", "test.zip");
}
Related
I'm having strange problem with this piece of code which basically zips files (docs) and uploads them to blob storage.
v11SDK: (docs)
var blockBlobClient = new BlockBlobClient(ConnectionString, ContainerName, "test-blob.zip");
// Saved zip is valid
// using (FileStream zipStream = new FileStream(#"C:\Users\artur\Desktop\test-local.zip", FileMode.OpenOrCreate))
// Uploaded zip is invalid
using (var stream = await blockBlobClient.OpenWriteAsync(true))
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create))
{
var readmeEntry = archive .CreateEntry("Readme.txt");
using (StreamWriter writer = new StreamWriter(readmeEntry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
await stream.FlushAsync();
}
v12SDK: (docs)
var blobClient = new BlobClient(ConnectionString, InputContainerName, "test-blob.zip");
using var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create))
{
var readmeEntry = archive.CreateEntry("Readme.txt");
using StreamWriter writer = new StreamWriter(readmeEntry.Open());
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
await writer.FlushAsync();
}
stream.Position = 0;
await blobClient.UploadAsync(stream, true);
await stream.FlushAsync();
}
Saving zip file locally produces a valid zip (164 bytes). Saving zip to blob storage (using storage emulator) produces invalid zip (102 bytes).
I can't figure out why
Here is the correct code.
The problem was premature disposing of inner stream by ZipArchive. Note in my code below, I have passed leaveInnerStreamOpen as true while creating ZipArchive since we are already disposing stream in the outer using. Also for V11 code, I have switched to MemoryStream instead of OpenWrite of blob stream since did not have control to set stream position to 0 if we use OpenWrite. And you don't need any Flush :)
v11SDK:
var blockBlobClient = new BlockBlobClient(ConnectionString, ContainerName, "test-blob.zip");
using var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create, true))
{
var readmeEntry = archive.CreateEntry("Readme.txt");
using (StreamWriter writer = new StreamWriter(readmeEntry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
}
stream.Position = 0;
await blockBlobClient.UploadAsync(stream);
v12SDK:
var blobClient = new BlobClient(ConnectionString, InputContainerName, "test-blob.zip");
using var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create, true))
{
var readmeEntry = archive.CreateEntry("Readme.txt");
using StreamWriter writer = new StreamWriter(readmeEntry.Open());
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
}
stream.Position = 0;
await blobClient.UploadAsync(stream, true);
I am receiving a IFormFileCollection and I was wondering how can I archive this collection into a zip/rar/whatever archive file, and send this archive as a stream somewhere else to be stored?
I want to work only in memory via Stream (s) since I will send it over HTTP later on.
class ArchiveService {
public Stream ArchiveFiles(string archiveName, IEnumerable<IFormFile> files) {
using MemoryStream stream = new MemoryStream();
using (System.IO.Compression.ZipArchive archive = ZipFile.Open([in memory!], ZipArchiveMode.Create)) {
foreach (var file in files) {
archive.CreateEntry(file.FileName, CompressionLevel.Fastest);
}
//something like -> archive.CopyTo(stream);
}
return stream;
}
}
Create an archive in memory and traverse the collection, adding the files to the archive.
The returned stream will contain the files compressed into the archive.
You can then do as you wish with the stream
public class ArchiveService {
public Stream ArchiveFiles(IEnumerable<IFormFile> files) {
MemoryStream stream = new MemoryStream();
using (ZipArchive archive = new ZipArchive(stream, ZipArchiveMode.Create, leaveOpen: true)) {
foreach (IFormFile file in files) {
var entry = archive.CreateEntry(file.FileName, CompressionLevel.Fastest);
using (Stream target = entry.Open()) {
file.CopyTo(target);
}
}
}
stream.Position = 0;
return stream;
}
public async Task<Stream> ArchiveFilesAsync(IEnumerable<IFormFile> files) {
MemoryStream stream = new MemoryStream();
using (ZipArchive archive = new ZipArchive(stream, ZipArchiveMode.Create, leaveOpen: true)) {
foreach (IFormFile file in files) {
var entry = archive.CreateEntry(file.FileName, CompressionLevel.Fastest);
using (Stream target = entry.Open()) {
await file.OpenReadStream().CopyToAsync(target);
}
}
}
stream.Position = 0;
return stream;
}
}
I am using Zip Archive to create a zip folder with various files and subfolders and returning it as a memory stream like so.
public MemoryStream CreateAZipFolder(){
var stMarged = new System.IO.MemoryStream();
stMarged.Position = 0;
using (MemoryStream zipStream = new MemoryStream())
{
using (ZipArchive zip = new ZipArchive(zipStream, ZipArchiveMode.Create, true))
{
string[] fileEntries = Directory.GetFiles(#"C:\Applications\folder");
foreach (var fileName in fileEntries)
{
zip.CreateEntryFromFile(fileName, "Applications/folder/" + Path.GetFileName(fileName),
CompressionLevel.Optimal);
}
ZipArchiveEntry batchEntry = zip.CreateEntry("mybatchFile.bat");
using (StreamWriter writer = new StreamWriter(batchEntry.Open()))
{
writer.Write(batchFile);
}
//Add the xml file to zip folder
ZipArchiveEntry entry = zip.CreateEntry("nCounterConfig.xml");
using (StreamWriter writer = new StreamWriter(entry.Open()))
{
writer.Write(xdoc.OuterXml);
}
}
zipStream.Position = 0;
return zipStream;
I would like to add a directory with sub directories and files to this memory stream. I found that ZipFile has a method "CreateFromDirectory" which would be ideal except it requires a paramater for an output folder the method also does not have a return type. How can i zip all the files and subfolders in a directory and add them to my memory stream using ZipFile?
something like this
zip.CreateEntry(ZipFile.CreateFromDirectory(
#"C:\morefilestozip\", "",
CompressionLevel.Fastest, true));
I have got a code from this link:
http://www.codeproject.com/Tips/515704/Archive-Multiple-Files-In-Zip-Extract-Zip-Archive
that compresses and extracts a file in zip format.
However the compressing part in the code just creates an empty zipfile, so how can i add files programmatically in this zip archieve?
i have checked the doc for the ziparchieve class and it has a method for .net called CreateEntryFromFile(String, String) , however this method doesn't apply for .net windows store version.
this is the code we are concerned with:
private async void ZipClick(object sender, RoutedEventArgs e)
{
FileSavePicker picker = new FileSavePicker();
picker.FileTypeChoices.Add("Zip Files (*.zip)", new List<string> { ".zip" });
picker.SuggestedStartLocation = PickerLocationId.Desktop;
picker.SuggestedFileName = "1";
zipFile = await picker.PickSaveFileAsync();
using (var zipStream = await zipFile.OpenStreamForWriteAsync())
{
using (ZipArchive zip = new ZipArchive(zipStream, ZipArchiveMode.Create))
{
foreach (var file in storeFile)
{
ZipArchiveEntry entry = zip.CreateEntry(file.Name);
using (Stream ZipFile = entry.Open())
{
byte[] data = await GetByteFromFile(file);
ZipFile.Write(data, 0, data.Length);
}
}
}
}
}
I misunderstood the sample code, it actually can archive the the files you pass it for the first time, but i'm not sure if the same method can be used to add to add other files in the same zip file later.
public async void zipit(StorageFile zipFile, StorageFile file)
{
using (var zipToOpen = await zipFile.OpenStreamForWriteAsync())
{
using (ZipArchive archive = new ZipArchive(zipToOpen, ZipArchiveMode.Update))
{
ZipArchiveEntry readmeEntry = archive.CreateEntry(file.Name);
using (Stream writer = readmeEntry.Open())
{
//writer.WriteLine("Information about this package.");
//writer.WriteLine("========================");
byte[] data = await GetByteFromFile(file);
writer.Write(data, 0, data.Length);
}
}
}
}
where "zipFile" is the file you have chose to archive in (destination ) and "file" is the original non zipped file.
I create a zip file in a controller from a byte array and I return the zip file as a fileresult. When I download the zip File and extract the file, it is corrupt. I'm doing it this way:
byte[] fileBytes =array
MemoryStream fileStream = new MemoryStream(fileBytes);
MemoryStream outputStream = new MemoryStream();
fileStream.Seek(0, SeekOrigin.Begin);
using (ZipFile zipFile = new ZipFile())
{
zipFile.AddEntry(returnFileName, fileStream);
zipFile.Save(outputStream);
}
outputStream.Position = 0;
FileStreamResult fileResult = new FileStreamResult(outputStream, System.Net.Mime.MediaTypeNames.Application.Zip);
fileResult.FileDownloadName = returnFileName + ".zip";
return fileResult;
You might be unlucky hitting one of the open bugs in DotNetZip. There is e.g. an issue depending on the file size (https://dotnetzip.codeplex.com/workitem/14087).
Unfortunately, DotNetZip has some critical issues and the project seems no longer be actively be maintained. Better alternatives would be to use SharpZipLib (if you comply with their GPL-based license), or one of the .NET ports of zlib.
If you are on .NET 4.5 you can use the built-in classes in the System.IO.Compression namespace. The following sample can be found in the documentation of the ZipArchive class:
using System;
using System.IO;
using System.IO.Compression;
namespace ConsoleApplication
{
class Program
{
static void Main(string[] args)
{
using (var zipToOpen =
new FileStream(#"c:\tmp\release.zip", FileMode.Open))
{
using (var archive =
new ZipArchive(zipToOpen, ZipArchiveMode.Update))
{
var readmeEntry = archive.CreateEntry("Readme.txt");
using (var writer = new StreamWriter(readmeEntry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
}
}
}
}
}
public class HomeController : Controller
{
public FileResult Index()
{
FileStreamResult fileResult = new FileStreamResult(GetZippedStream(), System.Net.Mime.MediaTypeNames.Application.Zip);
fileResult.FileDownloadName = "result" + ".zip";
return fileResult;
}
private static Stream GetZippedStream()
{
byte[] fileBytes = Encoding.ASCII.GetBytes("abc");
string returnFileName = "something";
MemoryStream fileStream = new MemoryStream(fileBytes);
MemoryStream resultStream = new MemoryStream();
using (ZipFile zipFile = new ZipFile())
{
zipFile.AddEntry(returnFileName, fileStream);
zipFile.Save(resultStream);
}
resultStream.Position = 0;
return resultStream;
}
}