Since System.IO.Compression seems to be out of reach for now if I want to use both dotnet core + net461, I've tried with SharpCompress.
The "read zip" part was easy, but I am having trouble finding out how to write to a zip stream.
The wiki of the project is a bit outdated. This is the only example that I've found that applies to writing to streams. I've tried to follow it and adapt it to my needs, but I am stuck at the exception it throws:
using Microsoft.VisualStudio.TestTools.UnitTesting;
using SharpCompress.Common;
using SharpCompress.Compressors.Deflate;
using SharpCompress.Writers;
using System;
using System.IO;
namespace DbManager.DjdbCore.Tests
{
[TestClass]
public class ZipTests
{
public ZipTests()
{
Directory.SetCurrentDirectory(AppContext.BaseDirectory);
}
[TestMethod]
public void Test()
{
var zip = File.OpenWrite(#"..\..\..\..\..\test-resources\zip_file_test.zip");
var writerOptions = new WriterOptions(CompressionType.Deflate);
var zipWriter = WriterFactory.Open(zip, ArchiveType.Zip, writerOptions);
var memoryStream = new MemoryStream();
var binaryWriter = new BinaryWriter(memoryStream);
binaryWriter.Write("Test string inside binary file - text to fill it up: qoiwjqefñlawijfñlaskdjfioqwjefñalskvndñaskvnqo`wiefowainvñaslkfjnwpowiqjfeopwiqjnfjñlaskdjfñlasdfjiowiqjefñaslkdjfñalskjfpqwoiefjqw");
var deflateStream = new DeflateStream(memoryStream, SharpCompress.Compressors.CompressionMode.Compress);
deflateStream.Write(memoryStream.ToArray(), 0, Convert.ToInt32(memoryStream.Length));
// EXCEPTION: SharpCompress.Compressors.Deflate.ZlibException: 'Cannot Read after Writing.'
// Source code: if (_streamMode != StreamMode.Reader) { throw new ZlibException("Cannot Read after Writing."); }
zipWriter.Write("test_file_inside_zip.bin", deflateStream, DateTime.Now);
zip.Flush();
zipWriter.Dispose();
zip.Dispose();
}
}
}
In case it helps, this is what I used (and it worked, but only in dotnet core) using the library System.IO.Compression:
private void WriteAsZipBinary()
{
//Open the zip file if it exists, else create a new one
var zip = ZipPackage.Open(this.FileFullPath, FileMode.OpenOrCreate, FileAccess.ReadWrite);
var zipStream = ZipManager.GetZipWriteStream(zip, nameOfFileInsideZip);
var memoryStream = new MemoryStream();
var binaryWriter = new BinaryWriter(memoryStream);
// Here is where strings etc are written to the binary file:
WriteStuffInBinaryStream(ref binaryWriter);
//Read all of the bytes from the file to add to the zip file
byte[] bites = new byte[Convert.ToInt32(memoryStream.Length - 1) + 1];
memoryStream.Position = 0;
memoryStream.Read(bites, 0, Convert.ToInt32(memoryStream.Length));
binaryWriter.Dispose();
binaryWriter = null;
memoryStream.Dispose();
memoryStream = null;
zipStream.Position = 0;
zipStream.Write(bites, 0, bites.Length);
zip.Close();
}
public static Stream GetZipWriteStream(Package zip, string renamedFileName)
{
//Replace spaces with an underscore (_)
string uriFileName = renamedFileName.Replace(" ", "_");
//A Uri always starts with a forward slash "/"
string zipUri = string.Concat("/", Path.GetFileName(uriFileName));
Uri partUri = new Uri(zipUri, UriKind.Relative);
string contentType = "Zip"; // System.Net.Mime.MediaTypeNames.Application.Zip;
//The PackagePart contains the information:
// Where to extract the file when it's extracted (partUri)
// The type of content stream (MIME type): (contentType)
// The type of compression: (CompressionOption.Normal)
PackagePart pkgPart = zip.CreatePart(partUri, contentType, CompressionOption.Normal);
//Compress and write the bytes to the zip file
return pkgPart.GetStream();
}
I'll post here the answer on github from #adamhathcock (the owner of the project):
[TestMethod]
public void Test()
{
var writerOptions = new WriterOptions(CompressionType.Deflate);
using(var zip = File.OpenWrite(#"..\..\..\..\..\test-resources\zip_file_test.zip"))
using(var zipWriter = WriterFactory.Open(zip, ArchiveType.Zip, writerOptions))
{
var memoryStream = new MemoryStream();
var binaryWriter = new BinaryWriter(memoryStream);
binaryWriter.Write("Test string inside binary file - text to fill it up: qoiwjqefñlawijfñlaskdjfioqwjefñalskvndñaskvnqo`wiefowainvñaslkfjnwpowiqjfeopwiqjnfjñlaskdjfñlasdfjiowiqjefñaslkdjfñalskjfpqwoiefjqw");
memoryStream.Position = 0;
zipWriter.Write("test_file_inside_zip.bin", memoryStream, DateTime.Now);
}
}
2 things:
You forgot to reset the MemoryStream after writing to it so it can be read.
You don't need to manually use the DeflateStream. You've told the ZipWriter what compression to use. If it worked, you would have double compressed the bytes which would be garbage really.
Related
I have files (from 3rd parties) that are being FTP'd to a directory on our server. I download them and process them even 'x' minutes. Works great.
Now, some of the files are .zip files. Which means I can't process them. I need to unzip them first.
FTP has no concept of zip/unzipping - so I'll need to grab the zip file, unzip it, then process it.
Looking at the MSDN zip api, there seems to be no way i can unzip to a memory stream?
So is the only way to do this...
Unzip to a file (what directory? need some -very- temp location ...)
Read the file contents
Delete file.
NOTE: The contents of the file are small - say 4k <-> 1000k.
Zip compression support is built in:
using System.IO;
using System.IO.Compression;
// ^^^ requires a reference to System.IO.Compression.dll
static class Program
{
const string path = ...
static void Main()
{
using(var file = File.OpenRead(path))
using(var zip = new ZipArchive(file, ZipArchiveMode.Read))
{
foreach(var entry in zip.Entries)
{
using(var stream = entry.Open())
{
// do whatever we want with stream
// ...
}
}
}
}
}
Normally you should avoid copying it into another stream - just use it "as is", however, if you absolutely need it in a MemoryStream, you could do:
using(var ms = new MemoryStream())
{
stream.CopyTo(ms);
ms.Position = 0; // rewind
// do something with ms
}
You can use ZipArchiveEntry.Open to get a stream.
This code assumes the zip archive has one text file.
using (FileStream fs = new FileStream(path, FileMode.Open))
using (ZipArchive zip = new ZipArchive(fs) )
{
var entry = zip.Entries.First();
using (StreamReader sr = new StreamReader(entry.Open()))
{
Console.WriteLine(sr.ReadToEnd());
}
}
using (ZipArchive archive = new ZipArchive(webResponse.GetResponseStream()))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
Stream s = entry.Open();
var sr = new StreamReader(s);
var myStr = sr.ReadToEnd();
}
}
Looks like here is what you need:
using (var za = ZipFile.OpenRead(path))
{
foreach (var entry in za.Entries)
{
using (var r = new StreamReader(entry.Open()))
{
//your code here
}
}
}
You can use SharpZipLib among a variety of other libraries to achieve this.
You can use the following code example to unzip to a MemoryStream, as shown on their wiki:
using ICSharpCode.SharpZipLib.Zip;
// Compresses the supplied memory stream, naming it as zipEntryName, into a zip,
// which is returned as a memory stream or a byte array.
//
public MemoryStream CreateToMemoryStream(MemoryStream memStreamIn, string zipEntryName) {
MemoryStream outputMemStream = new MemoryStream();
ZipOutputStream zipStream = new ZipOutputStream(outputMemStream);
zipStream.SetLevel(3); //0-9, 9 being the highest level of compression
ZipEntry newEntry = new ZipEntry(zipEntryName);
newEntry.DateTime = DateTime.Now;
zipStream.PutNextEntry(newEntry);
StreamUtils.Copy(memStreamIn, zipStream, new byte[4096]);
zipStream.CloseEntry();
zipStream.IsStreamOwner = false; // False stops the Close also Closing the underlying stream.
zipStream.Close(); // Must finish the ZipOutputStream before using outputMemStream.
outputMemStream.Position = 0;
return outputMemStream;
// Alternative outputs:
// ToArray is the cleaner and easiest to use correctly with the penalty of duplicating allocated memory.
byte[] byteArrayOut = outputMemStream.ToArray();
// GetBuffer returns a raw buffer raw and so you need to account for the true length yourself.
byte[] byteArrayOut = outputMemStream.GetBuffer();
long len = outputMemStream.Length;
}
Ok so combining all of the above, suppose you want to in a very simple way take a zip file called
"file.zip" and extract it to "C:\temp" folder. (Note: This example was only tested for compress text files) You may need to do some modifications for binary files.
using System.IO;
using System.IO.Compression;
static void Main(string[] args)
{
//Call it like this:
Unzip("file.zip",#"C:\temp");
}
static void Unzip(string sourceZip, string targetPath)
{
using (var z = ZipFile.OpenRead(sourceZip))
{
foreach (var entry in z.Entries)
{
using (var r = new StreamReader(entry.Open()))
{
string uncompressedFile = Path.Combine(targetPath, entry.Name);
File.WriteAllText(uncompressedFile,r.ReadToEnd());
}
}
}
}
I am trying to read a file within a zip to check if that file has a certain string in it. But I can seem to get the "file" (memory stream) into a string in order to search it.
When I use the following code "stringOfStream" is always blank, what am I doing wrong? The reader always has a length and read byte returns different numbers.
using (ZipFile zip = ZipFile.Read(currentFile.FullName))
{
ZipEntry e = zip[this.searchFile.Text];
using (MemoryStream reader = new MemoryStream())
{
e.Extract(reader);
var stringReader = new StreamReader(reader);
var stringOfStream = stringReader.ReadToEnd();
}
}
Thanks
I think when you call Extract the position of the stream goes to the end of the file, so you need to reposition again to get the data.
Can you try this please :
using (ZipFile zip = ZipFile.Read(currentFile.FullName))
{
ZipEntry e = zip[this.searchFile.Text];
using (MemoryStream reader = new MemoryStream())
{
e.Extract(reader);
reader.Position = 0;
var stringReader = new StreamReader(reader);
var stringOfStream = stringReader.ReadToEnd();
}
}
Check if it works or not.
This is a very strange question.
I using C# to create a pass.json and save it to memoryStream, it work normally. After that I create the manifest.json SHA1 data which including that pass.json, the string of manifest.json like this and it is totally correct.
{"icon.png": "9423bd00e2b01c59a3265c38b5062fac7da0752d",
"icon#2x.png": "4d1db55bdaca70b685c013529a1c0dcbd7046524",
"logo.png": "ee5b053e63dbfe3b78378c15d163331d68a0ede8",
"logo#2x.png": "2f9e3a55bded1163620719a4d6c1ad496ed40c17",
"pass.json": "fd68bf77757d3057263a9aca0e5110ddd933934a"}
After generate pkpass as my phone, it can't open. I change the pass.json SHA1 code as "fd68bf77757d3057263a9aca0e5110ddd933934a" without using a value to save it, it work.
The coding like following:
// This version run success
var strPass = JavascriptSerialize(details);
var sw = new StreamWriter(assetsFolder + #"pass.json");
sw.Write(strPass);
sw.Close();
manifest.passjson = GetSha1Hash(assetsFolder + manifest.GetAssetBoardingPass(libPkPass_object_boardingPass.JsonObjects.AssetTypes.passjson));
//manifest.passjson = "2f9e3a55bded1163620719a4d6c1ad496ed40c17"
// end
// This version run fail
var strPass = JavascriptSerialize(details);
MemoryStream stream = new MemoryStream();
StreamWriter writer = new StreamWriter(strPass);
writer.Write(s);
writer.Flush();
stream.Position = 0;
var a = GetSha1HashMemory(passStream);
private static string GetSha1HashMemory(Stream passStream)
{
//var bs = new BufferedStream(passStream);
using (SHA1Managed sha = new SHA1Managed())
{
byte[] checksum = sha.ComputeHash(passStream);
string sendCheckSum = BitConverter.ToString(checksum)
.Replace("-", string.Empty);
return sendCheckSum.ToString().ToLower();
}
}
manifest.passjson = a;
//manifest.passjson = "2f9e3a55bded1163620719a4d6c1ad496ed40c17" (same data )
//end
What is going on?????? I can find out any question that string is wrong.
The pkpass provide in here (sendspace).
Can any body told me where is wrong?
Big Thank!
Two mistakes :
ComputeHash(Stream) and using Stream
ComputeHash(Stream) : ComputeHash stream only using System.IO.Stream, but not MemoryStream, change to ComputeHash(bytes[]) can handle it
using Stream: I try to pass the stream to other function, it is not a good example, the stream need to create a new one and it may replace some bytes at your computer stream. In this case, I just need to call this function will out open new one, it will fix
StringBuilder formatted;
using (var sha1 = new SHA1Managed())
{
//var bytePass = ReadFully(passStream);
var bytePass = passStream.ToArray();
var hash = sha1.ComputeHash(bytePass);
formatted = new StringBuilder(2 * hash.Length);
foreach (var b in hash)
{
formatted.AppendFormat("{0:X2}", b);
}
}
manifest.passjson = formatted.ToString().ToLower();
I have files (from 3rd parties) that are being FTP'd to a directory on our server. I download them and process them even 'x' minutes. Works great.
Now, some of the files are .zip files. Which means I can't process them. I need to unzip them first.
FTP has no concept of zip/unzipping - so I'll need to grab the zip file, unzip it, then process it.
Looking at the MSDN zip api, there seems to be no way i can unzip to a memory stream?
So is the only way to do this...
Unzip to a file (what directory? need some -very- temp location ...)
Read the file contents
Delete file.
NOTE: The contents of the file are small - say 4k <-> 1000k.
Zip compression support is built in:
using System.IO;
using System.IO.Compression;
// ^^^ requires a reference to System.IO.Compression.dll
static class Program
{
const string path = ...
static void Main()
{
using(var file = File.OpenRead(path))
using(var zip = new ZipArchive(file, ZipArchiveMode.Read))
{
foreach(var entry in zip.Entries)
{
using(var stream = entry.Open())
{
// do whatever we want with stream
// ...
}
}
}
}
}
Normally you should avoid copying it into another stream - just use it "as is", however, if you absolutely need it in a MemoryStream, you could do:
using(var ms = new MemoryStream())
{
stream.CopyTo(ms);
ms.Position = 0; // rewind
// do something with ms
}
You can use ZipArchiveEntry.Open to get a stream.
This code assumes the zip archive has one text file.
using (FileStream fs = new FileStream(path, FileMode.Open))
using (ZipArchive zip = new ZipArchive(fs) )
{
var entry = zip.Entries.First();
using (StreamReader sr = new StreamReader(entry.Open()))
{
Console.WriteLine(sr.ReadToEnd());
}
}
using (ZipArchive archive = new ZipArchive(webResponse.GetResponseStream()))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
Stream s = entry.Open();
var sr = new StreamReader(s);
var myStr = sr.ReadToEnd();
}
}
Looks like here is what you need:
using (var za = ZipFile.OpenRead(path))
{
foreach (var entry in za.Entries)
{
using (var r = new StreamReader(entry.Open()))
{
//your code here
}
}
}
You can use SharpZipLib among a variety of other libraries to achieve this.
You can use the following code example to unzip to a MemoryStream, as shown on their wiki:
using ICSharpCode.SharpZipLib.Zip;
// Compresses the supplied memory stream, naming it as zipEntryName, into a zip,
// which is returned as a memory stream or a byte array.
//
public MemoryStream CreateToMemoryStream(MemoryStream memStreamIn, string zipEntryName) {
MemoryStream outputMemStream = new MemoryStream();
ZipOutputStream zipStream = new ZipOutputStream(outputMemStream);
zipStream.SetLevel(3); //0-9, 9 being the highest level of compression
ZipEntry newEntry = new ZipEntry(zipEntryName);
newEntry.DateTime = DateTime.Now;
zipStream.PutNextEntry(newEntry);
StreamUtils.Copy(memStreamIn, zipStream, new byte[4096]);
zipStream.CloseEntry();
zipStream.IsStreamOwner = false; // False stops the Close also Closing the underlying stream.
zipStream.Close(); // Must finish the ZipOutputStream before using outputMemStream.
outputMemStream.Position = 0;
return outputMemStream;
// Alternative outputs:
// ToArray is the cleaner and easiest to use correctly with the penalty of duplicating allocated memory.
byte[] byteArrayOut = outputMemStream.ToArray();
// GetBuffer returns a raw buffer raw and so you need to account for the true length yourself.
byte[] byteArrayOut = outputMemStream.GetBuffer();
long len = outputMemStream.Length;
}
Ok so combining all of the above, suppose you want to in a very simple way take a zip file called
"file.zip" and extract it to "C:\temp" folder. (Note: This example was only tested for compress text files) You may need to do some modifications for binary files.
using System.IO;
using System.IO.Compression;
static void Main(string[] args)
{
//Call it like this:
Unzip("file.zip",#"C:\temp");
}
static void Unzip(string sourceZip, string targetPath)
{
using (var z = ZipFile.OpenRead(sourceZip))
{
foreach (var entry in z.Entries)
{
using (var r = new StreamReader(entry.Open()))
{
string uncompressedFile = Path.Combine(targetPath, entry.Name);
File.WriteAllText(uncompressedFile,r.ReadToEnd());
}
}
}
}
I'm working in C#, and I'm downloading for the internet a zip file with one XML file in it. and I wish to load this XML file. This is what I have so far:
byte[] data;
WebClient webClient = new WebClient();
try {
data = webClient.DownloadData(downloadUrl);
}
catch (Exception ex) {
Console.WriteLine("Error in DownloadData (Ex:{0})", ex.Message);
throw;
}
if (data == null) {
Console.WriteLine("Bulk data is null");
throw new Exception("Bulk data is null");
}
//Create the stream
MemoryStream stream = new MemoryStream(data);
XmlDocument document = new XmlDocument();
//Gzip
GZipStream gzipStream = new GZipStream(stream, CompressionMode.Decompress);
//Load report straight from the gzip stream
try {
document.Load(gzipStream);
}
catch (Exception ex) {
Console.WriteLine("Error in Load (Ex:{0})", ex.Message);
throw;
}
in document.Load I'm always getting the following exception:
The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.
What I'm doing wrong?
Apparently SharpZipLib is now unmaintained and you probably want to avoid it:
https://stackoverflow.com/a/593030
In .NET 4.5 there is now built in support for zip files, so for your example it would be:
var data = new WebClient().DownloadData(downloadUrl);
//Create the stream
var stream = new MemoryStream(data);
var document = new XmlDocument();
//zip
var zipArchive = new ZipArchive(stream);
//Load report straight from the zip stream
document.Load(zipArchive.Entries[0].Open());
If you have a byte array that contains a zip archive with a single file, you can use the ZipArchive class to get an unzipped byte array with the file's data.
ZipArchive is contained in .NET 4.5, in the assembly System.IO.Compression.FileSystem (you need to reference it explicitly).
The following function, adapted from this answer, works for me:
public static byte[] UnzipSingleEntry(byte[] zipped)
{
using (var memoryStream = new MemoryStream(zipped))
{
using (var archive = new ZipArchive(memoryStream))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
using (var entryStream = entry.Open())
{
using (var reader = new BinaryReader(entryStream))
{
return reader.ReadBytes((int)entry.Length);
}
}
}
}
}
return null; // To quiet my compiler
}
I am using SharpZipLib and it's working great !
Below is a function that encapsulate the library
public static void Compress(FileInfo sourceFile, string destinationFileName,string destinationTempFileName)
{
Crc32 crc = new Crc32();
string zipFile = Path.Combine(sourceFile.Directory.FullName, destinationTempFileName);
zipFile = Path.ChangeExtension(zipFile, ZIP_EXTENSION);
using (FileStream fs = File.Create(zipFile))
{
using (ZipOutputStream zOut = new ZipOutputStream(fs))
{
zOut.SetLevel(9);
ZipEntry entry = new ZipEntry(ZipEntry.CleanName(destinationFileName));
entry.DateTime = DateTime.Now;
entry.ZipFileIndex = 1;
entry.Size = sourceFile.Length;
using (FileStream sourceStream = sourceFile.OpenRead())
{
crc.Reset();
long len = sourceFile.Length;
byte[] buffer = new byte[bufferSize];
while (len > 0)
{
int readSoFar = sourceStream.Read(buffer, 0, buffer.Length);
crc.Update(buffer, 0, readSoFar);
len -= readSoFar;
}
entry.Crc = crc.Value;
zOut.PutNextEntry(entry);
len = sourceStream.Length;
sourceStream.Seek(0, SeekOrigin.Begin);
while (len > 0)
{
int readSoFar = sourceStream.Read(buffer, 0, buffer.Length);
zOut.Write(buffer, 0, readSoFar);
len -= readSoFar;
}
}
zOut.Finish();
zOut.Close();
}
fs.Close();
}
}
As the others have mentioned GZip and Zip are not the same so you might need to use a zip library. I use a library called: DotNetZip - available from the below site:
http://dotnetzip.codeplex.com/
From GZipStream Class description:
Compressed GZipStream objects written to a file with an extension of .gz can be decompressed using many common compression tools; however, this class does not inherently provide functionality for adding files to or extracting files from .zip archives
So unless you control server-side files, I'd suggest looking for specific zip-targeted library (SharpZipLib for example).