Why GZipStream, based on MemoryStream reads from it only 24576bytes and cant read any further? I compress my gz with multithreading, in parts, but winrar could decompress it and GZipStream not.
static private bool ZipperWork2(string InputPath, string OutputPath, CompressionMode CompMode)
{
Mode = CompMode;
RawData = new byte[ThreadCount][];
CompressedData = new byte[ThreadCount][];
Thread[] CompressorThreads = new Thread[ThreadCount];
byte[] buffer = new byte[MAX_BLOCK_SIZE];
using (FileStream fs = new FileStream(InputPath, FileMode.Open))
{
using (FileStream fw = new FileStream(OutputPath, FileMode.Append))
{
while (fs.Position < fs.Length)
{
for (int i = 0; i < ThreadCount; i++)
{
int count = fs.Read(buffer, 0, MAX_BLOCK_SIZE);
RawData[i] = new byte[count];
Array.Copy(buffer, RawData[i], count);
CompressorThreads[i] = new Thread(WorkWithBlock2);
CompressorThreads[i].Start(i);
}
for (int i = 0; i < ThreadCount; i++)
{
CompressorThreads[i].Join();
fw.Write(CompressedData[i], 0, CompressedData[i].Length);
}
}
}
}
return true;
}
private static void WorkWithBlock2(object index)
{
int DataIndex = (int)index;
//int count=0,totalcount=0;
using (MemoryStream ms = new MemoryStream(RawData[DataIndex]))
{
using (MemoryStream output = new MemoryStream())
{
using (GZipStream gz = new GZipStream(ms, Mode))
{
gz.CopyTo(output);
}
CompressedData[DataIndex] = output.ToArray();
}
}
}
I trying to do it in a different ways, but result are always the same, decompress only 4KB (and it don't relate with my MAX_BLOCK_SIZE variable)
Your gzip stream may actually consist of multiple concatenated gzip streams. This is permitted by the gzip format. GZipStream is likely just reading the first one. All you'd need to do is repeat the operation on the subsequent input bytes until all of the input bytes have been consumed.
Related
I make an archiver with block-by-block reading and file compression. I put the compressed block in FileStream.
I am reading the 5 mb block. The problem is that if I compress a pic of 8 mb, then when I pull it out of the resulting archive, its sum-hash does not match the original and it opens pic halfway, and the size is the same... I don’t know what to try. I ask for help.
Read chunk void:
private byte[] ReadChunk(int chunkId)
{
using (var inFile = new FileStream(sourceFile, FileMode.Open, FileAccess.Read, FileShare.Read))
{
long filePosition = chunkId * chunkDataSize;
int bytesRead;
if (inFile.Length - filePosition <= chunkDataSize)
{
bytesRead = (int)(inFile.Length - filePosition);
}
else
{
bytesRead = chunkDataSize;
}
var lastBuffer = new byte[bytesRead];
inFile.Read(lastBuffer, 0, bytesRead);
return lastBuffer;
}
}
Compress and write void:
private void CompressBlock(byte[] bytesTo)
{
using (MemoryStream ms = new MemoryStream())
{
using (GZipStream gs = new GZipStream(ms, CompressionMode.Compress))
{
gs.Write(bytesTo, 0, bytesTo.Length);
}
byte[] compressedData = ms.ToArray();
using (var outFile = new FileStream(resultFile, FileMode.Append))
{
BitConverter.GetBytes(compressedData.Length).CopyTo(compressedData, 4);
outFile.Write(compressedData, 0, compressedData.Length);
}
}
}
I've got some strange problem with GZip Serializer.
Trying serializing object with data in it.
Following code give results(at POINT1 in debug): ms.Length = 100028 and uncompressedStream.Length=100027
After POINT1 there is exception "End of Stream encountered before parsing was completed.", which i think is result of this lost byte.
I am using .net 4.0.
//generating data
int length = 100000;
byte[] data = new byte[length];
for (int i = 0; i < length; i++)
{
data[i] = System.Convert.ToByte(i % 100 + i % 50);
}
//serialization into memory stream
IFormatter formatter = new BinaryFormatter();
var ms = new MemoryStream();
formatter.Serialize(ms, data);
ms.Seek(0, SeekOrigin.Begin);
//GZip zip
MemoryStream compressedStream = new MemoryStream();
var Compress = new GZipStream(compressedStream, CompressionMode.Compress);
ms.CopyTo(Compress);
compressedStream.Seek(0, SeekOrigin.Begin);
//GZip Unzip
MemoryStream uncompressedStream = new MemoryStream();
var Decompress = new GZipStream(compressedStream, CompressionMode.Decompress);
Decompress.CopyTo(uncompressedStream);
uncompressedStream.Seek(0, SeekOrigin.Begin);
//deserialization from memory stream
//POINT1
var oo = formatter.Deserialize(uncompressedStream);
var o = (byte[])oo;
//checking
Assert.AreEqual(data.Length, o.Length);
for (int i = 0; i < data.Length; i++)
Assert.AreEqual(data[i], o[i]);
Compression streams don't flush (and can't properly flush) until they are closed. You will need to close the GZipStream. Telling it not to close the underlying stream (one of the constructor arguments) will make this easier.
//generating data
int length = 100000;
byte[] data = new byte[length];
for (int i = 0; i < length; i++)
{
data[i] = System.Convert.ToByte(i % 100 + i % 50);
}
byte[] o;
//serialization into memory stream
IFormatter formatter = new BinaryFormatter();
using (var ms = new MemoryStream())
{
formatter.Serialize(ms, data);
ms.Seek(0, SeekOrigin.Begin);
//GZip zip
using(MemoryStream compressedStream = new MemoryStream())
{
using (var Compress = new GZipStream(compressedStream, CompressionMode.Compress, true))
{
ms.CopyTo(Compress);
}
compressedStream.Seek(0, SeekOrigin.Begin);
//GZip Unzip
using (MemoryStream uncompressedStream = new MemoryStream())
{
using (var Decompress = new GZipStream(compressedStream, CompressionMode.Decompress, true))
{
Decompress.CopyTo(uncompressedStream);
}
uncompressedStream.Seek(0, SeekOrigin.Begin);
var oo = formatter.Deserialize(uncompressedStream);
o = (byte[])oo;
}
}
//deserialization from memory stream
//POINT1
}
//checking
Debug.Assert(data.Length == o.Length);
for (int i = 0; i < data.Length; i++)
Debug.Assert(data[i] == o[i]);
i am trying to not exceeded memory max size so i have to check every time if It greater than Max Memory Size Then i flush it into zip file Stream . The Problem Here it replace memory stream with existence one in file stream ,Or Is there Any way To Do the Same Rquired with Another Way ( But With Out Using Any DLL Lib)
MemoryStream memoryStream = new MemoryStream();
FileStream fileStream = new FileStream(sbZipFolderName.ToString(),FileMode.Create);
foreach (FileInfo flInfo in ListfileFolderPaths)
{
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Update, true))
archive.CreateEntryFromFile(flInfo.FullName, slastFolderName + "/" + flInfo.DirectoryName.Replace(new DirectoryInfo(sFolderPath.ToString()).FullName, "") + "/" + flInfo.Name);
if (memoryStream.Length > MaxSize)
{
using (fileStream = new FileStream(sFolderPath + "/" + slastFolderName + ".zip", FileMode.Create))
{
memoryStream.Seek(0, SeekOrigin.Begin);
memoryStream.CopyTo(fileStream);
memoryStream = new MemoryStream();
}
}
}
if ((memoryStream != null) && (memoryStream.Length > 0))
memoryStream.CopyTo(fileStream);
You can use theGzip archive to compress a file.
This is the compression:
public static byte[] Compress(byte[] raw)
{
using (MemoryStream memory = new MemoryStream())
{
using (GZipStream gzip = new GZipStream(memory,
CompressionMode.Compress, true))
{
gzip.Write(raw, 0, raw.Length);
}
return memory.ToArray();
}
}
}
And this to decompression :
static byte[] Decompress(byte[] gzip)
{
// Create a GZIP stream with decompression mode.
// ... Then create a buffer and write into while reading from the GZIP stream.
using (GZipStream stream = new GZipStream(new MemoryStream(gzip), CompressionMode.Decompress))
{
const int size = 4096;
byte[] buffer = new byte[size];
using (MemoryStream memory = new MemoryStream())
{
int count = 0;
do
{
count = stream.Read(buffer, 0, size);
if (count > 0)
{
memory.Write(buffer, 0, count);
}
}
while (count > 0);
return memory.ToArray();
}
}
}
}
Tell me if it worked.
Goodluck.
I have a function that creates a zip file a string array of files passed. The function does succeed in creating the zip file and the zip entry files inside it, but these zip entry files are empty. I've tried a couple of different methods - the function code below is the closest I've gotten to something working:
public static bool ZipFile(string[] arrFiles, string sZipToDirectory, string sZipFileName)
{
if (Directory.Exists(sZipToDirectory))
{
FileStream fNewZipFileStream;
ZipOutputStream zos;
try {
fNewZipFileStream = File.Create(sZipToDirectory + sZipFileName);
zos = new ZipOutputStream(fNewZipFileStream);
for (int i = 0; i < arrFiles.Length; i++) {
ZipEntry entry = new ZipEntry(arrFiles[i].Substring(arrFiles[i].LastIndexOf("/") + 1));
zos.PutNextEntry(entry);
FileStream fStream = File.OpenRead(arrFiles[i]);
BufferedStream bfStrm = new BufferedStream(fStream);
byte[] buffer = new byte[bfStrm.Length];
int count;
while ((count = bfStrm.Read(buffer, 0, 1024)) != -1) {
zos.Write(buffer);
}
bfStrm.Close();
fStream.Close();
zos.CloseEntry();
}
zos.Close();
fNewZipFileStream.Close();
return true;
}
catch (Exception ex)
{
string sErr = ex.Message;
return false;
}
finally
{
fNewZipFileStream = null;
zos = null;
}
}
else
{
return false;
}
}
I think it's got to do with the byte stream handling. I've tried this bit of code that handles the stream but it goes into an infinite loop:
while ((count = fStream.Read(buffer, 0, 1024)) != -1) {
zos.Write(buffer, 0, count);
}
fStream.Close();
I found a solution that is quite simple - I used the ReadAllBytes method of the static File class.
ZipEntry entry = new ZipEntry(arrFiles[i].Substring(arrFiles[i].LastIndexOf("/") + 1));
zos.PutNextEntry(entry);
byte[] fileContents = File.ReadAllBytes(arrFiles[i]);
zos.Write(fileContents);
zos.CloseEntry();
Using Read() on a FileStream returns the amount of bytes read into the stream or 0 if the end of the stream has been reached. It will never return a value of -1.
From MSDN:
The total number of bytes read into the buffer. This might be less than the number of bytes requested if that number of bytes are not currently available, orzero if the end of the stream is reached.
I'd modify your code to the following:
System.IO.FileStream fos = new System.IO.FileStream(sZipToDirectory + sZipFileName, FileMode.Create);
Java.Util.Zip.ZipOutputStream zos = new Java.Util.Zip.ZipOutputStream(fos);
byte[] buffer = new byte[1024];
for (int i = 0; i < arrFiles.Length; i++) {
FileInfo fi = new FileInfo (arrFiles[i]);
Java.IO.FileInputStream fis = new Java.IO.FileInputStream(fi.FullName);
ZipEntry entry = new ZipEntry(arrFiles[i].Substring(arrFiles[i].LastIndexOf("/") + 1));
zos.PutNextEntry(entry);
int count = 0;
while ((count = fis.Read(buffer)) > 0) {
zos.Write(buffer, 0, count);
}
fis.Close();
zos.CloseEntry();
}
This is nearly identical to the code I've used for creating zip archives on Android in the past.
Are you allowed to use SharpZip? It's really easy to use.
Here is a blog post I wrote to extract zip files
private static void upzip(string url)
{
WebClient wc = new WebClient();
wc.DownloadFile(url, "temp.zip");
//unzip
ZipFile zf = null;
try
{
zf = new ZipFile(File.OpenRead("temp.zip"));
foreach (ZipEntry zipEntry in zf)
{
string fileName = zipEntry.Name;
byte[] buffer = new byte[4096];
Stream zipStream = zf.GetInputStream(zipEntry);
using (FileStream streamWriter = File.Create( fileName))
{
StreamUtils.Copy(zipStream, streamWriter, buffer);
}
}
}
finally
{
if (zf != null)
{
zf.IsStreamOwner = true;
zf.Close();
}
}
}
private void ZipFolder(string[] _files, string zipFileName)
{
using var memoryStream = new MemoryStream();
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
foreach (var item in _files)
{
var demoFile = archive.CreateEntry(Path.GetFileName(item));
using var readStreamW = File.OpenRead(item);
using (var entryStream = demoFile.Open())
{
using (var streamWriter = new StreamWriter(entryStream))
{
readStreamW.Seek(0, SeekOrigin.Begin);
readStreamW.CopyTo(streamWriter.BaseStream);
}
}
}
}
using var fileStream = new FileStream(zipFileName, FileMode.Create);
memoryStream.Seek(0, SeekOrigin.Begin);
memoryStream.CopyTo(fileStream);
}
i am using: `private void get_stocks_data()
{
byte[] result;
byte[] buffer = new byte[4096];
WebRequest wr = WebRequest.Create("http://www.tase.co.il/TASE/Pages/ExcelExport.aspx?sn=he-IL_ds&enumTblType=AllSecurities&Columns=he-IL_Columns&Titles=he-IL_Titles&TblId=0&ExportType=1");
using (WebResponse response = wr.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (MemoryStream memoryStream = new MemoryStream())
{
int count = 0;
do
{
count = responseStream.Read(buffer, 0, buffer.Length);
memoryStream.Write(buffer, 0, count);
} while (count != 0);
result = memoryStream.ToArray();
write_data_to_excel(result);
}
}
}`
to download the excel file,
And this method to fill the file on my computer:
private void write_data_to_excel(byte[] input)
{
StreamWriter str = new StreamWriter("stockdata.xls");
for (int i = 0; input.Length > i; i++)
{
str.WriteLine(input[i].ToString());
}
str.Close();
}
The result is that i get a lot of numbers...
What am i doing wrong? the file i am downloadin is excel version 2003, on my computer i have 2007...
Thanks.
I would suggest that you use WebClient.DownloadFile() instead.
This is a higher level method that will abstract from creating the request manually, dealing with encoding, etc.
Problem is in your Write_data_to_excel function
as you are using StreamWriter.WriteLine method it needs string,
you are passing byte as string so your binary value say 10 will be now string 10
try FileStream f = File.OpenWrite("stockdata.xlsx");
f.Write(input,0,input.Length); this will work.