Develop Reference

GZipStream sum compressing blocks to file

I make an archiver with block-by-block reading and file compression. I put the compressed block in FileStream.
I am reading the 5 mb block. The problem is that if I compress a pic of 8 mb, then when I pull it out of the resulting archive, its sum-hash does not match the original and it opens pic halfway, and the size is the same... I don’t know what to try. I ask for help.
Read chunk void:
private byte[] ReadChunk(int chunkId)
{
using (var inFile = new FileStream(sourceFile, FileMode.Open, FileAccess.Read, FileShare.Read))
{
long filePosition = chunkId * chunkDataSize;
int bytesRead;
if (inFile.Length - filePosition <= chunkDataSize)
{
bytesRead = (int)(inFile.Length - filePosition);
}
else
{
bytesRead = chunkDataSize;
}
var lastBuffer = new byte[bytesRead];
inFile.Read(lastBuffer, 0, bytesRead);
return lastBuffer;
}
}
Compress and write void:
private void CompressBlock(byte[] bytesTo)
{
using (MemoryStream ms = new MemoryStream())
{
using (GZipStream gs = new GZipStream(ms, CompressionMode.Compress))
{
gs.Write(bytesTo, 0, bytesTo.Length);
}
byte[] compressedData = ms.ToArray();
using (var outFile = new FileStream(resultFile, FileMode.Append))
{
BitConverter.GetBytes(compressedData.Length).CopyTo(compressedData, 4);
outFile.Write(compressedData, 0, compressedData.Length);
}
}
}

Correct way to use GZipStream in dotNET C#

I'm working with GZipStream at the moment using .net 3.5.
I have two methods listed below. As input file I use text file which consists of chars 's'. Size of the file is 2MB. This code works fine if I use .net 4.5 but with .net 3.5 after compress and decompress I get file of size 435KB which of course isn't the same with source file.
If I try to decompress file via WinRAR it is also looks good (the same with source file).
If I try decompress file using GZipStream from .net4.5 (file compressed via GZipStream from .net 3.5) the result is bad.
UPD:
In general I really need to read the file as several separate gzip chunks, in this case all the bytes of copressed files are read at one call of the Read() method so I still don't understand why decompressing doesn't works.
public void CompressFile()
{
string fileIn = #"D:\sin2.txt";
string fileOut = #"D:\sin2.txt.pgz";
using (var fout = File.Create(fileOut))
{
using (var fin = File.OpenRead(fileIn))
{
using (var zip = new GZipStream(fout, CompressionMode.Compress))
{
var buffer = new byte[1024 * 1024 * 10];
int n = fin.Read(buffer, 0, buffer.Length);
zip.Write(buffer, 0, n);
}
}
}
}
public void DecompressFile()
{
string fileIn = #"D:\sin2.txt.pgz";
string fileOut = #"D:\sin2.1.txt";
using (var fsout = File.Create(fileOut))
{
using (var fsIn = File.OpenRead(fileIn))
{
var buffer = new byte[1024 * 1024 * 10];
int n;
while ((n = fsIn.Read(buffer, 0, buffer.Length)) > 0)
{
using (var ms = new MemoryStream(buffer, 0, n))
{
using (var zip = new GZipStream(ms, CompressionMode.Decompress))
{
int nRead = zip.Read(buffer, 0, buffer.Length);
fsout.Write(buffer, 0, nRead);
}
}
}
}
}
}

You're trying to decompress each "chunk" as if it's a separate gzip file. Don't do that - just read from the GZipStream in a loop:
using (var fsout = File.Create(fileOut))
{
using (var fsIn = File.OpenRead(fileIn))
{
using (var zip = new GZipStream(fsIn, CompressionMode.Decompress))
{
var buffer = new byte[1024 * 32];
int bytesRead;
while ((bytesRead = zip.Read(buffer, 0, buffer.Length)) > 0)
{
fsout.Write(buffer, 0, bytesRead);
}
}
}
}
Note that your compression code should look similar, reading in a loop rather than assuming a single call to Read will read all the data.
(Personally I'd skip fsIn, and just use new GZipStream(File.OpenRead(fileIn)) but that's just a personal preference.)

First, as #Jon Skeet mentioned, you are not using Stream.Read method correctly. It doesn't matter if your buffer is big enough or not, the stream is allowed to return less bytes than requested, with zero indicating no more, so reading from stream should always be performed in a loop.
However the main problem in your decompress code is the way you share the buffer. Your read the input into a buffer, than wrap it in a MemoryStream (note that the constructor used does not make a copy of the passed array, but actually sets it as it's internal buffer), and then you try to read and write to that buffer at the same time. Taking into account that decompressing writes data "faster" than reading, it's surprising that your code works at all.
The correct implementation is quite simple
static void CompressFile()
{
string fileIn = #"D:\sin2.txt";
string fileOut = #"D:\sin2.txt.pgz";
using (var input = File.OpenRead(fileIn))
using (var output = new GZipStream(File.Create(fileOut), CompressionMode.Compress))
Write(input, output);
}
static void DecompressFile()
{
string fileIn = #"D:\sin2.txt.pgz";
string fileOut = #"D:\sin2.1.txt";
using (var input = new GZipStream(File.OpenRead(fileIn), CompressionMode.Decompress))
using (var output = File.Create(fileOut))
Write(input, output);
}
static void Write(Stream input, Stream output, int bufferSize = 10 * 1024 * 1024)
{
var buffer = new byte[bufferSize];
for (int readCount; (readCount = input.Read(buffer, 0, buffer.Length)) > 0;)
output.Write(buffer, 0, readCount);
}

out of memory exception in compress large files using io.compression

i am trying to not exceeded memory max size so i have to check every time if It greater than Max Memory Size Then i flush it into zip file Stream . The Problem Here it replace memory stream with existence one in file stream ,Or Is there Any way To Do the Same Rquired with Another Way ( But With Out Using Any DLL Lib)
MemoryStream memoryStream = new MemoryStream();
FileStream fileStream = new FileStream(sbZipFolderName.ToString(),FileMode.Create);
foreach (FileInfo flInfo in ListfileFolderPaths)
{
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Update, true))
archive.CreateEntryFromFile(flInfo.FullName, slastFolderName + "/" + flInfo.DirectoryName.Replace(new DirectoryInfo(sFolderPath.ToString()).FullName, "") + "/" + flInfo.Name);
if (memoryStream.Length > MaxSize)
{
using (fileStream = new FileStream(sFolderPath + "/" + slastFolderName + ".zip", FileMode.Create))
{
memoryStream.Seek(0, SeekOrigin.Begin);
memoryStream.CopyTo(fileStream);
memoryStream = new MemoryStream();
}
}
}
if ((memoryStream != null) && (memoryStream.Length > 0))
memoryStream.CopyTo(fileStream);

You can use theGzip archive to compress a file.
This is the compression:
public static byte[] Compress(byte[] raw)
{
using (MemoryStream memory = new MemoryStream())
{
using (GZipStream gzip = new GZipStream(memory,
CompressionMode.Compress, true))
{
gzip.Write(raw, 0, raw.Length);
}
return memory.ToArray();
}
}
}
And this to decompression :
static byte[] Decompress(byte[] gzip)
{
// Create a GZIP stream with decompression mode.
// ... Then create a buffer and write into while reading from the GZIP stream.
using (GZipStream stream = new GZipStream(new MemoryStream(gzip), CompressionMode.Decompress))
{
const int size = 4096;
byte[] buffer = new byte[size];
using (MemoryStream memory = new MemoryStream())
{
int count = 0;
do
{
count = stream.Read(buffer, 0, size);
if (count > 0)
{
memory.Write(buffer, 0, count);
}
}
while (count > 0);
return memory.ToArray();
}
}
}
}
Tell me if it worked.
Goodluck.

Decompress string in java from compressed string in C#

I was searching for the correct solution to decompress the string in java coming from c# code.I tried myself with lot of techniques in java like(gzip,inflatter etc.).but didn't get the solution.i got some error while trying to decompress the string in java from compressed string from c# code.
My C# code to compress the string is,
public static string CompressString(string text)
{
byte[] byteArray = Encoding.GetEncoding(1252).GetBytes(text);// Encoding.ASCII.GetBytes(text);
using (var ms = new MemoryStream())
{
// Compress the text
using (var ds = new DeflateStream(ms, CompressionMode.Compress))
{
ds.Write(byteArray, 0, byteArray.Length);
}
return Convert.ToBase64String(ms.ToArray());
}
}
And decompress the string in java using,
private static void compressAndDecompress(){
try {
// Encode a String into bytes
String string = "xxxxxxSAMPLECOMPRESSEDSTRINGxxxxxxxxxx";
// // Compress the bytes
byte[] decoded = Base64.decodeBase64(string.getBytes());
byte[] output = new byte[4096];
// Decompress the bytes
Inflater decompresser = new Inflater();
decompresser.setInput(decoded);
int resultLength = decompresser.inflate(output);
decompresser.end();
// Decode the bytes into a String
String outputString = new String(output, 0, resultLength, "UTF-8");
System.out.println(outputString);
} catch(java.io.UnsupportedEncodingException ex) {
ex.printStackTrace();
} catch (java.util.zip.DataFormatException ex) {
ex.printStackTrace();
}
}
I get this exception when running the above code:
java.util.zip.DataFormatException: incorrect header check
Kindly give me the sample code in java to decompress the string java.Thanks

My C# code to compress is
private string Compress(string text)
{
byte[] buffer = Encoding.UTF8.GetBytes(text);
MemoryStream ms = new MemoryStream();
using (GZipStream zip = new GZipStream(ms, CompressionMode.Compress, true))
{
zip.Write(buffer, 0, buffer.Length);
}
ms.Position = 0;
MemoryStream outStream = new MemoryStream();
byte[] compressed = new byte[ms.Length];
ms.Read(compressed, 0, compressed.Length);
byte[] gzBuffer = new byte[compressed.Length + 4];
System.Buffer.BlockCopy(compressed, 0, gzBuffer, 4, compressed.Length);
System.Buffer.BlockCopy(BitConverter.GetBytes(buffer.Length), 0, gzBuffer, 0, 4);
return Convert.ToBase64String(gzBuffer);
}
Java code to decompress the text is
private String Decompress(String compressedText)
{
byte[] compressed = compressedText.getBytes("UTF8");
compressed = org.apache.commons.codec.binary.Base64.decodeBase64(compressed);
byte[] buffer=new byte[compressed.length-4];
buffer = copyForDecompression(compressed,buffer, 4, 0);
final int BUFFER_SIZE = 32;
ByteArrayInputStream is = new ByteArrayInputStream(buffer);
GZIPInputStream gis = new GZIPInputStream(is, BUFFER_SIZE);
StringBuilder string = new StringBuilder();
byte[] data = new byte[BUFFER_SIZE];
int bytesRead;
while ((bytesRead = gis.read(data)) != -1)
{
string.append(new String(data, 0, bytesRead));
}
gis.close();
is.close();
return string.toString();
}
private byte[] copyForDecompression(byte[] b1,byte[] b2,int srcoffset,int dstoffset)
{
for(int i=0;i<b2.length && i<b1.length;i++)
{
b2[i]=b1[i+4];
}
return b2;
}
This code works perfectly fine for me.

Had exactly the same issue. Could solve it via
byte[] compressed = Base64Utils.decodeFromString("mybase64encodedandwithc#zippedcrap");
Inflater decompresser = new Inflater(true);
decompresser.setInput(compressed);
byte[] result = new byte[4096];
decompresser.inflate(result);
decompresser.end();
System.out.printf(new String(result));
The magic happens with the boolen parameter on instantiating the Inflator
BW Hubert

For beloved googlers,
As #dbw mentioned,
according to post How to decompress stream deflated with java.util.zip.Deflater in .NET?,
java.util.zip.deflater equivalent in c# the default deflater used in C#
is not having any java equivalent that's why users prefer Gzip, Ziplib
or some other zip techniques.
a relatively simple method would be using GZip.
And for the accepted answer, one problem is that in this method you should append the data size to the compressed string yourself, and more importantly as per my own experience in our production app, It is buggy when the string reaches ~2000 chars!
the bug is in the System.io.Compression.GZipStream
any way using SharpZipLib in c# the problem goes away and everything would be as simple as following snippets:
JAVA:
import android.util.Base64;
import com.google.android.gms.common.util.IOUtils;
import org.jetbrains.annotations.Nullable;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
public class CompressionHelper {
#Nullable
public static String compress(#Nullable String data) {
if(data == null || data.length() == 0)
return null;
try {
// Create an output stream, and a gzip stream to wrap over.
ByteArrayOutputStream bos = new ByteArrayOutputStream(data.length());
GZIPOutputStream gzip = new GZIPOutputStream(bos);
// Compress the input string
gzip.write(data.getBytes());
gzip.close();
byte[] compressed;
// Convert to base64
compressed = Base64.encode(bos.toByteArray(),Base64.NO_WRAP);
bos.close();
// return the newly created string
return new String(compressed);
} catch(IOException e) {
return null;
}
}
#Nullable
public static String decompress(#Nullable String compressedText) {
if(compressedText == null || compressedText.length() == 0)
return null;
try {
// get the bytes for the compressed string
byte[] compressed = compressedText.getBytes("UTF-8");
// convert the bytes from base64 to normal string
compressed = Base64.decode(compressed, Base64.NO_WRAP);
ByteArrayInputStream bis = new ByteArrayInputStream(compressed);
GZIPInputStream gis = new GZIPInputStream(bis);
byte[] bytes = IOUtils.toByteArray(gis);
return new String(bytes, "UTF-8");
}catch (IOException e){
e.printStackTrace();
}
return null;
}
}
and c#:
using ICSharpCode.SharpZipLib.GZip; //PM> Install-Package SharpZipLib
using System;
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace GeneralTools
{
public static class CompressionTools
{
public static string CompressString(string text)
{
if (string.IsNullOrEmpty(text))
return null;
byte[] buffer = Encoding.UTF8.GetBytes(text);
using (var compressedStream = new MemoryStream())
{
GZip.Compress(new MemoryStream(buffer), compressedStream, false);
byte[] compressedData = compressedStream.ToArray();
return Convert.ToBase64String(compressedData);
}
}
public static string DecompressString(string compressedText)
{
if (string.IsNullOrEmpty(compressedText))
return null;
byte[] gZipBuffer = Convert.FromBase64String(compressedText);
using (var memoryStream = new MemoryStream())
{
using (var compressedStream = new MemoryStream(gZipBuffer))
{
var decompressedStream = new MemoryStream();
GZip.Decompress(compressedStream, decompressedStream, false);
return Encoding.UTF8.GetString(decompressedStream.ToArray()).Trim();
}
}
}
}
}
you may also find the codes here

If anyone still interested, here's my full solution with outputstream to handle unknown string size. Using C# DeflateStream and Java Inflater (based on Hubert Ströbitzer answer).
C# Compression:
string CompressString(string raw)
{
byte[] uncompressedData = Encoding.UTF8.GetBytes(raw);
MemoryStream output = new MemoryStream();
using (DeflateStream dStream = new DeflateStream(output, CompressionLevel.Optimal))
{
dStream.Write(uncompressedData, 0, uncompressedData.Length);
}
string compressedString = Convert.ToBase64String(output.ToArray());
return compressedString;
}
Java decompress:
String decompressString(String compressedString) {
byte[] compressed = Base64Utils.decodeFromString(compressedString);
Inflater inflater = new Inflater(true);
inflater.setInput(compressed);
//Using output stream to handle unknown size of decompressed string
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
try {
while(!inflater.finished()){
int count = inflater.inflate(buffer);
outputStream.write(buffer, 0, count);
}
inflater.end();
outputStream.close();
} catch (DataFormatException e) {
//Handle DataFormatException
} catch (IOException e) {
//Handle IOException
}
return outputStream.toString();
}

Programmatic compression/decompression to MemoryStream with GZipStream

I built (based on a CodeProject article) a wrapper class (C#) to use a GZipStream to compress a MemoryStream. It compresses fine but doesn't decompress. I've looked at many other examples that have the same problem, and I feel like I'm following what's said but still am getting nothing when I decompress. Here's the compression and decompression methods:
public static byte[] Compress(byte[] bSource)
{
using (MemoryStream ms = new MemoryStream())
{
using (GZipStream gzip = new GZipStream(ms, CompressionMode.Compress, true))
{
gzip.Write(bSource, 0, bSource.Length);
gzip.Close();
}
return ms.ToArray();
}
}
public static byte[] Decompress(byte[] bSource)
{
try
{
using (MemoryStream ms = new MemoryStream())
{
using (GZipStream gzip = new GZipStream(ms, CompressionMode.Decompress, true))
{
gzip.Read(bSource, 0, bSource.Length);
gzip.Close();
}
return ms.ToArray();
}
}
catch (Exception ex)
{
throw new Exception("Error decompressing byte array", ex);
}
}
Here's an example of how I use it:
string sCompressed = Convert.ToBase64String(CompressionHelper.Compress("Some Text"));
// Other Processes
byte[] bReturned = CompressionHelper.Decompress(Convert.FromBase64String(sCompressed));
// bReturned has no elements after this line is executed

There is a bug in Decompress method.
The code does not read content of bSource. On the contrary, it overrides its content wile reading from empty gzip, created based on empty memory stream.
Basically what your version of code is doing:
//create empty memory
using (MemoryStream ms = new MemoryStream())
//create gzip stream over empty memory stream
using (GZipStream gzip = new GZipStream(ms, CompressionMode.Compress, true))
// write from empty stream to bSource
gzip.Write(bSource, 0, bSource.Length);
The fix could look like this:
public static byte[] Decompress(byte[] bSource)
{
using (var inStream = new MemoryStream(bSource))
using (var gzip = new GZipStream(inStream, CompressionMode.Decompress))
using (var outStream = new MemoryStream())
{
gzip.CopyTo(outStream);
return outStream.ToArray();
}
}

The OP said in an edit, now rolled back:
Thanks to Alex's explanation of what was going wrong, I was able to fix the Decompress method. Unfortunately, I'm using .Net 3.5, so I wasn't able to implement the Stream.CopyTo method he suggested. With his explanation, though, I was able to figure out a solution. I made the appropriate changes to the Decompress method below.
public static byte[] Decompress(byte[] bSource)
{
try
{
using (var instream = new MemoryStream(bSource))
{
using (var gzip = new GZipStream(instream, CompressionMode.Decompress))
{
using (var outstream = new MemoryStream())
{
byte[] buffer = new byte[4096];
while (true)
{
int delta = gzip.Read(buffer, 0, buffer.Length);
if (delta > 0)
outstream.Write(buffer, 0, delta);
if (delta < 4096)
break;
}
return outstream.ToArray();
}
}
}
}
catch (Exception ex)
{
throw new Exception("Error decompressing byte array", ex);
}
}