I've made a simple proxy server. Works fine, but not the gzip decompressing:
The magic number is not correct. Be sure .
No error, but the gzipstream won't decompress.
It works as follows:
private void HandleTraffic()
{
while ((bytesRead = realServer.Receive(buffer, 0, BODYBUFFERSIZE, SocketFlags.None)) > 0)
{
decompressedBuffer = new byte[BODYBUFFERSIZE];
if (t == "gzip")
{
MemoryStream outputStream = new MemoryStream(), inputStream = new MemoryStream();
inputStream.Write(buffer, 0, bytesRead);
inputStream.Position = 0;
inputStream.Flush();
using (GZipStream gz = new GZipStream(inputStream, CompressionMode.Decompress))
{
gz.CopyTo(outputStream);
}
decompressedBuffer = outputStream.ToArray();
MessageBox.Show(ASCIIEncoding.ASCII.GetString(decompressedBuffer));
}
}
}
InputStream is filled, but outputStream isn't decompressed or gives me the error 'The magic number is not correct'.
In this job, sockets has been required.
You don't need to process the data in a proxy server in any way whatsoever, unzipping, deflating, whatsoever. Just copy the bytes. The only data you need to even look at is the first line from the client, the HTTP CONNECT command.
The solution to this for myself was to turn off Fiddler.
Related
I'm trying to read the bytes from a file that I've retrieved from an external source, however when running the code I get the following error:
System.NotSupportedException: Specified method is not supported.\r\n at System.Net.Http.HttpBaseStream.get_Length()
My code in question is as follows:
var responseBody = (request.GetResponseAsync().Result).GetResponseStream();
byte[] file;
var br = new BinaryReader(responseBody);
file = br.ReadBytes((int)br.BaseStream.Length);
using (MemoryStream ms = new MemoryStream())
{
int read;
while ((read = responseBody.Read(file, 0, file.Length)) > 0)
{
ms.Write(file, 0, read);
}
}
var result = new MemoryStream(file);
It fails on the following line:
file = br.ReadBytes((int)br.BaseStream.Length);
I can't seem to find a solution to the issue, can anyone suggest a fix?
The stream you feed into the binary reader is not seekable. You can check the CanSeek property to see this.
For example, when chunked encoding is used by the server to return the response, there is no way to know the file size beforehand.
Try this. You are already reading from the response stream, and not from the BinaryReader.
Note: You can also use CopyTo()
var responseBody = (request.GetResponseAsync().Result).GetResponseStream();
byte[] tempBuffer = new byte[8192];
MemoryStream ms = new MemoryStream();
int read;
while ((read = responseBody.Read(tempBuffer, 0, tempBuffer.Length)) > 0)
{
ms.Write(tempBuffer, 0, read);
}
var result = ms;
You can also get the response size from the response header:
response.Headers["Content-Length"]
i'm trying to read .gz files using binary reader by first unzipping with gzipstream, and then creating a new binary reader with the gzipstream. however, when i try to use the BaseStream.Position and BaseStream.Length of BinaryReader (to know when i'm at the end of my file), i get a NotSupportedException, checking the doc for these fields in GZipStream Class shows:
Length
This property is not supported and always throws a NotSupportedException.(Overrides Stream.Length.)
Position
This property is not supported and always throws a NotSupportedException.(Overrides Stream.Position.)
so my question is how can i know when i'm at the end of my file when reading a decompressed GZipStream using BinaryReader? thanks
here is my code:
Stream stream = new MemoryStream(textAsset.bytes);
GZipStream zippedStream = new GZipStream(stream, CompressionMode.Decompress);
using (BinaryReader reader = new BinaryReader(zippedStream))
while(reader.BaseStream.Position != reader.BaseStream.Length)
{
//do stuff with BinaryReader
}
the above throws:
NotSupportedException: Operation is not supported. System.IO.Compression.DeflateStream.get_Position()
due to the BaseStream.Position call in the while()
You can copy your zippedStream to MemoryStream instance, that can be read fully using ToArray function. That is the easiest solution I can think of.
Stream stream = new MemoryStream(textAsset.bytes);
byte[] result;
using (GZipStream zippedStream = new GZipStream(stream, CompressionMode.Decompress))
{
using (MemoryStream reader = new MemoryStream())
{
zippedStream.CopyTo(reader);
result = reader.ToArray();
}
}
Alternatively if you want to read stream in chunks
using (GZipStream zippedStream = new GZipStream(stream, CompressionMode.Decompress))
{
byte[] buffer = new byte[16 * 1024];
int read;
while ((read = zippedStream.Read(buffer, 0, buffer.Length)) > 0)
{
// do work
}
}
Depending on what you are decoding you can read the first type into a byte array using the BinaryReader and then use BitConverter to convert these bytes into the type you want. You can then use BinaryReader as normal until the start of the next record.
byte[] headermarker = new byte[4];
int count;
// if bytes available in underlying stream.
while ((count = br.Read(headermarker, 0, 4) > 0 )
{
Int32 marker = BitConverter.ToInt32(headermarker, 0);
//
// now use Binary Reader for the rest of the record until we loop
//
}
Does anyone know why I'm getting the "Unexpected end of data" error message when un-gzipping the gzip file?
To verify the bytes data is not corrupted, I use the FooTest4.csv to write to file and was able to opened the file successfully.
Both 'FooTest3.csv.gzand 'FooTest2.csv.gz ran into "Unexpected end of data" when un-gzipping.
public static List<byte> CompressFile(List<byte> parmRawBytes)
{
//Initialize variables...
List<byte> returnModifiedBytes = null;
File.WriteAllBytes(#"X:\FooTest4.csv", parmRawBytes.ToArray());
using (var memoryStream = new MemoryStream())
{
using (var gzipStream = new GZipStream(memoryStream, CompressionMode.Compress, false))
{
gzipStream.Write(parmRawBytes.ToArray(), 0, parmRawBytes.ToArray().Length);
gzipStream.Flush();
File.WriteAllBytes(#"X:\FooTest3.csv.gz", memoryStream.ToArray());
returnModifiedBytes = memoryStream.ToArray().ToList();
}
}
File.WriteAllBytes(#"X:\FooTest2.csv.gz", returnModifiedBytes.ToArray());
return returnModifiedBytes;
}
GZipStream needs to be closed so it can write some terminating data to the end of the buffer to complete the gzip encoding.
byte[] inputBytes = ...;
using (var compressedStream = new MemoryStream())
{
using (var compressor = new GZipStream(compressedStream, CompressionMode.Compress))
{
compressor.Write(inputBytes, 0, inputBytes.Length);
}
// get bytes after the gzip stream is closed
File.WriteAllBytes(pathToFile, compressedStream.ToArray());
}
Instead of loading the bytes, compressing and saving them you could do do compression and writing at once. Also I don't know why you're using List<Byte> instead of byte[], maybe this could be it.
void CompressFile(string inputPath, string outputPath)
{
Stream readStream = new FileStream(inputPath, Filemode.Open);
Stream writeStream = new FileStream(outputPath, FileMode.Create);
Stream compressionStream = new GZipStream(writeStream. CompressionMode.Compress);
byte[] data = new byte[readStream.Length];
readStream.Read(data, 0, data.Length);
compressionStream.Write(data, 0, data.Length);
readStream.Close();
writeStream.Close();
}
byte[] CompressFile(string inputPath)
{
byte[] data = File.ReadAllBytes(inputPath);
MemoryStream memStream = new MemoryStream(data);
var gzipStream = new GZipStream(memStream, CompressionMode.Compress);
gzipStream.Write(data, 0, data.Length);
gzipStream.Close();
return gzipStream.ToArray();
}
PS: I wrote the code in the text editor, so there might be errors. Also you say the error is on the "unzippiing", why no show us the unzip code?
I'm working with GZipStream at the moment using .net 3.5.
I have two methods listed below. As input file I use text file which consists of chars 's'. Size of the file is 2MB. This code works fine if I use .net 4.5 but with .net 3.5 after compress and decompress I get file of size 435KB which of course isn't the same with source file.
If I try to decompress file via WinRAR it is also looks good (the same with source file).
If I try decompress file using GZipStream from .net4.5 (file compressed via GZipStream from .net 3.5) the result is bad.
UPD:
In general I really need to read the file as several separate gzip chunks, in this case all the bytes of copressed files are read at one call of the Read() method so I still don't understand why decompressing doesn't works.
public void CompressFile()
{
string fileIn = #"D:\sin2.txt";
string fileOut = #"D:\sin2.txt.pgz";
using (var fout = File.Create(fileOut))
{
using (var fin = File.OpenRead(fileIn))
{
using (var zip = new GZipStream(fout, CompressionMode.Compress))
{
var buffer = new byte[1024 * 1024 * 10];
int n = fin.Read(buffer, 0, buffer.Length);
zip.Write(buffer, 0, n);
}
}
}
}
public void DecompressFile()
{
string fileIn = #"D:\sin2.txt.pgz";
string fileOut = #"D:\sin2.1.txt";
using (var fsout = File.Create(fileOut))
{
using (var fsIn = File.OpenRead(fileIn))
{
var buffer = new byte[1024 * 1024 * 10];
int n;
while ((n = fsIn.Read(buffer, 0, buffer.Length)) > 0)
{
using (var ms = new MemoryStream(buffer, 0, n))
{
using (var zip = new GZipStream(ms, CompressionMode.Decompress))
{
int nRead = zip.Read(buffer, 0, buffer.Length);
fsout.Write(buffer, 0, nRead);
}
}
}
}
}
}
You're trying to decompress each "chunk" as if it's a separate gzip file. Don't do that - just read from the GZipStream in a loop:
using (var fsout = File.Create(fileOut))
{
using (var fsIn = File.OpenRead(fileIn))
{
using (var zip = new GZipStream(fsIn, CompressionMode.Decompress))
{
var buffer = new byte[1024 * 32];
int bytesRead;
while ((bytesRead = zip.Read(buffer, 0, buffer.Length)) > 0)
{
fsout.Write(buffer, 0, bytesRead);
}
}
}
}
Note that your compression code should look similar, reading in a loop rather than assuming a single call to Read will read all the data.
(Personally I'd skip fsIn, and just use new GZipStream(File.OpenRead(fileIn)) but that's just a personal preference.)
First, as #Jon Skeet mentioned, you are not using Stream.Read method correctly. It doesn't matter if your buffer is big enough or not, the stream is allowed to return less bytes than requested, with zero indicating no more, so reading from stream should always be performed in a loop.
However the main problem in your decompress code is the way you share the buffer. Your read the input into a buffer, than wrap it in a MemoryStream (note that the constructor used does not make a copy of the passed array, but actually sets it as it's internal buffer), and then you try to read and write to that buffer at the same time. Taking into account that decompressing writes data "faster" than reading, it's surprising that your code works at all.
The correct implementation is quite simple
static void CompressFile()
{
string fileIn = #"D:\sin2.txt";
string fileOut = #"D:\sin2.txt.pgz";
using (var input = File.OpenRead(fileIn))
using (var output = new GZipStream(File.Create(fileOut), CompressionMode.Compress))
Write(input, output);
}
static void DecompressFile()
{
string fileIn = #"D:\sin2.txt.pgz";
string fileOut = #"D:\sin2.1.txt";
using (var input = new GZipStream(File.OpenRead(fileIn), CompressionMode.Decompress))
using (var output = File.Create(fileOut))
Write(input, output);
}
static void Write(Stream input, Stream output, int bufferSize = 10 * 1024 * 1024)
{
var buffer = new byte[bufferSize];
for (int readCount; (readCount = input.Read(buffer, 0, buffer.Length)) > 0;)
output.Write(buffer, 0, readCount);
}
I have some code that downloads gzipped files, and decompresses them. The problem is, I can't get it to decompress the whole file, it only reads the first 4096 bytes and then about 500 more.
Byte[] buffer = new Byte[4096];
int count = 0;
FileStream fileInput = new FileStream("input.gzip", FileMode.Open, FileAccess.Read, FileShare.Read);
FileStream fileOutput = new FileStream("output.dat", FileMode.Create, FileAccess.Write, FileShare.None);
GZipStream gzipStream = new GZipStream(fileInput, CompressionMode.Decompress, true);
// Read from gzip steam
while ((count = gzipStream.Read(buffer, 0, buffer.Length)) > 0)
{
// Write to output file
fileOutput.Write(buffer, 0, count);
}
// Close the streams
...
I've checked the downloaded file; it's 13MB when compressed, and contains one XML file. I've manually decompressed the XML file, and the content is all there. But when I do it with this code, it only outputs the very beginning of the XML file.
Anyone have any ideas why this might be happening?
EDIT
Try not leaving the GZipStream open:
GZipStream gzipStream = new GZipStream(fileInput, CompressionMode.Decompress,
false);
or
GZipStream gzipStream = new GZipStream(fileInput, CompressionMode.Decompress);
I ended up using a gzip executable to do the decompression instead of a GZipStream. It can't handle the file for some reason, but the executable can.
Same thing happened to me. In my case only reads up to 6 lines and then reached end of file. So I realized that although the extension is gz, it was compressed by another algorithm not supported by GZipStream. So I used SevenZipSharp library and it worked. This is my code
You can use SevenZipSharp library
using (var input = File.OpenRead(lstFiles[0]))
{
using (var ds = new SevenZipExtractor(input))
{
//ds.ExtractionFinished += DsOnExtractionFinished;
var mem = new MemoryStream();
ds.ExtractFile(0, mem);
using (var sr = new StreamReader(mem))
{
var iCount = 0;
String line;
mem.Position = 0;
while ((line = sr.ReadLine()) != null && iCount < 100)
{
iCount++;
LstOutput.Items.Add(line);
}
}
}
}
Are you calling Close or Flush on fileOutput? (Or just wrap it in a using, which is recommended practice.) If you don't the file might not be flushed to disk when your program ends.