Develop Reference

How to get the length of the compressed data from DeflateStream?

The following is a simple compression method I wrote using DeflateStream:
public static int Compress(
byte[] inputData,
int inputStartIndex,
int inputLength,
byte[] outputData,
int outputStartIndex,
int outputLength)
{
if (inputData == null)
throw new ArgumentNullException("inputData must be non-null");
MemoryStream memStream = new MemoryStream(outputData, outputStartIndex, outputLength);
using (DeflateStream dstream = new DeflateStream(memStream, CompressionLevel.Optimal))
{
dstream.Write(inputData, inputStartIndex, inputLength);
return (int)(memStream.Position - outputStartIndex);
}
}
What is special in this method is that I didn't use the parameter-less constructor of MemoryStream. This is because it is a high-throughput server. Array outputData is rented from ArrayPool, to be used to hold the compressed bytes, so that after I make use of it I can return it to ArrayPool.
The compression happened properly, and the compressed data is properly placed into outputData, but memStream.Position was zero, so I can't find out how many bytes have been written into the MemoryStream.
Only part of outputData is occupied by the compressed data. How do I find out the length of the compressed data?

MemoryStream.Position is 0 because data was not actually written there yet at the point you read Position. Instead, tell DeflateStream to leave underlying stream (MemoryStream) open, then dispose DeflateStream. At this point you can be sure it's done writing whatever it needs. Now you can read MemoryStream.Position to check how many bytes were written:
public static int Compress(
byte[] inputData,
int inputStartIndex,
int inputLength,
byte[] outputData,
int outputStartIndex,
int outputLength)
{
if (inputData == null)
throw new ArgumentNullException("inputData must be non-null");
using (var memStream = new MemoryStream(outputData, outputStartIndex, outputLength)) {
// leave open
using (DeflateStream dstream = new DeflateStream(memStream, CompressionLevel.Optimal, leaveOpen: true)) {
dstream.Write(inputData, inputStartIndex, inputLength);
}
return (int) memStream.Position; // now it's not 0
}
}
You also don't need to substract outputStartIndex, because Position is already relative to that index you passed to constructor.

Write a file to the beginning of a specific sector of a hard drive

I am trying to write a file to a HDD but at a specific sector, I get have no data being wrote. Its about 1.5GB, Shouldnt I see activity right away?
using (FileStream stream = new FileStream("/Resources/MyFileName.NoExt", FileMode.Open)) // Grab project zip from the embedded resources.
{
rvalsfp = SetFilePointerEx(diskHandle, sec943715712 * numBytesPerSector, out movethighlong, EMoveMethod.Begin);
while (stream.Position < stream.Length)
{
//byte array to hold file bytes
byte[] bits = new byte[stream.Length];
//read in the bytes
stream.Read(bits, 0, (int)stream.Length);
//write out the bytes
rval = WriteFile(diskHandle, bits, (int)stream.Length, out numBytesWritten, IntPtr.Zero);
}
stream.Close();
}

Correct way to use GZipStream in dotNET C#

I'm working with GZipStream at the moment using .net 3.5.
I have two methods listed below. As input file I use text file which consists of chars 's'. Size of the file is 2MB. This code works fine if I use .net 4.5 but with .net 3.5 after compress and decompress I get file of size 435KB which of course isn't the same with source file.
If I try to decompress file via WinRAR it is also looks good (the same with source file).
If I try decompress file using GZipStream from .net4.5 (file compressed via GZipStream from .net 3.5) the result is bad.
UPD:
In general I really need to read the file as several separate gzip chunks, in this case all the bytes of copressed files are read at one call of the Read() method so I still don't understand why decompressing doesn't works.
public void CompressFile()
{
string fileIn = #"D:\sin2.txt";
string fileOut = #"D:\sin2.txt.pgz";
using (var fout = File.Create(fileOut))
{
using (var fin = File.OpenRead(fileIn))
{
using (var zip = new GZipStream(fout, CompressionMode.Compress))
{
var buffer = new byte[1024 * 1024 * 10];
int n = fin.Read(buffer, 0, buffer.Length);
zip.Write(buffer, 0, n);
}
}
}
}
public void DecompressFile()
{
string fileIn = #"D:\sin2.txt.pgz";
string fileOut = #"D:\sin2.1.txt";
using (var fsout = File.Create(fileOut))
{
using (var fsIn = File.OpenRead(fileIn))
{
var buffer = new byte[1024 * 1024 * 10];
int n;
while ((n = fsIn.Read(buffer, 0, buffer.Length)) > 0)
{
using (var ms = new MemoryStream(buffer, 0, n))
{
using (var zip = new GZipStream(ms, CompressionMode.Decompress))
{
int nRead = zip.Read(buffer, 0, buffer.Length);
fsout.Write(buffer, 0, nRead);
}
}
}
}
}
}

You're trying to decompress each "chunk" as if it's a separate gzip file. Don't do that - just read from the GZipStream in a loop:
using (var fsout = File.Create(fileOut))
{
using (var fsIn = File.OpenRead(fileIn))
{
using (var zip = new GZipStream(fsIn, CompressionMode.Decompress))
{
var buffer = new byte[1024 * 32];
int bytesRead;
while ((bytesRead = zip.Read(buffer, 0, buffer.Length)) > 0)
{
fsout.Write(buffer, 0, bytesRead);
}
}
}
}
Note that your compression code should look similar, reading in a loop rather than assuming a single call to Read will read all the data.
(Personally I'd skip fsIn, and just use new GZipStream(File.OpenRead(fileIn)) but that's just a personal preference.)

First, as #Jon Skeet mentioned, you are not using Stream.Read method correctly. It doesn't matter if your buffer is big enough or not, the stream is allowed to return less bytes than requested, with zero indicating no more, so reading from stream should always be performed in a loop.
However the main problem in your decompress code is the way you share the buffer. Your read the input into a buffer, than wrap it in a MemoryStream (note that the constructor used does not make a copy of the passed array, but actually sets it as it's internal buffer), and then you try to read and write to that buffer at the same time. Taking into account that decompressing writes data "faster" than reading, it's surprising that your code works at all.
The correct implementation is quite simple
static void CompressFile()
{
string fileIn = #"D:\sin2.txt";
string fileOut = #"D:\sin2.txt.pgz";
using (var input = File.OpenRead(fileIn))
using (var output = new GZipStream(File.Create(fileOut), CompressionMode.Compress))
Write(input, output);
}
static void DecompressFile()
{
string fileIn = #"D:\sin2.txt.pgz";
string fileOut = #"D:\sin2.1.txt";
using (var input = new GZipStream(File.OpenRead(fileIn), CompressionMode.Decompress))
using (var output = File.Create(fileOut))
Write(input, output);
}
static void Write(Stream input, Stream output, int bufferSize = 10 * 1024 * 1024)
{
var buffer = new byte[bufferSize];
for (int readCount; (readCount = input.Read(buffer, 0, buffer.Length)) > 0;)
output.Write(buffer, 0, readCount);
}

c# Stream Reading and Deserialization

I have this code:
public static List<ReplicableObject> ParseStreamForObjects(Stream stream)
{
List<ReplicableObject> result = new List<ReplicableObject>();
while (true)
{
// HERE I want to check that there's at least four bytes left in the stream
BinaryReader br = new BinaryReader(stream);
int length = br.ReadInt32();
// HERE I want to check that there's enough bytes left in the stream
byte[] bytes = br.ReadBytes(length);
MemoryStream ms = new MemoryStream(bytes);
ms.Position = 0;
result.Add((ReplicableObject) Formatter.Deserialize(ms));
ms.Close();
br.Close();
}
return result;
}
Unfortunately, the stream object is always going to be a TCP stream, which means no seek operations. So how can I check to make sure that I'm not over-running the stream where I've put the // HERE comments?

I don't think there's any way to query a NetworkStream to find the data you're looking for. What you'll probably need to do is buffer whatever data the stream makes available into another data structure, then parse objects out of that structure once you know it's got enough bytes in it.
The NetworkStream class provides a DataAvailable property that tells you if any data is available to be read, and the Read() method returns a value indicating how many bytes it actually retrieved. You should be able to use those values to do the buffering you need.

See Mr. Skeets page
Sometimes, you don't know the length of the stream in advance (for instance a network stream) and just want to read the whole lot into a buffer. Here's a method to do just that:
/// <summary>
/// Reads data from a stream until the end is reached. The
/// data is returned as a byte array. An IOException is
/// thrown if any of the underlying IO calls fail.
/// </summary>
/// <param name="stream">The stream to read data from</param>
public static byte[] ReadFully (Stream stream)
{
byte[] buffer = new byte[32768];
using (MemoryStream ms = new MemoryStream())
{
while (true)
{
int read = stream.Read (buffer, 0, buffer.Length);
if (read <= 0)
return ms.ToArray();
ms.Write (buffer, 0, read);
}
}
}
This should give you some ideas. Once you have the byte array, checking the Length will be easy to do.
In your example, it would look something like this:
int bytes_to_read = 4;
byte[] length_bytes = new byte[bytes_to_read];
int bytes_read = stream.Read(length_bytes, 0, length_bytes.Length);
// Check that there's at least four bytes left in the stream
if(bytes_read != bytes_to_read) break;
int bytes_in_msg = BitConverter.ToInt32(length_bytes);
byte[] msg_bytes = new byte[bytes_in_msg];
bytes_read = stream.Read(msg_bytes, 0, msg_bytes.Length);
// Check that there's enough bytes left in the stream
if(bytes_read != bytes_in_msg ) break;
...

Send binary file in web service soap response

I have response stream from a ftp web request that returns binary file.
I wanted to get the binary data into byte[] and then encode that array and send it as web service response.
Stream responseStream = webResponse.GetResponseStream();
byte[] byteToEncode= ReadFully(responseStream,1024);
String str=Convert.ToBase64String(byteToEncode);
I used that function I found online to convert the stream into byte[]
/// <summary>
/// Reads data from a stream until the end is reached. The
/// data is returned as a byte array. An IOException is
/// thrown if any of the underlying IO calls fail.
/// </summary>
/// <param name="stream">The stream to read data from</param>
/// <param name="initialLength">The initial buffer length</param>
public static byte[] ReadFully (Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, just use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[initialLength];
int read=0;
int chunk;
while ( (chunk = stream.Read(buffer, read, buffer.Length-read)) > 0)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte==-1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length*2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read]=(byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
But I got an exception says:
This stream does not support seek operations.
I would appreciate any help.
Thanks,
Sarah

You'll probably need to re-write the ReadFully function to avoid properties like Length, which need to know more about the stream than is possible when the full stream hasn't been completely received.