Here is the method I like to use. I believe, there is nothing new with this code.
public static byte[] ReadFully(Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 1K.
if (initialLength < 1)
{
initialLength = 1024;
}
byte[] buffer = new byte[initialLength];
int read = 0;
int chunk;
while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
My goal is to read data sent from TCP Clients e.g. box{"id":1,"aid":1}
It is a command to interpret in my application in Jason-like text.
And this text is not necessarily at the same size each time.
Next time there can be run{"id":1,"aid":1,"opt":1}.
The method called by this line;
var serializedMessageBytes = ReadFully(_receiveMemoryStream, 1024);
Please click to see; Received data in receiveMemoryStream
Although we can see the data in the stream,
in the ReadFully method, "chunck" always return 0 and the method returns {byte[0]}.
Any help effort greatly appreciated.
Looking at your stream in the Watch window, the Position of the stream (19) is at the end of the data, hence there is nothing left to read. This is possibly because you have just written data to the stream and have not subsequently reset the position.
Add a stream.Position = 0; or stream.Seek(0, System.IO.SeekOrigin.Begin); statement at the start of the function if you are happy to always read from the start of the stream, or check the code that populates the stream. Note though that some stream implementations do not support seeking.
Related
I'm trying to GetResponseStream from a GET request and convert this into byte[]. The Stream is non-seekable so I can't access stream.Length. response.ContentLength is unreliable.
private byte[] streamToBytes(Stream stream, int bufferSize = 4096)
{
byte[] buffer = new byte[bufferSize];
int read = 0;
int pos = 0;
List<byte> bytes = new List<byte>();
while (true) { // `bufferSize > read` does **not** mean stream end
while (bufferSize == (read = stream.Read(buffer, pos, bufferSize))) {
pos += read;
bytes.AddRange(buffer);
}
if (read > 0) {
byte[] _buffer = new byte[read];
Array.Copy(buffer, _buffer, read);
bytes.AddRange(_buffer);
} else break;
}
return bytes.ToArray();
}
An ArgumentOutOfRangeException gets thrown on the Read in the second iteration of the while loop. MSDN says
ArgumentOutOfRangeException offset or count is negative.
I know this can't be true because offset (pos) is 0 + read >= 0 and count (bufferSize) is 4096 so why am I getting exceptions thrown at me?
I'm trying to keep streamToBytes as generic as possible so I can use it in future async methods, too.
If how the request is made helps, here are the relevant bits
HttpWebRequest request = (HttpWebRequest)WebRequest.Create((new Uri("http://google.com")).ToString());
request.Method = "GET";
request.KeepAlive = true;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream stream = response.GetResponseStream();
byte[] responseBytes = streamToBytes(stream);
As a simpler alternative:
using(var memStream = new MemoryStream())
{
stream.CopyTo(memStream);
return memStream.ToArray();
}
Your code has two mistakes:
The offset on Read is an offset into the buffer, not the position in the stream (as #Ulugbek already noted). This means you don't need the pos variable anymore.
You cannot assume that bufferSize == read even before the stream reached its end. You need to check for > 0 instead.
So your reading loop becomes:
while ((read = stream.Read(buffer, 0, bufferSize)) > 0)
{
bytes.AddRange(buffer.Take(read));
}
You can now drop the special handling for the last block. Simplifying your code to:
private byte[] streamToBytes(Stream stream, int bufferSize = 4096)
{
byte[] buffer = new byte[bufferSize];
int read = 0;
List<byte> bytes = new List<byte>();
while ((read = stream.Read(buffer, 0, bufferSize)) > 0)
{
bytes.AddRange(buffer.Take(read));
}
return bytes.ToArray();
}
Using List<byte> instead of MemoryStream isn't a great idea either. AddRange need to iterate over the bytes individually, instead of using a low level copy operation. Replacing the list with a memory stream, the code becomes:
private byte[] streamToBytes(Stream stream, int bufferSize = 4096)
{
byte[] buffer = new byte[bufferSize];
int read = 0;
using(var bytes = new MemoryStream())
{
while ((read = stream.Read(buffer, 0, bufferSize)) > 0)
{
bytes.Write(buffer, 0, read);
}
return bytes.ToArray();
}
}
You could even split it into two parts, one does the copying into the memory stream, the other handles the creating of the memory stream and turning it into a byte array:
private static void CopyStream(Stream source, Stream destination, int bufferSize = 4096)
{
byte[] buffer = new byte[bufferSize];
int read = 0;
while ((read = source.Read(buffer, 0, bufferSize)) > 0)
{
destination.Write(buffer, 0, read);
}
}
private static byte[] StreamToBytes(Stream stream, int bufferSize = 4096)
{
using(var memStream = new MemoryStream())
{
CopyStream(stream, memStream);
return memStream.ToArray();
}
}
This is probably very similar to what Stream.CopyTo does internally.
Change
stream.Read(buffer, pos, bufferSize)
to
stream.Read(buffer, 0, bufferSize)
I need to read a bunch of line from device through telnet. When I start reading it the device send the data and finishes sending. The problem is when I review the received data I can see some characters are missing. What's the problem ?
Here's my function which does the receiving task:
//calling the function
string out_string = Encoding.Default.GetString(ReadFully(readStream,0));
//the function which read the data
public static byte[] ReadFully(Stream stream, int initialLength)
{ if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[initialLength];
int read = 0;
int chunk;
while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0 && (Byte)stream.ReadByte() != 65)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
Your conditional (Byte)stream.ReadByte() != 65 is throwing away a character if it is not 65 (ASCII A).
It's that second part of your while loop:
while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0 &&
(Byte)stream.ReadByte() != 65) //<-- Here
Your always reading an extra byte in each loop and (if it's not 65) never storing that byte anywhere.
Note, Stream.ReadByte:
Reads a byte from the stream and advances the position within the stream by one byte ...
(My emphasis)
I have a client-server application where the server transmits a 4-byte integer specifying how large the next transmission is going to be. When I read the 4-byte integer on the client side (specifying FILE_SIZE), the next time I read the stream I get FILE_SIZE + 4 bytes read.
Do I need to specify the offset to 4 when reading from this stream, or is there a way to automatically advance the NetworkStream so my offset can always be 0?
SERVER
NetworkStream theStream = theClient.getStream();
//...
//Calculate file size with FileInfo and put into byte[] size
//...
theStream.Write(size, 0, size.Length);
theStream.Flush();
CLIENT
NetworkStream theStream = theClient.getStream();
//read size
byte[] size = new byte[4];
int bytesRead = theStream.Read(size, 0, 4);
...
//read content
byte[] content = new byte[4096];
bytesRead = theStream.Read(content, 0, 4096);
Console.WriteLine(bytesRead); // <-- Prints filesize + 4
Right; found it; FileInfo.Length is a long; your call to:
binWrite.Write(fileInfo.Length);
writes 8 bytes, little-endian. You then read that back via:
filesize = binRead.ReadInt32();
which little-endian will give you the same value (for 32 bits, at least). You have 4 00 bytes left unused in the stream, though (from the high-bytes of the long) - hence the 4 byte mismatch.
Use one of:
binWrite.Write((int)fileInfo.Length);
filesize = binRead.ReadInt64();
NetworkStream certainly advances, but in both cases, your read is unreliable; a classic "read known amount of content" would be:
static void ReadAll(Stream source, byte[] buffer, int bytes) {
if(bytes > buffer.Length) throw new ArgumentOutOfRangeException("bytes");
int bytesRead, offset = 0;
while(bytes > 0 && (bytesRead = source.Reader(buffer, offset, bytes)) > 0) {
offset += bytesRead;
bytes -= bytesRead;
}
if(bytes != 0) throw new EndOfStreamException();
}
with:
ReadAll(theStream, size, 4);
...
ReadAll(theStream, content, contentLength);
note also that you need to be careful with endianness when parsing the length-prefix.
I suspect you simply aren't reading the complete data.
This is a little more tricky than I first imagined. I'm trying to read n bytes from a stream.
The MSDN claims that Read does not have to return n bytes, it just must return at least 1 and up to n bytes, with 0 bytes being the special case of reaching the end of the stream.
Typically, I'm using something like
var buf = new byte[size];
var count = stream.Read (buf, 0, size);
if (count != size) {
buf = buf.Take (count).ToArray ();
}
yield return buf;
I'm hoping for exactly size bytes but by spec FileStream would be allowed to return a large number of 1-byte chunks as well. This must be avoided.
One way to solve this would be to have 2 buffers, one for reading and one for collecting the chunks until we got the requested number of bytes. That's a little cumbersome though.
I also had a look at BinaryReader but its spec also does not clearly state that n bytes will be returned for sure.
To clarify: Of course, upon the end of the stream the returned number of bytes may be less than size - that's not a problem. I'm only talking about not receiving n bytes even though they are available in the stream.
A slightly more readable version:
int offset = 0;
while (offset < count)
{
int read = stream.Read(buffer, offset, count - offset);
if (read == 0)
throw new System.IO.EndOfStreamException();
offset += read;
}
Or written as an extension method for the Stream class:
public static class StreamUtils
{
public static byte[] ReadExactly(this System.IO.Stream stream, int count)
{
byte[] buffer = new byte[count];
int offset = 0;
while (offset < count)
{
int read = stream.Read(buffer, offset, count - offset);
if (read == 0)
throw new System.IO.EndOfStreamException();
offset += read;
}
System.Diagnostics.Debug.Assert(offset == count);
return buffer;
}
}
Simply; you loop;
int read, offset = 0;
while(leftToRead > 0 && (read = stream.Read(buf, offset, leftToRead)) > 0) {
leftToRead -= read;
offset += read;
}
if(leftToRead > 0) throw new EndOfStreamException(); // not enough!
After this, buf should have been populated with exactly the right amount of data from the stream, or will have thrown an EOF.
Getting everything together from answers here I came up with the following solution. It relies on a source stream length. Works on .NET core 3.1
/// <summary>
/// Copy stream based on source stream length
/// </summary>
/// <param name="source"></param>
/// <param name="destination"></param>
/// <param name="bufferSize">
/// A value that is the largest multiple of 4096 and is still smaller than the LOH threshold (85K).
/// So the buffer is likely to be collected at Gen0, and it offers a significant improvement in Copy performance.
/// </param>
/// <returns></returns>
private async Task CopyStream(Stream source, Stream destination, int bufferSize = 81920)
{
var buffer = new byte[bufferSize];
var offset = 0;
while (offset < source.Length)
{
var leftToRead = source.Length - offset;
var lengthToRead = leftToRead - buffer.Length < 0 ? (int)(leftToRead) : buffer.Length;
var read = await source.ReadAsync(buffer, 0, lengthToRead).ConfigureAwait(false);
if (read == 0)
break;
await destination.WriteAsync(buffer, 0, lengthToRead).ConfigureAwait(false);
offset += read;
}
destination.Seek(0, SeekOrigin.Begin);
}
I'm trying to encrypt and decrypt a file stream over a socket using RijndaelManaged, but I keep bumping into the exception
CryptographicException: Length of the data to decrypt is invalid.
at System.Security.Cryptography.RijndaelManagedTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing)
The exception is thrown at the end of the using statement in receiveFile, when the whole file has been transferred.
I tried searching the web but only found answers to problems that arise when using Encoding when encrypting and decrypting a single string. I use a FileStream, so I don't specify any Encoding to be used, so that should not be the problem. These are my methods:
private void transferFile(FileInfo file, long position, long readBytes)
{
// transfer on socket stream
Stream stream = new FileStream(file.FullName, FileMode.Open);
if (position > 0)
{
stream.Seek(position, SeekOrigin.Begin);
}
// if this should be encrypted, wrap the encryptor stream
if (UseCipher)
{
stream = new CryptoStream(stream, streamEncryptor, CryptoStreamMode.Read);
}
using (stream)
{
int read;
byte[] array = new byte[8096];
while ((read = stream.Read(array, 0, array.Length)) > 0)
{
streamSocket.Send(array, 0, read, SocketFlags.None);
position += read;
}
}
}
private void receiveFile(FileInfo transferFile)
{
byte[] array = new byte[8096];
// receive file
Stream stream = new FileStream(transferFile.FullName, FileMode.Append);
if (UseCipher)
{
stream = new CryptoStream(stream, streamDecryptor, CryptoStreamMode.Write);
}
using (stream)
{
long position = new FileInfo(transferFile.Path).Length;
while (position < transferFile.Length)
{
int maxRead = Math.Min(array.Length, (int)(transferFile.Length - position));
int read = position < array.Length
? streamSocket.Receive(array, maxRead, SocketFlags.None)
: streamSocket.Receive(array, SocketFlags.None);
stream.Write(array, 0, read);
position += read;
}
}
}
This is the method I use to set up the ciphers. byte[] init is a generated byte array.
private void setupStreamCipher(byte[] init)
{
RijndaelManaged cipher = new RijndaelManaged();
cipher.KeySize = cipher.BlockSize = 256; // bit size
cipher.Mode = CipherMode.ECB;
cipher.Padding = PaddingMode.ISO10126;
byte[] keyBytes = new byte[32];
byte[] ivBytes = new byte[32];
Array.Copy(init, keyBytes, 32);
Array.Copy(init, 32, ivBytes, 0, 32);
streamEncryptor = cipher.CreateEncryptor(keyBytes, ivBytes);
streamDecryptor = cipher.CreateDecryptor(keyBytes, ivBytes);
}
Anyone have an idea in what I might be doing wrong?
It looks to me like you're not properly sending the final block. You need to at least FlushFinalBlock() the sending CryptoStream in order to ensure that the final block (which the receiving stream is looking for) is sent.
By the way, CipherMode.ECB is more than likely an epic fail in terms of security for what you're doing. At least use CipherMode.CBC (cipher-block chaining) which actually uses the IV and makes each block dependent on the previous one.
EDIT: Whoops, the enciphering stream is in read mode. In that case you need to make sure you read to EOF so that the CryptoStream can deal with the final block, rather than stopping after readBytes. It's probably easier to control if you run the enciphering stream in write mode.
One more note: You cannot assume that bytes in equals bytes out. Block ciphers have a fixed block size they process, and unless you are using a cipher mode that converts the block cipher to a stream cipher, there will be padding that makes the ciphertext longer than the plaintext.
After the comment made by Jeffrey Hantin, I changed some lines in receiveFile to
using (stream) {
FileInfo finfo = new FileInfo(transferFile.Path);
long position = finfo.Length;
while (position < transferFile.Length) {
int maxRead = Math.Min(array.Length, (int)(transferFile.Length - position));
int read = position < array.Length
? streamSocket.Receive(array, maxRead, SocketFlags.None)
: streamSocket.Receive(array, SocketFlags.None);
stream.Write(array, 0, read);
position += read;
}
}
->
using (stream) {
int read = array.Length;
while ((read = streamSocket.Receive(array, read, SocketFlags.None)) > 0) {
stream.Write(array, 0, read);
if ((read = streamSocket.Available) == 0) {
break;
}
}
}
And voila, she works (because of the ever so kind padding that I didn't care to bother about earlier). I'm not sure what happens if Available returns 0 even though all data hasn't been transferred, but I'll tend to that later in that case. Thanks for your help Jeffrey!
Regards.
cipher.Mode = CipherMode.ECB;
Argh! Rolling your own security code is almost always a bad idea.
Mine i just removed the padding and it works
Commented this out - cipher.Padding = PaddingMode.ISO10126;