I need to read a bunch of line from device through telnet. When I start reading it the device send the data and finishes sending. The problem is when I review the received data I can see some characters are missing. What's the problem ?
Here's my function which does the receiving task:
//calling the function
string out_string = Encoding.Default.GetString(ReadFully(readStream,0));
//the function which read the data
public static byte[] ReadFully(Stream stream, int initialLength)
{ if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[initialLength];
int read = 0;
int chunk;
while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0 && (Byte)stream.ReadByte() != 65)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
Your conditional (Byte)stream.ReadByte() != 65 is throwing away a character if it is not 65 (ASCII A).
It's that second part of your while loop:
while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0 &&
(Byte)stream.ReadByte() != 65) //<-- Here
Your always reading an extra byte in each loop and (if it's not 65) never storing that byte anywhere.
Note, Stream.ReadByte:
Reads a byte from the stream and advances the position within the stream by one byte ...
(My emphasis)
Related
I currently have a memorystream with length of about 30000 (Named memStream here)
I wished to read this memorystream in chunks using the following code (I picked up on the net and modified somewhat):
byte[] chunk = new byte[4096];
bool hasNext = true;
while(hasNext)
{
int index = 0;
while (index < chunk.Length)
{
int bytesRead = memStream.Read(chunk, index, chunk.Length - index);
if (bytesRead == 0)
{
break;
}
index += bytesRead;
//Do something with this chunk
}
if (index != 0) // Our previous chunk may have been the last one
{
//Do something with the last chunk
}
if (index != chunk.Length) // We didn't read a full chunk: we're done
{
hasNext = false;
}
}
yet the following read()method doesn't appear to be working
int bytesRead = memStream.Read(chunk, index, chunk.Length - index);
WHERE
chunk: new byte[4096]
index: 0
memstream: capacitiy & length : 34272
memstream: position 0 (according to VS watch)
Always returns
0 bytesRead
Chunk with all values containing '0'
Any idea why? Could this be a rights permission?
Thank you for your time.
After creating and filling the MemoryStream, you need to set the read position to the begining like so:
memStream.Seek(0, SeekOrigin.Begin);
Here is the method I like to use. I believe, there is nothing new with this code.
public static byte[] ReadFully(Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 1K.
if (initialLength < 1)
{
initialLength = 1024;
}
byte[] buffer = new byte[initialLength];
int read = 0;
int chunk;
while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
My goal is to read data sent from TCP Clients e.g. box{"id":1,"aid":1}
It is a command to interpret in my application in Jason-like text.
And this text is not necessarily at the same size each time.
Next time there can be run{"id":1,"aid":1,"opt":1}.
The method called by this line;
var serializedMessageBytes = ReadFully(_receiveMemoryStream, 1024);
Please click to see; Received data in receiveMemoryStream
Although we can see the data in the stream,
in the ReadFully method, "chunck" always return 0 and the method returns {byte[0]}.
Any help effort greatly appreciated.
Looking at your stream in the Watch window, the Position of the stream (19) is at the end of the data, hence there is nothing left to read. This is possibly because you have just written data to the stream and have not subsequently reset the position.
Add a stream.Position = 0; or stream.Seek(0, System.IO.SeekOrigin.Begin); statement at the start of the function if you are happy to always read from the start of the stream, or check the code that populates the stream. Note though that some stream implementations do not support seeking.
While converting some older code to use async in c#, I started seeing problems in variations of return values from the Read() and ReadAsync() methods of the DeflateStream.
I thought that the transition from synchronous code like
bytesRead = deflateStream.Read(buffer, 0, uncompressedSize);
to it's equivalent asynchronous version of
bytesRead = await deflateStream.ReadAsync(buffer, 0, uncompressedSize);
should always return the same value.
See updated code added to bottom of question - that uses streams the correct way - hence making the initial question irrelevant
I found that after number of iterations this didn't hold true, and in my specific case was causing random errors in the converted application.
Am I missing something here?
Below is simple repro case (in a console app), where the Assert will break for me in the ReadAsync method on iteration #412, giving output that looks like this:
....
ReadAsync #410 - 2055 bytes read
ReadAsync #411 - 2055 bytes read
ReadAsync #412 - 453 bytes read
---- DEBUG ASSERTION FAILED ----
My question is, why is the DeflateStream.ReadAsync method returning 453 bytes at this point?
Note: this only happens with certain input strings - the massive StringBuilder stuff in the CreateProblemDataString was the best way I could think of constructing the string for this post.
class Program
{
static byte[] DataAsByteArray;
static int uncompressedSize;
static void Main(string[] args)
{
string problemDataString = CreateProblemDataString();
DataAsByteArray = Encoding.ASCII.GetBytes(problemDataString);
uncompressedSize = DataAsByteArray.Length;
MemoryStream memoryStream = new MemoryStream();
using (DeflateStream deflateStream = new DeflateStream(memoryStream, CompressionMode.Compress, true))
{
for (int i = 0; i < 1000; i++)
{
deflateStream.Write(DataAsByteArray, 0, uncompressedSize);
}
}
// now read it back synchronously
Read(memoryStream);
// now read it back asynchronously
Task retval = ReadAsync(memoryStream);
retval.Wait();
}
static void Read(MemoryStream memoryStream)
{
memoryStream.Position = 0;
using (DeflateStream deflateStream = new DeflateStream(memoryStream, CompressionMode.Decompress, true))
{
byte[] buffer = new byte[uncompressedSize];
int bytesRead = -1;
int i = 0;
while (bytesRead > 0 || bytesRead == -1)
{
bytesRead = deflateStream.Read(buffer, 0, uncompressedSize);
System.Diagnostics.Debug.WriteLine("Read #{0} - {1} bytes read", i, bytesRead);
System.Diagnostics.Debug.Assert(bytesRead == 0 || bytesRead == uncompressedSize);
i++;
}
}
}
static async Task ReadAsync(MemoryStream memoryStream)
{
memoryStream.Position = 0;
using (DeflateStream deflateStream = new DeflateStream(memoryStream, CompressionMode.Decompress, true))
{
byte[] buffer = new byte[uncompressedSize];
int bytesRead = -1;
int i = 0;
while (bytesRead > 0 || bytesRead == -1)
{
bytesRead = await deflateStream.ReadAsync(buffer, 0, uncompressedSize);
System.Diagnostics.Debug.WriteLine("ReadAsync #{0} - {1} bytes read", i, bytesRead);
System.Diagnostics.Debug.Assert(bytesRead == 0 || bytesRead == uncompressedSize);
i++;
}
}
}
/// <summary>
/// This is one of the strings of data that was causing issues.
/// </summary>
/// <returns></returns>
static string CreateProblemDataString()
{
StringBuilder sb = new StringBuilder();
sb.Append("0601051081 ");
sb.Append(" ");
sb.Append(" 225021 0300420");
sb.Append("34056064070072076361102 13115016017");
sb.Append("5 192 230237260250 2722");
sb.Append("73280296 326329332 34535535");
sb.Append("7 3 ");
sb.Append(" 4");
sb.Append(" ");
sb.Append(" 50");
sb.Append("6020009 030034045 063071076 360102 13");
sb.Append("1152176160170 208206 23023726025825027227328");
sb.Append("2283285 320321333335341355357 622005009 0");
sb.Append("34053 060070 361096 130151176174178172208");
sb.Append("210198 235237257258256275276280290293 3293");
sb.Append("30334 344348350 ");
sb.Append(" ");
sb.Append(" ");
sb.Append(" ");
sb.Append(" 225020012014 046042044034061");
sb.Append("075078 361098 131152176160170 208195210 230");
sb.Append("231260257258271272283306 331332336 3443483");
sb.Append("54 29 ");
sb.Append(" ");
sb.Append(" 2");
sb.Append("5 29 06 0");
sb.Append("1 178 17");
sb.Append("4 205 2");
sb.Append("05 195 2");
sb.Append("31 231 23");
sb.Append("7 01 01 0");
sb.Append("2 260 26");
sb.Append("2 274 2");
sb.Append("72 274 01 01 0");
sb.Append("3 1 5 3 6 43 52 ");
return sb.ToString();
}
}
UPDATED CODE TO READ STREAMS INTO BUFFER CORRECTLY
Output now looks like this:
...
ReadAsync #410 - 2055 bytes read
ReadAsync #411 - 2055 bytes read
ReadAsync PARTIAL #412 - 453 bytes read, offset for next read = 453
ReadAsync #412 - 1602 bytes read
ReadAsync #413 - 2055 bytes read
...
static void Read(MemoryStream memoryStream)
{
memoryStream.Position = 0;
using (DeflateStream deflateStream = new DeflateStream(memoryStream, CompressionMode.Decompress, true))
{
byte[] buffer = new byte[uncompressedSize]; // buffer to hold known fixed size record.
int bytesRead; // number of bytes read from Read operation
int offset = 0; // offset for writing into buffer
int i = -1; // counter to track iteration #
while ((bytesRead = deflateStream.Read(buffer, offset, uncompressedSize - offset)) > 0)
{
offset += bytesRead; // offset in buffer for results of next reading
System.Diagnostics.Debug.Assert(offset <= uncompressedSize, "should never happen - because would mean more bytes read than requested.");
if (offset == uncompressedSize) // buffer full, complete fixed size record in buffer.
{
offset = 0; // buffer is now filled, next read to start at beginning of buffer again.
i++; // increment counter that tracks iteration #
System.Diagnostics.Debug.WriteLine("Read #{0} - {1} bytes read", i, bytesRead);
}
else // buffer still not full
{
System.Diagnostics.Debug.WriteLine("Read PARTIAL #{0} - {1} bytes read, offset for next read = {2}", i+1, bytesRead, offset);
}
}
}
}
static async Task ReadAsync(MemoryStream memoryStream)
{
memoryStream.Position = 0;
using (DeflateStream deflateStream = new DeflateStream(memoryStream, CompressionMode.Decompress, true))
{
byte[] buffer = new byte[uncompressedSize]; // buffer to hold known fixed size record.
int bytesRead; // number of bytes read from Read operation
int offset = 0; // offset for writing into buffer
int i = -1; // counter to track iteration #
while ((bytesRead = await deflateStream.ReadAsync(buffer, offset, uncompressedSize - offset)) > 0)
{
offset += bytesRead; // offset in buffer for results of next reading
System.Diagnostics.Debug.Assert(offset <= uncompressedSize, "should never happen - because would mean more bytes read than requested.");
if (offset == uncompressedSize) // buffer full, complete fixed size record in buffer.
{
offset = 0; // buffer is now filled, next read to start at beginning of buffer again.
i++; // increment counter that tracks iteration #
System.Diagnostics.Debug.WriteLine("ReadAsync #{0} - {1} bytes read", i, bytesRead);
}
else // buffer still not full
{
System.Diagnostics.Debug.WriteLine("ReadAsync PARTIAL #{0} - {1} bytes read, offset for next read = {2}", i+1, bytesRead, offset);
}
}
}
}
Damien's comments are exactly correct. But, your mistake is a common enough one and IMHO the question deserves an actual answer, if for no other reason than to help others who make the same mistake more easily find the answer to the question.
So, to be clear:
As is true for all of the stream-oriented I/O methods in .NET where one provides a byte[] buffer and the number of bytes read is returned by the method, the only assumptions you can make about the number of bytes are:
The number will not be larger than the maximum number of bytes you asked to read (i.e. passed to the method as the count of bytes to read)
The number will be non-negative, and will be greater than 0 as long as there were in fact data remaining to be read (0 will be returned when you reach the end of the stream).
When reading using any of these methods, you cannot even count on the same method always returning the same number of bytes (depending on context…obviously in some cases, this is in fact deterministic, but you should still not rely on that), and there is no guarantee of any sort that different methods, even those which are reading from the same source, will always return the same number of bytes as some other method.
It is up to the caller to read the bytes as a stream, taking into account the return value specifying the number of bytes read for each call, and reassembling those bytes in whatever manner is appropriate for that particular stream of bytes.
Note that when dealing with Stream objects, you can use the Stream.CopyTo() method. Of course, it only copies to another Stream object. But in many cases, the destination object can be used without treating it as a Stream. E.g. you just want to write the data as a file, or you want to copy it to a MemoryStream and then use the MemoryStream.ToArray() method to turn that into an array of bytes (which you can then access without any concern about how many bytes have been read in a given read operation…by the time you get to the array, all of them have been read :) ).
I'm trying to send a long string from a python server to a C# client. The string is 230400 bytes long. I'm both sending and receiving in chunks of 64 bytes. Server code:
import socket
def initialize():
global s
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 1719))
s.listen()
initialize()
while(1):
sock, addr = s.accept()
msgstr = generate_msg_string() # irrelevant
msglen = len(msgstr)
totalsent = 0
while totalsent < msglen:
sent = sock.send(msgstr[totalsent:totalsent+64])
totalsent = totasent + sent
sock.close()
Client code:
TcpClient tcpClient = new TcpClient();
tcpClient.Connect(ip, 1719);
byte[] ba = new byte[230400];
byte[] buffer = new byte[64];
tcpClient.ReceiveBufferSize = 64
int i=0;
while(i != 230400)
{
stream.Read(buffer, 0, 64);
buffer.CopyTo(ba, i);
i += 64;
}
tcpClient.Close();
I've checked a few connections in a row - the first 1523 bytes are correct and all the rest are gibberish - at least seemingly random.
Any idea what might be the cause?
while(i != 230400)
{
stream.Read(buffer, 0, 64);
buffer.CopyTo(ba, i);
i += 64;
}
The fundamental error here is assuming that Read read 64 bytes. It can read any of:
0 if the socket gets closed for any reason
64 bytes if that happens to be available and it chooses to
1-63 bytes, just for fun
You are not guaranteed anything other than "non-positive if the stream is closed, else at least 1 byte and no more than 64 bytes"
You must must must read the return value of Read and only process that much of the buffer. This remains the case if you switch to Socket.Receive, by the way.
Also - why don't you just fill ba in the first place, increment in the offset and decrementing the count each time?
int count = 230400, offset = 0, read;
byte[] ba = new byte[count];
while(count > 0 && (read=stream.Read(ba, offset, count)) > 0)
{
offset += read;
count -= read;
}
if(read!=0) throw new EndOfStreamException();
It seems that I hurried with the question.
Changing TcpClient to Socket fixed the problem. The approach remained the same.
I have a client-server application where the server transmits a 4-byte integer specifying how large the next transmission is going to be. When I read the 4-byte integer on the client side (specifying FILE_SIZE), the next time I read the stream I get FILE_SIZE + 4 bytes read.
Do I need to specify the offset to 4 when reading from this stream, or is there a way to automatically advance the NetworkStream so my offset can always be 0?
SERVER
NetworkStream theStream = theClient.getStream();
//...
//Calculate file size with FileInfo and put into byte[] size
//...
theStream.Write(size, 0, size.Length);
theStream.Flush();
CLIENT
NetworkStream theStream = theClient.getStream();
//read size
byte[] size = new byte[4];
int bytesRead = theStream.Read(size, 0, 4);
...
//read content
byte[] content = new byte[4096];
bytesRead = theStream.Read(content, 0, 4096);
Console.WriteLine(bytesRead); // <-- Prints filesize + 4
Right; found it; FileInfo.Length is a long; your call to:
binWrite.Write(fileInfo.Length);
writes 8 bytes, little-endian. You then read that back via:
filesize = binRead.ReadInt32();
which little-endian will give you the same value (for 32 bits, at least). You have 4 00 bytes left unused in the stream, though (from the high-bytes of the long) - hence the 4 byte mismatch.
Use one of:
binWrite.Write((int)fileInfo.Length);
filesize = binRead.ReadInt64();
NetworkStream certainly advances, but in both cases, your read is unreliable; a classic "read known amount of content" would be:
static void ReadAll(Stream source, byte[] buffer, int bytes) {
if(bytes > buffer.Length) throw new ArgumentOutOfRangeException("bytes");
int bytesRead, offset = 0;
while(bytes > 0 && (bytesRead = source.Reader(buffer, offset, bytes)) > 0) {
offset += bytesRead;
bytes -= bytesRead;
}
if(bytes != 0) throw new EndOfStreamException();
}
with:
ReadAll(theStream, size, 4);
...
ReadAll(theStream, content, contentLength);
note also that you need to be careful with endianness when parsing the length-prefix.
I suspect you simply aren't reading the complete data.