I was wondering if there is any method or property that allow us to see if there are available bytes to read in the stream associated to a BinaryReader (in my case, it is a NetworkStream, since I am performing a TCP communication).
I have checked the documentation and the only method I have seen is PeekChar(), but it only checks if there is a next byte (character), so in case there are many bytes to read, making a while loop to increase a counter may be inneficient.
Regarding the TCP communication, the problem is that the application protocol behind the TCP was not defined by me, and I am just trying to figure out how it works! Of course there will be some "length field" that will give me some clues about the bytes to read, but right know I am just checking how it works and this question came to my mind.
The BinaryReader itself doesn't have a DataAvailable property, but the NetworkStream does.
NetworkStream stream = new NetworkStream(socket);
BinaryReader reader = new BinaryReader(stream);
while (true)
{
if (stream.DataAvailable)
{
// call reader.ReadBytes(...) like you normally would!
}
// do other stuff here!
Thread.Sleep(100);
}
If you don't have a handle to the original NetworkStream at the point where you would call reader.ReadBytes(), you can use the BaseStream property.
while (true)
{
NetworkStream stream = reader.BaseStream as NetworkStream;
if (stream.DataAvailable)
{
// call reader.ReadBytes(...) like you normally would!
}
// do other stuff here!
Thread.Sleep(100);
}
BinaryReader will block until it reads all bytes required. The only exception is if it detects end of stream. But NetworkStream is an open stream and does not have the end of stream condition. So you can either create class with basic readers (ReadInt, ReadDouble, etc) that uses peek, reads byte by byte and does not block; or use another async technology.
Related
This question already has an answer here:
TcpClient's NetworkStream reads incomplete data unless Thread.Sleep(1) is called
(1 answer)
Closed 1 year ago.
So I have read more on this online and haven't found any solution. I know the problem is that my ReadAsync is faster than the connection is sending data. But I don't like to use the Thread.Sleep(1) because maybe 1 is not enough, maybe there is a hiccup in the connection. Anything can happen.
This is my code, sometimes my dataBytes Length == 0. But if I debug and set a breakpoint, there is always data. When I set a Thread.Sleep of 500ms works.
So what is happening is that DataAvailable is sometimes false, while more data is coming.
using (var client = (TcpClient)c)
{
using (NetworkStream stream = client.GetStream())
{
using (MemoryStream memory = new MemoryStream())
{
do
{
byte[] b = new byte[256];
int read = await stream.ReadAsync(b, 0, b.Length);
await memory.WriteAsync(b, 0, read);
} while (stream.DataAvailable && stream.CanRead);
memory.Seek(0, SeekOrigin.Begin);
byte[] dataBytes = memory.ToArray();
}
}
}
To test this connection I use RestSharp to send an HTTP message to the code above.
My question is, how can I fix this in a way I am not depending on a Thread.Sleep.
DataAvailable doesn't tell you whether more data is coming; it only tells you whether data is available in the buffers right now, and can be useful in deciding whether to do a synchronous read vs an asynchronous read. It should not have any place in a while loop that determines the end of the read operation.
If you want to read a socket to the end of the data ever, you need to keep reading until Read returns a non-positive value (assuming you're not doing zero-length reads for async IO reasons, which you aren't). However, most socket work doesn't involve a scenario where you can just read from start to end in one big chunk, and you are instead required to implement "framing" (see the second section here), allowing you to detect individual messages (which are completely different to individual Read calls).
I have an exe which simulates a video stream. I connect to it and very occasionally read the expected data, but usually I get only the first 28 bytes and then 65508 bytes of zeros. Assume the video stream is working correctly.
TcpClient tcpClient = new TcpClient ();
int port = 13000;
myIP = IPAddress.Loopback.ToString();
tcpClient.Connect (myIP, port);
NetworkStream netStream = tcpClient.GetStream ();
byte[] bytes = new byte[tcpClient.ReceiveBufferSize];
netStream.Read (bytes, 0, (int)tcpClient.ReceiveBufferSize);
string dataString = Encoding.ASCII.GetString (bytes);
Console.WriteLine("\ndataString: "+dataString.Substring(0,1000));
Console.WriteLine("\nnumber of bytes read: "+bytes.Length);
tcpClient.Close ();
// Closing the tcpClient instance does not close the network stream.
netStream.Close();
How can I make it so that I get the expected output every time?
TCP represents a (bi-directional) stream of data. You're supposed to keep reading from it in a loop, and parsing the data as you need them. It doesn't have a concept of messages - ten writes on one side can result in a single read on the other side just as easily as one write on one side can result in ten reads on the other side.
The contract you have with TCP is as follows:
If there is data in the receive buffer, Read returns immediately, filling the buffer you provided with as much data as is available, up to the length of the buffer. The number of bytes read is the return value of Read.
If there is no data in the receive buffer, Read will block until there's at least a single byte of data. Then it follows as in the first case.
If the other sides shuts down the socket, Read will return zero.
So to get TCP working, you need a loop. How exactly you form the loop depends on what you're trying to do. If you're really working with data that is logically a stream (e.g. audio data), just keep reading as fast as you can and process whatever data you get as it comes in. If you need to send messages, you need to implement a message protocol. If you need a one-off message, you can just keep reading until Read returns zero.
Your case can be handled with the first approach - keep reading until the stream closes, and push the received data forward. Assuming the data is actually a UTF8 stream of text, the basic receiver would look something like this:
using (var client = new TcpClient())
{
tcpClient.Connect(myIP, port);
var stream = client.GetStream();
var buffer = new byte[4096]; // Adapt the size based on what you want to do with the data
var charBuffer = new char[4096];
var decoder = Encoding.UTF8.GetDecoder();
int bytesRead;
while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) != 0)
{
var expectedChars = decoder.GetCharCount(buffer, 0, bytesRead);
if (charBuffer.Length < expectedChars) charBuffer = new char[expectedChars];
var charCount = decoder.GetChars(buffer, 0, bytesRead, charBuffer, 0);
Console.Write(new string(charBuffer, 0, charCount));
}
client.Close();
}
Note that this does no error handling, so don't use it as is in any production code. You would probably also want to use Async methods if you're expecting more than a few simultaneous connections. It's just to illustrate the basic way one would handle a stream of data being received over TCP.
If you want some more insight into dealing with TCP, I have a few very simple examples at https://github.com/Luaancz/Networking. I haven't found any good tutorials or code samples for C#, so if this isn't enough, you'll probably have to dig deeper into the documentation around sockets, TCP and all that.
Or just use an existing networking library, rather than trying to write your own :) TCP is still very low level.
I'm using the following code to send a file over tcp.
If i send many times the same file consecutively to test if it is robust, i receive the first file correctly and the other messed up.
All messed up files have the same incorrect bytes and if i Sleep(a while) all files are transfered correctly. I noticed I must instantiate a new buffer while reading my file to get everything done right. But i don't get why.
I fear my solution to reinstantiate a buffer could be just hiding another major problem. Any suggestion?
using(var fileStream = new FileStream(file, FileMode.Open, FileAccess.Read))
{
using(var binaryReader = new BinaryReader(fileStream))
{
var _sendingBuffer = new byte[BUFFER_SIZE];
int length = (int)fileStream.Length;
int bytesRead = 0;
//Ensure we reached the end of the stream regardless of encoding
while (binaryReader.BaseStream.Position != binaryReader.BaseStream.Length)
{
bytesRead = binaryReader.Read( _sendingBuffer, 0, _sendingBuffer.Length);
_socket.BeginSend(_sendingBuffer, 0, bytesRead, SocketFlags.None, SendFileCallback, null);
//without this i received some messed up data
_sendingBuffer = new byte[BUFFER_SIZE];
}
}
}
BeginSend is an asynchronous operation. It will only be guaranteed to be started after you call it, it won't be finished immediatly. As long as the socket is sending the passed data, that data must not be mutated.
The end of the operation will be signaled through the AsyncCallback callback parameter.
Your problem is exactly that you mutated the transmit buffer while the transmit was still in progress. Creating a new array for each transmit call fixes this.
Other ways to fix the problem:
Use the blocking Socket.Send function which will block until the whole data was sent and the buffer can be reused. This will also make your error handling much easier, because the error will not show up through the AsyncCallback.
Make your complete program acting asynchronously, e.g. using C#5's async Task and async/await functionalities
Therefore:
First read one part of the file asynchronously.
When the async read finishes send it asynchronously through the socket
When this completes and there is more data to read go back to 1.
I'm trying to understand the use of a buffer in the socket scheme of things. Here is what I understand, and if someone could tell me if I'm correct or give feedback to where I might be misunderstanding that would be great.
Using the Asynchronous Sockets detailed on MSDN example (all below is with reference to that example)
Below describes a state object of which I understand the point of:
// State object for reading client data asynchronously
public class StateObject
{
// Client socket.
public Socket workSocket = null;
// Size of receive buffer.
public const int BufferSize = 1024;
// Receive buffer.
public byte[] buffer = new byte[BufferSize];
// Received data string.
public StringBuilder sb = new StringBuilder();
}
Now it says the buffer is 1024 bytes. When you recieve data, you are only asking for 1024 bytes at a time? And in the async method ReceiveCallback, it only ask for 1024 and then says this may not be all of the data, so now retrieve the next 1024.
So basically, the input stream sent to the socket is read in 1024 byte chunks and you keep reading till you've hit an end point (by you're own definition)?
So basically, the input stream sent to the socket is read in 1024 byte chunks and you keep reading till you've hit an end point (by you're own definition)?
Correct. Since TCP is just a stream of data there is really no way of know the length of each if your messages in the stream (unless you always have a fixed length). hence you need to use some kind of buffer to read from the stream to be able to detect the end of your messages.
Can anyone point out the flaw in this code? I'm retrieving some HTML with TcpClient. NetworkStream.Read() never seems to finish when talking to an IIS server. If I go use the Fiddler proxy instead, it works fine, but when talking directly to the target server the .read() loop won't exit until the connection exceptions out with an error like "the remote server has closed the connection".
internal TcpClient Client { get; set; }
/// bunch of other code here...
try
{
NetworkStream ns = Client.GetStream();
StreamWriter sw = new StreamWriter(ns);
sw.Write(request);
sw.Flush();
byte[] buffer = new byte[1024];
int read=0;
try
{
while ((read = ns.Read(buffer, 0, buffer.Length)) > 0)
{
response.AppendFormat("{0}", Encoding.ASCII.GetString(buffer, 0, read));
}
}
catch //(SocketException se)
{
}
finally
{
Close();
}
Update
In the debugger, I can see the entire response coming through immediately and being appended to my StringBuilder (response). It just appears that the connection isn't being closed when the server is done sending the response, or my code isn't detecting it.
Conclusion
As has been said here, it's best to take advantage of the offerings of the protocol (in the case of HTTP, the Content-Length header) to determine when a transaction is complete. However, I've found that not all pages have content-length set. So, I'm now using a hybrid solution:
For ALL transactions, set the request's Connection header to "close", so that the server is discouraged from keeping the socket open. This improves the chances that the server will close the connection when it is through responding to your request.
If Content-Length is set, use it to determine when a request is complete.
Else, set the NetworkStream's RequestTimeout property to a large, but reasonable, value like 1 second. Then, loop on NetworkStream.Read() until either a) the timeout occurs, or b) you read fewer bytes than you asked for.
Thanks to everyone for their excellent and detailed responses.
Contrary to what the documentation for NetworkStream.Read implies, the stream obtained from a TcpClient does not simply return 0 for the number of bytes read when there is no data available - it blocks.
If you look at the documentation for TcpClient, you will see this line:
The TcpClient class provides simple methods for connecting, sending, and receiving stream data over a network in synchronous blocking mode.
Now my guess is that if your Read call is blocking, it's because the server has decided not to send any data back. This is probably because the initial request is not getting through properly.
My first suggestion would be to eliminate the StreamWriter as a possible cause (i.e. buffering/encoding nuances), and write directly to the stream using the NetworkStream.Write method. If that works, make sure that you're using the correct parameters for the StreamWriter.
My second suggestion would be not to depend on the result of a Read call to break the loop. The NetworkStream class has a DataAvailable property that is designed for this. The correct way to write a receive loop is:
NetworkStream netStream = client.GetStream();
int read = 0;
byte[] buffer = new byte[1024];
StringBuilder response = new StringBuilder();
do
{
read = netStream.Read(buffer, 0, buffer.Length);
response.Append(Encoding.ASCII.GetString(buffer, 0, read));
}
while (netStream.DataAvailable);
Read the response until you reach a double CRLF. What you now have is the Response headers.
Parse the headers to read the Content-Length header which will be the count of bytes left in the response.
Here is a regular expression that can catch the Content-Length header.
David's Updated Regex
Content-Length: (?<1>\d+)\r\n
Content-Length
Note
If the server does not properly set this header I would not use it.
Not sure if this is helpful or not but with HTTP 1.1 the underlying connection to the server might not be closed so maybe the stream doesn't get closed either? The idea being that you can reuse the connection to send a new request. I think you have to use the content-length. Alternatively use the WebClient or WebRequest classes instead.
I may be wrong, but it looks like your call to Write is writing (under the hood) to the stream ns (via StreamWriter). Later, you're reading from the same stream (ns). I don't quite understand why are you doing this?
Anyway, you may need to use Seek on the stream, to move to the location where you want to start reading. I'd guess that it seeks to the end after writing. But as I said, I'm not really sure if this is a useful answer!
Two Suggestions...
Have you tried using the DataAvailable property of NetworkStream? It should return true if there is data to be read from the stream.
while (ns.DataAvailable)
{
//Do stuff here
}
Another option would be to change the ReadTimeOut to a low value so you don't end up blocking for a long time. It can be done like this:
ns.ReadTimeOut=100;