This question already has answers here:
Receiving data in TCP
(10 answers)
Closed 2 years ago.
I have written a simple TCP client and server. The problem lies with the client.
I'm having some trouble reading the entire response from the server. I must let the thread sleep to allow all the data be sent.
I've tried a few times to convert this code into a loop that runs until the server is finished sending data.
// Init & connect to client
TcpClient client = new TcpClient();
Console.WriteLine("Connecting.....");
client.Connect("192.168.1.160", 9988);
// Stream string to server
input += "\n";
Stream stm = client.GetStream();
ASCIIEncoding asen = new ASCIIEncoding();
byte[] ba = asen.GetBytes(input);
stm.Write(ba, 0, ba.Length);
// Read response from server.
byte[] buffer = new byte[1024];
System.Threading.Thread.Sleep(1000); // Huh, why do I need to wait?
int bytesRead = stm.Read(buffer, 0, buffer.Length);
response = Encoding.ASCII.GetString(buffer, 0, bytesRead);
Console.WriteLine("Response String: "+response);
client.Close();
The nature of streams that are built on top of sockets is that you have an open pipeline that transmits and receives data until the socket is closed.
However, because of the nature of client/server interactions, this pipeline isn't always guaranteed to have content on it to be read. The client and server have to agree to send content over the pipeline.
When you take the Stream abstraction in .NET and overlay it on the concept of sockets, the requirement for an agreement between the client and server still applies; you can call Stream.Read all you want, but if the socket that your Stream is connected to on the other side isn't sending content, the call will just wait until there is content.
This is why protocols exist. At their most basic level, they help define what a complete message that is sent between two parties is. Usually, the mechanism is something along the lines of:
A length-prefixed message where the number of bytes to be read is sent before the message
A pattern of characters used to mark the end of a message (this is less common depending on the content that is being sent, the more arbitrary any part of the message can be, the less likely this will be used)
That said you aren't adhering to the above; your call to Stream.Read is just saying "read 1024 bytes" when in reality, there might not be 1024 bytes to be read. If that's the case, the call to Stream.Read will block until that's been populated.
The reason the call to Thread.Sleep probably works is because by the time a second goes by, the Stream has 1024 bytes on it to read and it doesn't block.
Additionally, if you truly want to read 1024 bytes, you can't assume that the call to Stream.Read will populate 1024 bytes of data. The return value for the Stream.Read method tells you how many bytes were actually read. If you need more for your message, then you need to make additional calls to Stream.Read.
Jon Skeet wrote up the exact way to do this if you want a sample.
Try to repeat the
int bytesRead = stm.Read(buffer, 0, buffer.Length);
while bytesRead > 0. It is a common pattern for that as i remember.
Of course don't forget to pass appropriate params for buffer.
You dont know the size of data you will be reading so you have to set a mechanism to decide. One is timeout and another is using delimiters.
On your example you read whatever data from just one iteration(read) because you dont set the timeout for reading and using default value thats "0" milisecond. So you have to sleep just 1000 ms. You get same effect with using recieve time out to 1000 ms.
I think using lenght of data as prefix is not the real solution because when socket is closed by both sides, socket time-wait situation can not handled properly. Same data can be send to server and cause server to get exception . We used prefix-ending character sequence. After every read we check the data for start and end character sequence, if we cant get end characters, we call another read. But of course this works only if you have the control of server side and client side code.
In the TCP Client / Server I just wrote I generate the packet I want to send to a memory stream, then take the length of that stream and use it as a prefix when sending the data. That way the client knows how many bytes of data it's going to need to read for a full packet.
Related
I am writing a Small HttpServer, sometime I encounter a problem with missing POST Data.
By using Wireshark I discovered, that the Header is split into two segments.
I only get the first segment (636 Bytes), the second one (POST Data in this case) gets totally lost.
Here is a the relevant C# Code
string requestHeaderString = "";
StreamSocket socketStream = args.Socket;
IInputStream inputStream = socketStream.InputStream;
byte[] data = new byte[BufferSize];
IBuffer buffer = data.AsBuffer();
try
{
await inputStream.ReadAsync(buffer, BufferSize, InputStreamOptions.Partial);
// This is where things go missing, buffer.ToArray() should be 678 Bytes long,
// so Segment 1 (636 Bytes) and Segment 2 (42 Bytes) combined.
// But is only 636 Bytes long, so just the first Segment?!
requestHeaderString += Encoding.UTF8.GetString(buffer.ToArray());
}
catch (Exception e)
{
Debug.WriteLine("inputStream is not readable" + e.StackTrace);
return;
}
This code is in part of the StreamSocketListener ConnectionReceived Event.
Do I manually have to reassemble the TCP Segments, isn't this what the Systems TCP Stack should do?
Thanks,
David
The problem is the systems TCP stack treats the TCP stream just like any other stream. You don't get "messages" with streams, you just get a stream of bytes.
The receiving side has no way to tell when one "message" ends and where the next begins without you telling it some how. You must implement message framing on top of TCP, then on your receiving side you must repeatedly call Receive till you have received enough bytes to form a full message (this will involve using the int returned from the receive call to see how many bytes where processed).
Important note: If you don't know how many bytes you are expecting to get in total, for example you are doing message framing by using '\0' to seperate messages you may get the end of one message and the start of the next in a single Receive call. You will need to handle that situation.
EDIT: Sorry, I skipped over the fact you where reading HTTP. You must follow the protocol of HTTP. You must read in data till you see the pattern \r\n\r\n, once you get that you must parse the header and decode how much data is in the content portion of the HTTP message then repeatatly call read till you have read the number of bytes needed.
I have a network stream over TCP which is a Zlib stream in both ways (client to server and server to client).
For Zlib, I am using the latest Ionic.Zlib.dll.
For the first couple packets that get sent/received, it works fine, but when receiving the 3rd packet or so, it starts misbehaving.
Every read operation on the ZlibStream blocks until the packet is repeated by the other end.
For example:
// Where reader is a BinaryReader on the ZlibStream on the NetworkStream
int a = reader.ReadInt32(); // blocks until initial packet that is 12 bytes is received
int b = reader.ReadInt32(); // blocks until the packet is repeated
int c = reader.ReadInt32(); // blocks until the packet is repeated again
After that code, the packet has actually been sent 3 times and received only once.
Why is it blocking there? How can I get it to not block and continue as expected?
Could flushing in Zlib have anything to do with it? (The stream is flushed after every packet.)
Zlib, as well as other compression algorithms, operates with blocks of data. It bufferes some data from the source stream and then tries to compress it.
If you put in the ZlibStream data that is less than buffer size, nothing will actually come out of ZlibStream right away. Your data will be sitting there until buffer fills. To get that data compressed, flushing your original stream, and probably even compressed stream, would not be enough.
All your readInts should receive the right data if serve closes the ZlibStream.
If you want to receive packets from server in real-time, try completely compressing the packets first and then send them through the network stream.
First, you say
"Every read operation on the ZlibStream blocks until the packet is
repeated by the other end."
That means that there's a read on your server... E.g. the pattern should be like...
Client: Connects to server, asks for data
Server: Sends packet
Client: Reads packet
Client: Sends same packet back
Server: Reads Packet
Server: Sends new packet?
If that's the case, the server will always block since you're ReadInt32()ing 3 times. Are you positive the other int packets are sent by your server?
Second, I wouldn't use ZlibStream to read directly from a network socket. I'd read in the raw data from the network stream, buffer it, and have another method fire when data was in the buffer that decompressed it...
In psuedocode:
byte[] compressedDataBuffer
bool canReadFromBuffer = false;
ASYNC:
while(true):
get current network stream, see if there's data
if it has data:
add networkStream's current data to compressedDataBuffer
lock:
if compressedDataBuffer has new data/has enough data
canReadFromBuffer = true
else
canReadFromBuffer = false
else:
sleep(0); //yield cycles
SYNC:
while(true):
if canReadFromBuffer:
create a memory stream of compressedDataBuffer
create a zlibstream around that memory stream
perform operations on the zlibstream that are required.
I have an asynchronous read method...
private void read(IAsyncResult ar) {
//Get the Server State Object
ServerState state = (ServerState)ar.AsyncState;
//read from the socket
int readCount = state.socket.EndReceive(ar);
//check if reading is done, move on if so, trigger another read if not
if (readCount > 0) {
//purge the buffer and start another read
state.purgeBuffer();
state.socket.BeginReceive(state.buffer, 0, ServerState.bufferSize, 0, new AsyncCallback(read), state);
}
else {
//all bytes have been read, dispatch the message
dispatch(state);
}
}
The problem that I am having is that read is only 0 if the connection is closed. How do I say, this is the end of that message and pass the data on to the dispatcher, while leaving the socket open to accept new messages.
Thank you!
You should not rely on what is in the TCP buffer. You must process the incoming bytes as a stream somewhere. You can't really know whether its complete. Only one layer above can know when the message completed.
Example:
If you read HTTP responses the HTTP header will contain the byte count which is in the HTTP body. So you know how much to read.
You only know how much to read if the data follows a certain protocol and you interprete it. Imagine you receive a file over the socket. The first thing you would receive is the file size. Without that you would never know how much to read.
You should make your messages fit a particular format so that you can distinguish when they start and when end. Even if it is a stream of data it should be sent in packets.
One option is to send length of message first and then you know how much data to expect. But problem with that is if you loose sync you can never recover and you will never know what is message length and what is its content. It is good to use some special marking sequence to know when message begins. It is is not 100% error proof (sequence might appear in data) but certainly helps and allows to recover from sync loose. This is particularly important when reading from a binary stream like socket.
Even ancient RS232 serial protocol had its frame and stop bit to know when you got all the data.
Can anyone point out the flaw in this code? I'm retrieving some HTML with TcpClient. NetworkStream.Read() never seems to finish when talking to an IIS server. If I go use the Fiddler proxy instead, it works fine, but when talking directly to the target server the .read() loop won't exit until the connection exceptions out with an error like "the remote server has closed the connection".
internal TcpClient Client { get; set; }
/// bunch of other code here...
try
{
NetworkStream ns = Client.GetStream();
StreamWriter sw = new StreamWriter(ns);
sw.Write(request);
sw.Flush();
byte[] buffer = new byte[1024];
int read=0;
try
{
while ((read = ns.Read(buffer, 0, buffer.Length)) > 0)
{
response.AppendFormat("{0}", Encoding.ASCII.GetString(buffer, 0, read));
}
}
catch //(SocketException se)
{
}
finally
{
Close();
}
Update
In the debugger, I can see the entire response coming through immediately and being appended to my StringBuilder (response). It just appears that the connection isn't being closed when the server is done sending the response, or my code isn't detecting it.
Conclusion
As has been said here, it's best to take advantage of the offerings of the protocol (in the case of HTTP, the Content-Length header) to determine when a transaction is complete. However, I've found that not all pages have content-length set. So, I'm now using a hybrid solution:
For ALL transactions, set the request's Connection header to "close", so that the server is discouraged from keeping the socket open. This improves the chances that the server will close the connection when it is through responding to your request.
If Content-Length is set, use it to determine when a request is complete.
Else, set the NetworkStream's RequestTimeout property to a large, but reasonable, value like 1 second. Then, loop on NetworkStream.Read() until either a) the timeout occurs, or b) you read fewer bytes than you asked for.
Thanks to everyone for their excellent and detailed responses.
Contrary to what the documentation for NetworkStream.Read implies, the stream obtained from a TcpClient does not simply return 0 for the number of bytes read when there is no data available - it blocks.
If you look at the documentation for TcpClient, you will see this line:
The TcpClient class provides simple methods for connecting, sending, and receiving stream data over a network in synchronous blocking mode.
Now my guess is that if your Read call is blocking, it's because the server has decided not to send any data back. This is probably because the initial request is not getting through properly.
My first suggestion would be to eliminate the StreamWriter as a possible cause (i.e. buffering/encoding nuances), and write directly to the stream using the NetworkStream.Write method. If that works, make sure that you're using the correct parameters for the StreamWriter.
My second suggestion would be not to depend on the result of a Read call to break the loop. The NetworkStream class has a DataAvailable property that is designed for this. The correct way to write a receive loop is:
NetworkStream netStream = client.GetStream();
int read = 0;
byte[] buffer = new byte[1024];
StringBuilder response = new StringBuilder();
do
{
read = netStream.Read(buffer, 0, buffer.Length);
response.Append(Encoding.ASCII.GetString(buffer, 0, read));
}
while (netStream.DataAvailable);
Read the response until you reach a double CRLF. What you now have is the Response headers.
Parse the headers to read the Content-Length header which will be the count of bytes left in the response.
Here is a regular expression that can catch the Content-Length header.
David's Updated Regex
Content-Length: (?<1>\d+)\r\n
Content-Length
Note
If the server does not properly set this header I would not use it.
Not sure if this is helpful or not but with HTTP 1.1 the underlying connection to the server might not be closed so maybe the stream doesn't get closed either? The idea being that you can reuse the connection to send a new request. I think you have to use the content-length. Alternatively use the WebClient or WebRequest classes instead.
I may be wrong, but it looks like your call to Write is writing (under the hood) to the stream ns (via StreamWriter). Later, you're reading from the same stream (ns). I don't quite understand why are you doing this?
Anyway, you may need to use Seek on the stream, to move to the location where you want to start reading. I'd guess that it seeks to the end after writing. But as I said, I'm not really sure if this is a useful answer!
Two Suggestions...
Have you tried using the DataAvailable property of NetworkStream? It should return true if there is data to be read from the stream.
while (ns.DataAvailable)
{
//Do stuff here
}
Another option would be to change the ReadTimeOut to a low value so you don't end up blocking for a long time. It can be done like this:
ns.ReadTimeOut=100;
I am connecting to my mail server using IMAP and Telnet. Once I am connected I am marking all items in the inbox as read. Some times the inbox will only have a couple of e-mails, sometimes the inbox may have thousands of e-mails. I am storing the response from the server into a Byte array, but the Byte array has a fixed length.
Private client As New TcpClient("owa.company.com", 143)
Private data As [Byte]()
Private stream As NetworkStream = client.GetStream()
.
. some code here generates a response that I want to read
.
data = New [Byte](1024) {}
bytes = stream.Read(data, 0, data.Length)
But the response from the server varies based on how many e-mails are successfully marked as read since I get one line of confirmation for each e-mail processed. There are times where the response may contain only 10-20 lines, other times it will contain thousands of lines. Is there any way for me to be able to get the response from the server in its entirety? I mean it seems like I would have to know when the server was done processing my request, but I'm not sure how to go about accomplishing this.
So to reiterate my question is: How can I check in my program to see when the server is done processing a response?
I believe you can use the NetworkStream's DataAvailable property:
if( stream.CanRead)
{
do{
bytes = stream.Read(data, 0, data.Length);
//append the data read to wherever you want to hold it.
someCollectionHoldingTheFullResponse.Add( data);
} while( stream.DataAvailable);
}
At the end, "someCollectionHoldingTheFullResponse" (memory stream? string? List<byte>? up to your requirements) would hold the full response.
Why not just check the unread mail count? If there are no unread mail, then all have been marked as unread :)
This article has an interesting example of C# code communicating over TCP to a server. It shows how to use a While loop to wait until the server has sent over all data over the wire.
Concentrate on the HandleClientComm() routine, since this some code you may wish to use.