I have an HTTP connection with the server side using System.IO.Stream.Read to read the HTTP request body message.
The problem is that once in a couple of minutes the server gets stuck on the Read statement and doesn't continue until the socket timeout has been reached or the client has closed the connection.
int bytesRead = 0;
while (bytesRead < contentLength)
{
int got = stream.Stream.Read(buffer.Buffer, bytesRead, contentLength - bytesRead);
bytesRead += got;
}
It could happen if the stream did not have the amount of data specified by contentLength variable.
This is not the case because when following the tcp stream with WireShark I see that the whole message body (as specified by contentLength) has reached the server machine.
It happens only in the first time that the while loop has been "used", i.e. only in the first time that the stream didn't have "contentLength" number of bytes to read in one try and the while loop had to be re-entered.
Why does it get stuck and does not continue reading data?
I wonder if the stream is reporting early termination; you should also look at whether Read returned a non-positive number, i.e.
while (bytesRead < contentLength)
{
int got = stream.Stream.Read(
buffer.Buffer, bytesRead, contentLength - bytesRead);
if(got <= 0) throw new EndOfStreamException(string.Format(
"Expected {0} bytes; {1} bytes received", contentLength, bytesRead));
bytesRead += got;
}
Basically, if the stream has closed, every call to Read will return non-positive (probably 0) - so your while loop will become a tight cycle of "read 0, add 0, read 0, add 0, read 0, add 0, read 0, add 0".
As a final point, your approach suggests you are allocating a byte[] based on the incoming content-length header; just a warning: make sure you sanity-check this and limit it to sane values, otherwise a DOS attack is trivial. Also, if possible I would suggest trying to use the streaming API where possible, to avoid having to load it all into memory at once (unless you have limited the incoming size such that this isn't a concern).
Try this implementation, your count (contentLength - bytesRead) is wrong. It should be the buffer size.
byte[] buffer = new byte[bufferSize];
int count;
while ((count = stream.Stream.Read(buffer, 0, buffer.Length)) != 0)
{
// do something with the buffer using count as the end marker
destination.Write(buffer, 0, count);
}
If you just want a byte array from the stream, which is more like what you are attempting:
byte[] buffer = stream.Stream.ToArray()
Or to copy to another buffer:
byte[] data = stream.Stream.ToArray();
Array.CopyTo(data , buffer.Buffer, data.Length)
I've had similar errors only with clients that didn't flush their streams. MSDN doc on System.IO.Stream.Read says: "The implementation will block until at least one byte of data can be read, in the event that no data is available.". So for some reason there is no data available. I think you could set a certain ReadTimeout and stop waiting for more data after a reasonably short time.
A related question has also been posted here: C# NetworkStream.Read oddity. Maybe his solution can also help you.
It seems like it had something to do with the fact that I was using the Read method of both the NetworkStream (that's the call seen in the code above) and the StreamReader that was created by passing the NetworkStream object to it's constructor.
The NetworkStream.Read was used to read the http request message body while the StreamReader.Read was used to read the rest of the request (startline, headers...).
Although the calls happened synchroniously by the same thread, it may have been the cause to the behavior I experienced.
When changing the code to work only with a Socket and perform the reading directly from the socket, it has fixed the problem.
Related
I have an exe which simulates a video stream. I connect to it and very occasionally read the expected data, but usually I get only the first 28 bytes and then 65508 bytes of zeros. Assume the video stream is working correctly.
TcpClient tcpClient = new TcpClient ();
int port = 13000;
myIP = IPAddress.Loopback.ToString();
tcpClient.Connect (myIP, port);
NetworkStream netStream = tcpClient.GetStream ();
byte[] bytes = new byte[tcpClient.ReceiveBufferSize];
netStream.Read (bytes, 0, (int)tcpClient.ReceiveBufferSize);
string dataString = Encoding.ASCII.GetString (bytes);
Console.WriteLine("\ndataString: "+dataString.Substring(0,1000));
Console.WriteLine("\nnumber of bytes read: "+bytes.Length);
tcpClient.Close ();
// Closing the tcpClient instance does not close the network stream.
netStream.Close();
How can I make it so that I get the expected output every time?
TCP represents a (bi-directional) stream of data. You're supposed to keep reading from it in a loop, and parsing the data as you need them. It doesn't have a concept of messages - ten writes on one side can result in a single read on the other side just as easily as one write on one side can result in ten reads on the other side.
The contract you have with TCP is as follows:
If there is data in the receive buffer, Read returns immediately, filling the buffer you provided with as much data as is available, up to the length of the buffer. The number of bytes read is the return value of Read.
If there is no data in the receive buffer, Read will block until there's at least a single byte of data. Then it follows as in the first case.
If the other sides shuts down the socket, Read will return zero.
So to get TCP working, you need a loop. How exactly you form the loop depends on what you're trying to do. If you're really working with data that is logically a stream (e.g. audio data), just keep reading as fast as you can and process whatever data you get as it comes in. If you need to send messages, you need to implement a message protocol. If you need a one-off message, you can just keep reading until Read returns zero.
Your case can be handled with the first approach - keep reading until the stream closes, and push the received data forward. Assuming the data is actually a UTF8 stream of text, the basic receiver would look something like this:
using (var client = new TcpClient())
{
tcpClient.Connect(myIP, port);
var stream = client.GetStream();
var buffer = new byte[4096]; // Adapt the size based on what you want to do with the data
var charBuffer = new char[4096];
var decoder = Encoding.UTF8.GetDecoder();
int bytesRead;
while ((bytesRead = stream.Read(buffer, 0, buffer.Length)) != 0)
{
var expectedChars = decoder.GetCharCount(buffer, 0, bytesRead);
if (charBuffer.Length < expectedChars) charBuffer = new char[expectedChars];
var charCount = decoder.GetChars(buffer, 0, bytesRead, charBuffer, 0);
Console.Write(new string(charBuffer, 0, charCount));
}
client.Close();
}
Note that this does no error handling, so don't use it as is in any production code. You would probably also want to use Async methods if you're expecting more than a few simultaneous connections. It's just to illustrate the basic way one would handle a stream of data being received over TCP.
If you want some more insight into dealing with TCP, I have a few very simple examples at https://github.com/Luaancz/Networking. I haven't found any good tutorials or code samples for C#, so if this isn't enough, you'll probably have to dig deeper into the documentation around sockets, TCP and all that.
Or just use an existing networking library, rather than trying to write your own :) TCP is still very low level.
I am have a pretty weird issue right now with how i'm sending my data over sockets. My read buffer is getting all the data i'm sending through the socket, but for some reason the File.Write or File.WriteAsync methods i'm calling are not writing all the bytes, or just stopping(I think that's the issue) OR it could be something wrong with the sockets and how they're being read and written too.
So here is a snippet of my code, keep in mind, both of these are on the same file but are different threads.
byte[] Buffer = new Byte[1024];
int bytesRead;
while((bytesRead = ReceiverSocket.Receive(Buffer)) > 0)
{
totalBytesRead += bytesRead;
//myDownload.Write(Buffer, 0, bytesRead);
myDownload.WriteAsync(Buffer, 0, bytesRead);
}
The code above is my receive function for my download client. Keep in mind i'm getting the same results regardless of what write method I use.
And I am simply using this:
byte[] fileData = File.ReadAllBytes(fileToSend);
serverSocket.SendFile(fileToSend);
I send a file with filesize: 10560 and actually retrieve all the bytes over the receive loop, yet when I check my file after i'm done it only receives a portion like so:
TotalBytesRead
1024
2048
3072
...
...8192
9216
10240
10560
Which means it reads all the data, though in my program, it's still stuck inside the loop and when i close it to check the filesize, it's size is 8192, so it seems to me it didn't read the next 3 socket streams.
I'm at a loss for this, is there anything i'm doing wrong? Could this be a threading issue i'm not aware of?
I figured out the answer, I was locking on my
while((bytesRead = ReceiverSocket.Receive(Buffer)) > 0)
due to the socket always waiting on a receive, so the SendFile wouldn't necessarily send 0, so it was always stuck in the while loop, but one I closed the while, it finished writing to file.
I have an asynchronous TCP socket server in C# that I've made using the TcpListener/TcpClient wrappers for Socket. I'm pretty new to networking in general so I was unaware of how TCP treats data sent, and how it doesn't preserve the message boundaries. After a little research I think I've come up with a solid solution but I'm wondering if anyone has more advice for me.
Currently the data I'm sending is a byte[] of a serialized class object using protobuf (Google's data exchange library) https://code.google.com/p/protobuf/
After I serialize my data, and before it's sent, I've decided to append a 4 byte Int32 at the beginning of the byte array. My idea is that when the packet is sent to the server, it will parse out the Int32 and then wait until it's received that number of bytes before doing anything with the data, otherwise just calling BeginRead again.
Here is the code it's ran through before I write the data, and it seems to work fine, but I'm open to any performance suggestions:
public byte[] PackByteArrayForSending(byte[] data)
{
// [0-3] Packet Length
// [3-*] original data
// Get Int32 of the packet length
byte[] packetLength = BitConverter.GetBytes(data.Length);
// Allocate a new byte[] destination array the size of the original data length plus the packet length array size
byte[] dest = new byte[packetLength.Length + data.Length];
// Copy the packetLength array to the dest array
Buffer.BlockCopy(packetLength, 0, dest, 0, packetLength.Length);
// Copy the data array to the dest array
Buffer.BlockCopy(data, 0, dest, packetLength.Length, data.Length);
return dest;
}
I'm a little stuck on the server end. I have it reading the packetLength variable by using Buffer.BlockCopy to copy the first 4 bytes and then BitConverter.ToInt32 to read the length I should be getting. I'm not sure if I should constantly read the incoming data into a client-specific Stream object, or just use a while loop. Here's an example of the code I have on the server end so far:
NetworkStream networkStream = client.NetworkStream;
int bytesRead = networkStream.EndRead(ar);
if (bytesRead == 0)
{
Console.WriteLine("Got 0 bytes from {0}, marking as OFFLINE.", client.User.Username);
RemoveClient(client);
}
Console.WriteLine("Received {0} bytes.", bytesRead);
// Allocate a new byte array the size of the data that needs to be deseralized.
byte[] data = new byte[bytesRead];
// Copy the byte array into the toDeserialize buffer
Buffer.BlockCopy(
client.Buffer,
0,
data,
0,
bytesRead);
// Read the first Int32 tp get the packet length and then use the byte[4] to get the packetLength
byte[] packetLengthBytes = new byte[4];
Buffer.BlockCopy(
data,
0,
packetLengthBytes,
0,
packetLengthBytes.Length);
int packetLength = BitConverter.ToInt32(packetLengthBytes, 0);
// Now what do you recommend?
// If not all data is received, call
// networkStream.BeginRead(client.Buffer, 0, client.Buffer.Length, ReadCallback, client);
// and preserve the initial data in the client object
Thanks for your time and advice, I look forward to learning more about this subject.
TCP guarantees that the bytes stuffed into the stream at one end will fall out the other end in the same order and without loss or duplication. Expect nothing more, certainly not support for entities larger than a byte.
Typically I have prepended a header with a message length, type and subtype. There is often a correlation id provided to match requests to responses.
The basic pattern is to get bytes and append them to a buffer. If the data in the buffer is sufficient to contain a message header then extract the message length. If the data in the buffer is sufficient to contain the message then remove the message from the buffer and process it. Repeat with any remaining data until there are no complete messages to process. Depending on your application this may be the point to wait on a read or check the stream for additional data. Some applications may need to keep reading the stream from a separate thread to avoid throttling the sender.
Note that you cannot assume that you have a complete message length field. You may have three bytes left after processing a message and cannot extract an int.
Depending on your messages it may be more efficient to use a circular buffer rather than shuffling any remaining bytes each time a message is processed.
I'm wanting to do a protocol analysis that uses SSL/TLS fortunately I can install my own certificate and the DNS portion won't be an issue. My problem is what do I use to do this. I've considered using paros but it will be more trouble than it's worth. So I thought I could write two C# applications. The first is the pseudo server and the other is the pseudo client. The both have a tcp connection between the two of them that I can then use wireshark on. The problem is, is that I have very little experience with streams. So if anyone could point me to helpful articles or if the code is pretty short a sample would be great. Thank you in advanced.
It's not terribly hard to Read/Write to/from streams, you can't just connect the streams, you'll need to have your own code to do this. Preferably on it's own thread (or worker process or task or whatever threading concept you need).
public void ConnectStreams(Stream inStream, Stream outStream)
{
byte[] buffer = new byte[1024];
int bytesRead = 0;
while((bytesRead = inStream.Read(buffer, 0, 1024)) != 0)
{
outStream.Write(buffer, 0, bytesRead);
outStream.Flush();
}
}
Basically Streams operate on byte arrays. When we run this line:
while((bytesRead = inStream.Read(buffer, 0, 1024)) != 0)
We are basically saying, perform Read on inStream, put the read bytes into buffer, at index 0 (in buffer) and read a max of 1024 bytes.
Then we assign the return value into bytesRead which is the number of ACTUAL bytes read (between 0 and 1024 in this case) and if that is not equal to 0, continue looping.
Then we simply write it back into the outStream with the buffer containing the data, and the number of bytes actually read. We perform a flush to actually force the output out vs. stacking up in an internal buffer.
When the stream reaches the EOF, .Read will return 0, the loop will exit and you can continue on. This is how you "Connect" two streams at the most simple level.
I'm trying to write a simple SNPP (Simple Network Paging Protocol) client using sockets. Everything seems to be working well, except for a small inconsistency between servers.
When I send a command, I need to read the reply, which is usually a single chunk of data. However, Sprint's SNPP server sends replies in two parts. The first chunk of data is the first digit of the status code. The second chunk is the remainder. For example, when I attempt to receive the "220 Gateway ready" reply, it arrives like this:
2
I have to send another empty command to retrieve the rest:
20 Gateway ready
For the moment, I'm using:
byte[] buffer = new byte[256];
socket.Receive(buffer);
How can I make sure that I receive all of the available data after issuing a command without allocating a separate buffer for each chunk of data?
For chunked responses I would recommend you reading data like this:
using (var resultStream = new MemoryStream())
{
const int CHUNK_SIZE = 2 * 1024; // 2KB, could be anything that fits your needs
byte[] buffer = new byte[CHUNK_SIZE];
int bytesReceived;
while ((bytesReceived = socket.Receive(buffer, buffer.Length, SocketFlags.None)) > 0)
{
byte[] actual = new byte[bytesReceived];
Buffer.BlockCopy(buffer, 0, actual, 0, bytesReceived);
resultStream.Write(actual, 0, actual.Length);
}
// Do something with the resultStream, like resultStream.ToArray() ...
}
Try to check Socket.Available property to determine if do you need to call Receive again.
I think I understand your question. This overload of Socket.Receive allows you to pass an integer to specify an offset position to start placing data. If you got 1 byte, on the first call as in your example, you can call this overload with an offset of 1 and use the same buffer.
Socket.Receive returns an integer that is the number of bytes that were received. You could check if this is 1 and call Receive again.
byte[] buffer = new byte[256];
int len = socket.Receive(buffer);
if (len == 1)
socket.Receive(buffer, 1, buffer.Length - 1, SocketFlags.None);
All,
When a web-server is sending data in "chunks" it precedes each chunk with it's length (as a string indicating the hex value).
chunk-size[;chunk-extensions]
chunk-data
eg: to chunk 15 bytes:
F;
123456789ABCDEF
The biggest issue with receiving HTTP data from a Socket is determining how much data to read: if you have received all the available data and call the Recieve method again, then the Recieve method will block until the remote socket sends more data, which for a HTTP/1.0 connection will never happen.
You need to implement a reader that wraps around the socket; it should receive data into a buffer, and provide a "ReadLine" method that reads the contents of the buffer until it reads 13 followed by 10 (CRLF)