Currently in the process of writing some TCP socket code and running into a small issue.
Basically where I am confused is the following couple of lines of code.
NetworkStream clientStream = tcpClient.GetStream();
List<Byte> fullMessage = new List<Byte>();
Byte[] message = new Byte[4096];
Byte[] currentMessage = new Byte[4096];
Int32 bytesRead = 0;
if (clientStream.CanRead)
{
do
{
bytesRead = clientStream.Read(message, 0, 4096);
Array.Resize<Byte>(ref currentMessage, bytesRead);
Array.Copy(message, currentMessage, bytesRead);
fullMessage.AddRange(currentMessage);
} while (clientStream.DataAvailable);
}
Specifically regarding the best way to handle the fact even though the message byte array is declared at 4096 bytes the amount of data retrieved is arbitrary and cannot be computed.
So is the way I am handling the response considered a reasonable solution or is there a better way? (IE: Creating a new sized array based on the bytesRead value)
Use a memory stream:
NetworkStream clientStream = tcpClient.GetStream();
MemoryStream messageStream = new MemoryStream();
byte[] inbuffer = new byte[65535];
if (clientStream.CanRead)
{
do
{
var bytesRead = clientStream.Read(inbuffer, 0, buffer.Length);
messageStream.Write(inbuffer, 0, bytesRead);
} while (clientStream.DataAvailable);
}
messageStream.Position = 0;
var completeMessage = new byte[messageStream.Length];
messageStream.Write(completeMessage, 0, messageStream.Length);
If you know that the message won't exceed 4096 bytes, then you can write something like this:
int totalBytesRead = 0;
int bytesRead;
while ((bytesRead = clientStream.Read(message, totalBytesRead, message.Length - totalBytesRead)) !=0)
{
totalBytesRead += bytesRead;
}
totalBytesRead is used to tell clientStream.Read where to put the data that it copies.
clientStream.Read returns 0 if there is no data available.
Note that with this setup, you can't read more than message.Length bytes. If your packets can be larger, then I suggest making your buffer bigger. I wouldn't recommend continually resizing the array because that will end up fragmenting the large object heap (if messages become larger than 80 KB), and at some point you have to set a maximum on the size a message can be--even if you handle "arbitrarily large" messages.
Related
private byte[] ResponseAsync(byte[] buffer, string ip, int port)
{
byte[] buffer_ = new byte[10000]; // receiving buffer 10KB
TcpClient client = new TcpClient(ip, port);
NetworkStream stream = client.GetStream();
stream.Write(buffer, 0, buffer.Length);
//await write;
int i = 0;
while (stream.DataAvailable)
{
MessageBox.Show(i.ToString());
i = stream.Read(buffer_, 0, 1024);
}
return buffer_.Take(i).ToArray();
}
the code was async but I thought I was doing somethign wrong so made it synchrone
You're overwriting the beginning of your buffer each time you do a read and will only return the data read in the last iteration of the while loop.
As such you'll need to increment i with the amount of data read and then use that as the offset when copying data in to your buffer.
private byte[] ResponseAsync(byte[] buffer, string ip, int port)
{
byte[] buffer_ = new byte[10000]; // receiving buffer 10KB
TcpClient client = new TcpClient(ip, port);
NetworkStream stream = client.GetStream();
stream.Write(buffer, 0, buffer.Length);
//await write;
int i = 0;
while (stream.DataAvailable)
{
MessageBox.Show(i.ToString());
// write data to the appropriate point in buffer_ and update i
i += stream.Read(buffer_, i, 1024);
}
return buffer_.Take(i).ToArray();
}
Be aware though that if you receive more than 10,000 bytes this will throw an exception.
The chances are that you'll not read all the data from the stream as stream.DataAvailable will only say whether there's data available to be read not that the stream has finished.
Using a blocking, streaming .NET socket I'm connecting to a server. Whenever I'm reading small bits of data, all goes well and the data is received into my buffer:
using (var socket = new Socket(SocketType.Stream, ProtocolType.IP))
{
socket.Connect(IPAddress.Parse("127.0.0.1"), 5000);
byte[] buffer = new byte[BufferSize];
socket.Receive(buffer);
// Here buffer doesn't always contain all data the server sent me?
Console.WriteLine(Encoding.Default.GetString(buffer));
}
In some cases though, I'm not receiving everything the server sends me. Data seems to be chopped off. What can be the cause of this?
This is documented in the Receive() method, emphasis mine:
The Receive method reads data into the buffer parameter and returns the number of bytes successfully read. You can call Receive from both connection-oriented and connectionless sockets.
When ignoring the return value, you will not know what portion of your buffer actually contains relevant data. Depending on the protocol being used, you may or may not know the content length in advance. Some protocols provide this length, others close the connection when done, yet another can use a message boundary.
You'll have to hold the received data in another buffer, and return or output the entire message buffer when no more data is available or expected. This can be done like this:
int BufferSize = 1024;
using (var socket = new Socket(SocketType.Stream, ProtocolType.IP))
{
socket.Connect(IPAddress.Parse("127.0.0.1"), 5000);
byte[] buffer = new byte[BufferSize];
string message = "";
int bytesReceived;
do
{
bytesReceived = socket.Receive(buffer);
message += Encoding.ASCII.GetString(buffer, 0, bytesReceived);
} while (bytesReceived > 0);
Console.WriteLine(message);
}
The received bytes are ASCII characters (as defined in the made-up protocol), so each byte received indicates one character (you can't convert partially received multibyte unicode characters). The bytes are converted to a string and appended to the message variable. The code loops until the server closes the connection.
When the message size is known on beforehand, again, depending on the protocol being used, you can create a message buffer and copy the data there on each Receive():
// Received using another Receive() call
int messageSize = 1234;
int totalBytesReceived = 0;
byte[] messageBuffer = new byte[messageSize];
byte[] buffer = new byte[BufferSize];
int bytesReceived;
do
{
bytesReceived = socket.Receive(buffer);
// Copy the receive buffer into the message buffer, appending after
// previously received data (totalBytesReceived).
Buffer.BlockCopy(buffer, 0, messageBuffer, totalBytesReceived, bytesReceived);
totalBytesReceived += bytesReceived;
} while (bytesReceived > 0);
// This assumes the connection wasn't closed prematurely.
Console.WriteLine(Encoding.ASCII.GetString(messageBuffer));
This can in turn be put in a resuable method:
public byte[] ReceiveMessage(Socket socket, int messageSize)
{
byte[] messageBuffer = new byte[messageSize];
int bytesReceived = 0;
int totalBytesReceived = 0;
do
{
byte[] buffer = new byte[BufferSize];
// Receive at most the requested number of bytes, or the amount the
// buffer can hold, whichever is smaller.
int toReceive = Math.Min(messageSize - totalBytesReceived, BufferSize);
bytesReceived = socket.Receive(buffer, toReceive, SocketFlags.None);
// Copy the receive buffer into the message buffer, appending after
// previously received data (totalBytesReceived).
Buffer.BlockCopy(buffer, 0, messageBuffer, totalBytesReceived, bytesReceived);
totalBytesReceived += bytesReceived;
} while (bytesReceived > 0);
if (totalBytesReceived < messageSize)
{
throw new Exception("Server closed connection prematurely");
}
return messageBuffer;
}
There's the NetworkStream which wraps a socket, but it has the same reading issues as the socket itself. You will have to monitor the return value and keep calling Read() until you've received all bytes. Same goes for the TcpClient that has a GetStream() method, on whose return value you'll also have to continue reading until you've read all data.
I've been struggling with this and can't find a reason why my code is failing to properly read from a TCP server I've also written. I'm using the TcpClient class and its GetStream() method but something is not working as expected. Either the operation blocks indefinitely (the last read operation doesn't timeout as expected), or the data is cropped (for some reason a Read operation returns 0 and exits the loop, perhaps the server is not responding fast enough). These are three attempts at implementing this function:
// this will break from the loop without getting the entire 4804 bytes from the server
string SendCmd(string cmd, string ip, int port)
{
var client = new TcpClient(ip, port);
var data = Encoding.GetEncoding(1252).GetBytes(cmd);
var stm = client.GetStream();
stm.Write(data, 0, data.Length);
byte[] resp = new byte[2048];
var memStream = new MemoryStream();
int bytes = stm.Read(resp, 0, resp.Length);
while (bytes > 0)
{
memStream.Write(resp, 0, bytes);
bytes = 0;
if (stm.DataAvailable)
bytes = stm.Read(resp, 0, resp.Length);
}
return Encoding.GetEncoding(1252).GetString(memStream.ToArray());
}
// this will block forever. It reads everything but freezes when data is exhausted
string SendCmd(string cmd, string ip, int port)
{
var client = new TcpClient(ip, port);
var data = Encoding.GetEncoding(1252).GetBytes(cmd);
var stm = client.GetStream();
stm.Write(data, 0, data.Length);
byte[] resp = new byte[2048];
var memStream = new MemoryStream();
int bytes = stm.Read(resp, 0, resp.Length);
while (bytes > 0)
{
memStream.Write(resp, 0, bytes);
bytes = stm.Read(resp, 0, resp.Length);
}
return Encoding.GetEncoding(1252).GetString(memStream.ToArray());
}
// inserting a sleep inside the loop will make everything work perfectly
string SendCmd(string cmd, string ip, int port)
{
var client = new TcpClient(ip, port);
var data = Encoding.GetEncoding(1252).GetBytes(cmd);
var stm = client.GetStream();
stm.Write(data, 0, data.Length);
byte[] resp = new byte[2048];
var memStream = new MemoryStream();
int bytes = stm.Read(resp, 0, resp.Length);
while (bytes > 0)
{
memStream.Write(resp, 0, bytes);
Thread.Sleep(20);
bytes = 0;
if (stm.DataAvailable)
bytes = stm.Read(resp, 0, resp.Length);
}
return Encoding.GetEncoding(1252).GetString(memStream.ToArray());
}
The last one "works", but it certainly looks ugly to put a hard-coded sleep inside the loop considering that sockets already support read timeouts! Do I need to setup some property(ies) on the TcpClient of the NetworkStream? Does the problem resides in the server? The server don't close the connections, it is up to the client to do so. The above is also running inside the UI thread context (test program), maybe it has something to do with that...
Does someone know how to properly use NetworkStream.Read to read data until no more data is available? I guess what I'm wishing for is something like the old Win32 winsock timeout properties... ReadTimeout, etc. It tries to read until the timeout is reached, and then return 0... But it sometimes seem to return 0 when data should be available (or on the way.. can Read return 0 if is available?) and it then blocks indefinitely on the last read when data is not available...
Yes, I'm at a loss!
Networking code is notoriously difficult to write, test and debug.
You often have lots of things to consider such as:
what "endian" will you use for the data that is exchanged (Intel x86/x64 is based on little-endian) - systems that use big-endian can still read data that is in little-endian (and vice versa), but they have to rearrange the data. When documenting your "protocol" just make it clear which one you are using.
are there any "settings" that have been set on the sockets which can affect how the "stream" behaves (e.g. SO_LINGER) - you might need to turn certain ones on or off if your code is very sensitive
how does congestion in the real world which causes delays in the stream affect your reading/writing logic
If the "message" being exchanged between a client and server (in either direction) can vary in size then often you need to use a strategy in order for that "message" to be exchanged in a reliable manner (aka Protocol).
Here are several different ways to handle the exchange:
have the message size encoded in a header that precedes the data - this could simply be a "number" in the first 2/4/8 bytes sent (dependent on your max message size), or could be a more exotic "header"
use a special "end of message" marker (sentinel), with the real data encoded/escaped if there is the possibility of real data being confused with an "end of marker"
use a timeout....i.e. a certain period of receiving no bytes means there is no more data for the message - however, this can be error prone with short timeouts, which can easily be hit on congested streams.
have a "command" and "data" channel on separate "connections"....this is the approach the FTP protocol uses (the advantage is clear separation of data from commands...at the expense of a 2nd connection)
Each approach has its pros and cons for "correctness".
The code below uses the "timeout" method, as that seems to be the one you want.
See http://msdn.microsoft.com/en-us/library/bk6w7hs8.aspx. You can get access to the NetworkStream on the TCPClient so you can change the ReadTimeout.
string SendCmd(string cmd, string ip, int port)
{
var client = new TcpClient(ip, port);
var data = Encoding.GetEncoding(1252).GetBytes(cmd);
var stm = client.GetStream();
// Set a 250 millisecond timeout for reading (instead of Infinite the default)
stm.ReadTimeout = 250;
stm.Write(data, 0, data.Length);
byte[] resp = new byte[2048];
var memStream = new MemoryStream();
int bytesread = stm.Read(resp, 0, resp.Length);
while (bytesread > 0)
{
memStream.Write(resp, 0, bytesread);
bytesread = stm.Read(resp, 0, resp.Length);
}
return Encoding.GetEncoding(1252).GetString(memStream.ToArray());
}
As a footnote for other variations on this writing network code...when doing a Read where you want to avoid a "block", you can check the DataAvailable flag and then ONLY read what is in the buffer checking the .Length property e.g. stm.Read(resp, 0, stm.Length);
Setting the underlying socket ReceiveTimeout property did the trick. You can access it like this: yourTcpClient.Client.ReceiveTimeout. You can read the docs for more information.
Now the code will only "sleep" as long as needed for some data to arrive in the socket, or it will raise an exception if no data arrives, at the beginning of a read operation, for more than 20ms. I can tweak this timeout if needed. Now I'm not paying the 20ms price in every iteration, I'm only paying it at the last read operation. Since I have the content-length of the message in the first bytes read from the server I can use it to tweak it even more and not try to read if all expected data has been already received.
I find using ReceiveTimeout much easier than implementing asynchronous read... Here is the working code:
string SendCmd(string cmd, string ip, int port)
{
var client = new TcpClient(ip, port);
var data = Encoding.GetEncoding(1252).GetBytes(cmd);
var stm = client.GetStream();
stm.Write(data, 0, data.Length);
byte[] resp = new byte[2048];
var memStream = new MemoryStream();
var bytes = 0;
client.Client.ReceiveTimeout = 20;
do
{
try
{
bytes = stm.Read(resp, 0, resp.Length);
memStream.Write(resp, 0, bytes);
}
catch (IOException ex)
{
// if the ReceiveTimeout is reached an IOException will be raised...
// with an InnerException of type SocketException and ErrorCode 10060
var socketExept = ex.InnerException as SocketException;
if (socketExept == null || socketExept.ErrorCode != 10060)
// if it's not the "expected" exception, let's not hide the error
throw ex;
// if it is the receive timeout, then reading ended
bytes = 0;
}
} while (bytes > 0);
return Encoding.GetEncoding(1252).GetString(memStream.ToArray());
}
As per your requirement, Thread.Sleep is perfectly fine to use because you are not sure when the data will be available so you might need to wait for the data to become available. I have slightly changed the logic of your function this might help you little further.
string SendCmd(string cmd, string ip, int port)
{
var client = new TcpClient(ip, port);
var data = Encoding.GetEncoding(1252).GetBytes(cmd);
var stm = client.GetStream();
stm.Write(data, 0, data.Length);
byte[] resp = new byte[2048];
var memStream = new MemoryStream();
int bytes = 0;
do
{
bytes = 0;
while (!stm.DataAvailable)
Thread.Sleep(20); // some delay
bytes = stm.Read(resp, 0, resp.Length);
memStream.Write(resp, 0, bytes);
}
while (bytes > 0);
return Encoding.GetEncoding(1252).GetString(memStream.ToArray());
}
Hope this helps!
I've been trying to send a file from a client to a server application using the TCPClient class in C#. Before I send the actual data, I send some additional information like the exact file size and the file name, so the server application knows how much there is to read. The funny thing is everything was fine when I tested it on 127.0.0.1 - as soon as I replaced the IP address with the actual one, the server could only read about 1,5 KByte of the data that was sent. It still gets the filename and the file size, but theres no way it's retrieving the actual data.
For testing purposes, I replaced the image I was going to send with a simple string and the transmission went alright, so I suppose there is a problem with sending and receiving the data chunks, but I'm not getting any exceptions on the client side either.
Anyone got an idea? Cheers!
Edit:
Thanks so far, this is what I have got codewise. For the client:
IPAddress ipAddress = IPAddress.Parse("xx.xx.xx.xx");
int port = 3003;
int bufferSize = 1024;
TcpClient client = new TcpClient();
NetworkStream netStream;
// Connect to server
try
{
client.Connect(new IPEndPoint(ipAddress, port));
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
netStream = client.GetStream();
// Read bytes from image
byte[] data = File.ReadAllBytes("C:\\Users\\Dan\\Desktop\\asdf.jpg");
// Build the package
byte[] dataLength = BitConverter.GetBytes(data.Length);
byte[] package = new byte[4 + data.Length];
dataLength.CopyTo(package, 0);
data.CopyTo(package, 4);
// Send to server
int bytesSent = 0;
int bytesLeft = package.Length;
while (bytesLeft > 0)
{
int nextPacketSize = (bytesLeft > bufferSize) ? bufferSize : bytesLeft;
netStream.Write(package, bytesSent, nextPacketSize);
bytesSent += nextPacketSize;
bytesLeft -= nextPacketSize;
}
// Clean up
netStream.Close();
client.Close();
And the server:
TcpListener listen = new TcpListener(3003);
TcpClient client;
int bufferSize = 1024;
NetworkStream netStream;
int bytesRead = 0;
int allBytesRead = 0;
// Start listening
listen.Start();
// Accept client
client = listen.AcceptTcpClient();
netStream = client.GetStream();
// Read length of incoming data
byte[] length = new byte[4];
bytesRead = netStream.Read(length, 0, 4);
int dataLength = BitConverter.ToInt32(length,0);
// Read the data
int bytesLeft = dataLength;
byte[] data = new byte[dataLength];
while (bytesLeft > 0)
{
int nextPacketSize = (bytesLeft > bufferSize) ? bufferSize : bytesLeft;
bytesRead = netStream.Read(data, allBytesRead, nextPacketSize);
allBytesRead += bytesRead;
bytesLeft -= bytesRead;
}
// Save image to desktop
File.WriteAllBytes("C:\\Users\\Dan\\Desktop\\tcpimage.jpg", data);
// Clean up
netStream.Close();
client.Close();
About 1.5 KiB sounds like 1500 bytes, "the largest allowed by Ethernet at the network layer". This is the maximum transmission unit (mtu) forcing your network stack to split your file into several small packets.
You need to call the NetworkStream.Read in a loop to read every packet arrived. There's example code of this at MSDN.
Combine this with the default behavior of .NET; consolidating smaller packets to reduce the amount of packets sent, and you'll also see this behavior when sending smaller packets. This can be controlled with ServicePointManager.UseNagleAlgorithm or by using smaller scoped socket options.
Ok, don't know what I'm doing here, but in case of anyone uses this as referrence.
I got rid of unnecessary copying and done a very important improvement. Your way to calc nextPacketSize is not complete, since there could be less data, than 1024 avaliable, and you'll get extra nulls in between these chunks (I had a lot of headache with that, now so happy to figure out)
I made them as function, needed for several files, so here's code:
client
this one is almost the same
public void sendData(byte[] data, NetworkStream stream)
{
int bufferSize = 1024;
byte[] dataLength = BitConverter.GetBytes(data.Length);
stream.Write(dataLength, 0, 4);
int bytesSent = 0;
int bytesLeft = data.Length;
while (bytesLeft > 0)
{
int curDataSize = Math.Min(bufferSize, bytesLeft);
stream.Write(data, bytesSent, curDataSize);
bytesSent += curDataSize;
bytesLeft -= curDataSize;
}
}
server
public byte[] getData(TcpClient client)
{
NetworkStream stream = client.GetStream();
byte[] fileSizeBytes = new byte[4];
int bytes = stream.Read(fileSizeBytes, 0, 4);
int dataLength = BitConverter.ToInt32(fileSizeBytes, 0);
int bytesLeft = dataLength;
byte[] data = new byte[dataLength];
int bufferSize = 1024;
int bytesRead = 0;
while (bytesLeft > 0)
{
int curDataSize = Math.Min(bufferSize, bytesLeft);
if (client.Available < curDataSize)
curDataSize = client.Available; //This saved me
bytes = stream.Read(data, bytesRead, curDataSize);
bytesRead += curDataSize;
bytesLeft -= curDataSize;
}
return data;
}
I used some of your code for a test networking project I'm working on. I tweaked a couple things to fit the requirements of my project, but one alteration I needed to make to the original code is to add a "-4" to the line that was setting the bytesLeft variable. After that the code worked. My test file is 52KB & was transmitted successfully.
// Read the data
int bytesLeft = dataLength-4;
byte[] data = new byte[dataLength];
I have the following code:
private string Connect()
{
string responseData;
try
{
TcpClient client = new TcpClient(ServerIp, Port);
client.ReceiveBufferSize = Int32.MaxValue;
Byte[] data = Encoding.GetEncoding(1251).GetBytes(ReadyQuery);
NetworkStream stream = client.GetStream();
// send data
stream.Write(data, 0, data.Length);
// buffer
data = new Byte[65536];
Int32 bytes = stream.Read(data, 0, data.Length);
responseData = Encoding.GetEncoding(1251).GetString(data, 0, bytes);
// close all
stream.Close();
client.Close();
return responseData;
}
I have problem with a big message. The receive message size is 22K chars. I get only part of message.
How can I receive big messages?
PS. In the debugger bytes equal 4096.
You call stream.Read in a loop until you read the entire message. If you know the message size in advance it's relatively easy:
int messageSize = 22000;
int readSoFar = 0;
byte [] msg = new byte[messageSize];
while(readSoFar < messageSize)
{
var read = stream.Read(msg, readSoFar, msg.Length - readSoFar);
readSoFar += read;
if(read==0)
break; // connection was broken
}
If the message size is part of the message (say, encoded in the first 4 bytes), you should read those first and then do as I suggested.