Using a blocking, streaming .NET socket I'm connecting to a server. Whenever I'm reading small bits of data, all goes well and the data is received into my buffer:
using (var socket = new Socket(SocketType.Stream, ProtocolType.IP))
{
socket.Connect(IPAddress.Parse("127.0.0.1"), 5000);
byte[] buffer = new byte[BufferSize];
socket.Receive(buffer);
// Here buffer doesn't always contain all data the server sent me?
Console.WriteLine(Encoding.Default.GetString(buffer));
}
In some cases though, I'm not receiving everything the server sends me. Data seems to be chopped off. What can be the cause of this?
This is documented in the Receive() method, emphasis mine:
The Receive method reads data into the buffer parameter and returns the number of bytes successfully read. You can call Receive from both connection-oriented and connectionless sockets.
When ignoring the return value, you will not know what portion of your buffer actually contains relevant data. Depending on the protocol being used, you may or may not know the content length in advance. Some protocols provide this length, others close the connection when done, yet another can use a message boundary.
You'll have to hold the received data in another buffer, and return or output the entire message buffer when no more data is available or expected. This can be done like this:
int BufferSize = 1024;
using (var socket = new Socket(SocketType.Stream, ProtocolType.IP))
{
socket.Connect(IPAddress.Parse("127.0.0.1"), 5000);
byte[] buffer = new byte[BufferSize];
string message = "";
int bytesReceived;
do
{
bytesReceived = socket.Receive(buffer);
message += Encoding.ASCII.GetString(buffer, 0, bytesReceived);
} while (bytesReceived > 0);
Console.WriteLine(message);
}
The received bytes are ASCII characters (as defined in the made-up protocol), so each byte received indicates one character (you can't convert partially received multibyte unicode characters). The bytes are converted to a string and appended to the message variable. The code loops until the server closes the connection.
When the message size is known on beforehand, again, depending on the protocol being used, you can create a message buffer and copy the data there on each Receive():
// Received using another Receive() call
int messageSize = 1234;
int totalBytesReceived = 0;
byte[] messageBuffer = new byte[messageSize];
byte[] buffer = new byte[BufferSize];
int bytesReceived;
do
{
bytesReceived = socket.Receive(buffer);
// Copy the receive buffer into the message buffer, appending after
// previously received data (totalBytesReceived).
Buffer.BlockCopy(buffer, 0, messageBuffer, totalBytesReceived, bytesReceived);
totalBytesReceived += bytesReceived;
} while (bytesReceived > 0);
// This assumes the connection wasn't closed prematurely.
Console.WriteLine(Encoding.ASCII.GetString(messageBuffer));
This can in turn be put in a resuable method:
public byte[] ReceiveMessage(Socket socket, int messageSize)
{
byte[] messageBuffer = new byte[messageSize];
int bytesReceived = 0;
int totalBytesReceived = 0;
do
{
byte[] buffer = new byte[BufferSize];
// Receive at most the requested number of bytes, or the amount the
// buffer can hold, whichever is smaller.
int toReceive = Math.Min(messageSize - totalBytesReceived, BufferSize);
bytesReceived = socket.Receive(buffer, toReceive, SocketFlags.None);
// Copy the receive buffer into the message buffer, appending after
// previously received data (totalBytesReceived).
Buffer.BlockCopy(buffer, 0, messageBuffer, totalBytesReceived, bytesReceived);
totalBytesReceived += bytesReceived;
} while (bytesReceived > 0);
if (totalBytesReceived < messageSize)
{
throw new Exception("Server closed connection prematurely");
}
return messageBuffer;
}
There's the NetworkStream which wraps a socket, but it has the same reading issues as the socket itself. You will have to monitor the return value and keep calling Read() until you've received all bytes. Same goes for the TcpClient that has a GetStream() method, on whose return value you'll also have to continue reading until you've read all data.
Related
I recently started to work with TCP in C#. I'm now at the point where I want the client to receive the data sent by the server.
I know that there is no guarantee for the client to receive all data at once. If the size of the data sent is bigger than the buffer's size at the client's side, then the data will be sent in parts. So my question is: how can I store all my received data in a byte array, and then convert it to the actual message when all is received?
I've set the buffer size to 1, so I can see what happens when all sent data doesn't fit in the buffer. Here are my methods in which I call stream.BeginRead() in Client.cs:
// Deliberately setting the buffer size to 1, to simulate what happens when the message doesn't fit in the buffer.
int bufferSize = 1;
byte[] receiveBuffer;
private void ConnectCallback(IAsyncResult result)
{
client.EndConnect(result);
Console.WriteLine("Connected to server.");
stream = client.GetStream();
// At this point, the client is connected, and we're expecting a message: "Welcome!"
receiveBuffer = new byte[bufferSize];
stream.BeginRead(receiveBuffer, 0, receiveBuffer.Length, new AsyncCallback(ReadCallback), stream);
}
private void ReadCallback(IAsyncResult result)
{
int bytesLength = stream.EndRead(result);
// Should be "Welcome!". But of course it's "W", because the bufferSize is 1.
string message = Encoding.UTF8.GetString(receiveBuffer, 0, bytesLength);
Console.WriteLine("Received message: {0}", receivedMessage);
// Reset the buffer and begin reading a new message, which will be "e".
// However, I want the whole message ("Welcome!") in one byte array.
receiveBuffer = new byte[bufferSize];
stream.BeginRead(receiveBuffer, 0, receiveBuffer.Length, ReadCallback, null);
}
This is the output when sending the message "Welcome!":
Connected to server.
Received message: W
Received message: e
Received message: l
Received message: c
Received message: o
Received message: m
Received message: e
Received message: !
Should I temporary store the data until the whole message has arrived, and then convert that to a string?
Follow up question: What if 2 messages are sent closely after each other, for example Welcome! and then What's your name? How do I distinguish the two messages then?
Should I temporary store the data until the whole message has arrived, and then convert that to a string?
Yes, exactly.
Follow up question: What if 2 messages are sent closely after each other, for example Welcome! and then What's your name? How do I distinguish the two messages then?
The general approach is to send the length of the message before the message itself. That way the receiving end will know when it has received a complete package.
As 500 - Internal Server Error already pointed out, you use a buffer for it. Here is some Code example:
Receiving:
while (true) //you should use a bool variable here to stop this on disconnect.
{
byte[] bytes;
bytes = ReadNBytes(ns, 4);
//read out the length field we know is there, because the server always sends it.
int msgLenth = BitConverter.ToInt32(bytes, 0);
bytes = ReadNBytes(ns, msgLenth);
//working with the buffer...
if (bytes.Length > 0)
{
try
{
//do stuff here. bytes contains your complete message.
}
catch (Exception e) { Log(e.Message); }
}
}
public static byte[] ReadNBytes(NetworkStream stream, int n)
{
byte[] buffer = new byte[n];
try
{
int bytesRead = 0;
int chunk;
while (bytesRead < n)
{
chunk = stream.Read(buffer, (int)bytesRead, buffer.Length - (int)bytesRead);
if (chunk == 0)
{
// error out
Log("Unexpected disconnect");
stream.Close();
}
bytesRead += chunk;
}
}
catch (Exception e) { Log(e.Message); }
return buffer;
}
To send stuff use something linke this:
public static void SendObject(NetworkStream ns, byte[] data)
{
byte[] lengthBuffer = BitConverter.GetBytes(data.Length);
ns.Write(lengthBuffer, 0, lengthBuffer.Length);
ns.Write(data, 0, data.Length);
}
I hope that helped!
I am working on socket C#. I've implemented a client server application using socket, but the problem is that the client doesn't receive all data sent by the server.
Here is the client application code. What should I do so that it would receive all data sent by the server?
strRecieved = "";
Socket soc = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
IPEndPoint endPoint = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 9001);
soc.Connect(endPoint);
byte[] msgBuffer = Encoding.Default.GetBytes(path);
soc.Send(msgBuffer, 0, msgBuffer.Length, 0);
byte[] buffer = new byte[2000];
int rec = soc.Receive(buffer);
strRecieved = String.Format(Encoding.Default.GetString(buffer));
First of all. If you're implementing some kind of streaming feature ( tcp/udp/file ) you should consider using some kind of protocol.
What is a protocol? It's just a scheme to use when streaming data. Example:
[4Bytes - length][lengthBytes - message][1Byte - termination indicator]
Knowing the protocol you can read all of the incoming bytes simply as such :
byte[] buffer = new byte[4];
stream.ReadBytes(buffer, 0, 4); // cast that to int and read the rest
int packetLen = BitConverter.ToInt32(buffer, 0);
buffer = new byte[packetLen];
stream.ReadBytes(buffer, 0, buffer.Length); // all bytes that was sent
Remember that you have to subtract thease 4 bytes in the length before sending the message.
EDIT:
Simple example on how to send and receive data using shared protocol.
// sender.cs
string _stringToSend = "some fancy string";
byte[] encodedString = Encoding.UTF8.GetBytes(_stringToSend);
List<byte> buffer = new List<byte>();
buffer.AddRange(BitConverter.GetBytes(encodedString.Length));
buffer.AddRange(encodedString);
netStream.WriteBytes(buffer.ToArray(), 0, buffer.Count);
// netStream sent message in protocol [#LEN - 4Bytes][#MSG - #LENBytes]
// simply speaking something like: 5ABCDE
// receiver.cs
byte[] buffer = new byte[sizeof(int)];
netStream.ReadBytes(buffer, 0, buffer.Length);
// receiver got the length of the message eg. 5
int dataLen = BitConverter.ToInt32(buffer, 0);
buffer = new byte[dataLen];
// now we can read an actual message because we know it's length
netStream.ReadBytes(buffer, 0, buffer.Length);
string receivedString = Encoding.UTF8.GetString(buffer);
// received string is equal to "some fancy string"
Making it simpler
This technique forces you to use desired protocol which in this example will be :
First 4 bytes sizeof(int) are indicating the length of the incoming packet
Every byte further is your packet until the end.
So right now you should make ProtocolHelper object:
public static class ProtocolHelper
{
public byte[] PackIntoProtocol(string message)
{
List<byte> result = new List<byte>();
byte[] messageBuffer = Encoding.UTF8.GetBytes(message);
result.AddRange(BitConverter.GetBytes(messageBuffer.Length), 0); // this is the first part of the protocol ( length of the message )
result.AddRange(messageBuffer); // this is actual message
return result.ToArray();
}
public string UnpackProtocol(byte[] buffer)
{
return Encoding.UTF8.GetString(buffer, 0, buffer.Length);
}
}
Now ( depending on method you've chosen to read from network ) you have to send and receive your message.
// sender.cs
string meMessage = "network message 1";
byte[] buffer = ProtocolHelper.PackIntoProtocol(meMessage);
socket.Send(buffer, 0, buffer.Length, 0);
// receiver.cs
string message = string.Empty;
byte[] buffer = new byte[sizeof(int)]; // or simply new byte[4];
int received = socket.Receive(buffer);
if(received == sizeof(int))
{
int packetLen = BitConverter.ToInt32(buffer);// size of our message
buffer = new byte[packetLen];
received = socket.Receive(buffer);
if( packetLen == received ) // we have full buffer
{
message = PacketHelper.UnpackProtocol(buffer);
}
}
Console.WriteLine(message); // output: "network message 1"
You're limiting the size of received messages by 2KB as you're using new byte[2000].
I think you could either:
Size up you buffer to meet you message's size needs; and/or
Split you message into more than one socket messages.
Given that 4-8K is a good size for buffering socket messages and assuming RAM size is not a issue I would start with that, say, new byte[8000].
Also, you can send socket messages splitted in chunks. Maybe this is a good idea for the case. For example, if you have msg as the message (or object) you want to send:
private static async Task SendAnswer(Message msg, WebSocket socket)
{
var answer = System.Text.Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(msg).ToCharArray());
var bufferSize = 8000;
var endOfMessage = false;
for (var offset = 0; offset < answer.Length; offset += bufferSize)
{
if (offset + bufferSize >= answer.Length)
{
bufferSize = answer.Length - offset;
endOfMessage = true;
}
await socket.SendAsync(new ArraySegment<byte>(answer, offset, bufferSize),
WebSocketMessageType.Text, endOfMessage, CancellationToken.None);
}
}
And when receiving, you can also split the reception in chunks, so you can control you buffer (and therefore you memory consumption). After hanlding the whole message, you should wait for another message from the client to do more stuff. Source
private async Task ReceiveMessage(WebSocket webSocket)
{
var buffer = new byte[8000];
var result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
string msg = Encoding.UTF8.GetString(new ArraySegment<byte>(buffer, 0, result.Count).ToArray());
while (!result.EndOfMessage)
{
result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
msg += Encoding.UTF8.GetString(new ArraySegment<byte>(buffer, 0, result.Count).ToArray());
}
//At this point, `msg` has the whole message you sent, you can do whatever you want with it.
// [...]
//After you handle the message, wait for another message from the client
result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
}
await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
}
I am trying to receive data at my server of any length through tcp connection. First my client sends length of data to server through stream.write then it send the actual data.
At Client I receive the length and loop until whole the data is received successfully.
The problem is: "I receive 0 size on the server no matters what the length of data is". I tried to figure out the issue but could not get where the problem is. Any kind of help/hint would be appreciated.
Server Side Code:
byte[] lengthOfData = new byte[2048];
byte[] buffer;
try
{
stream = client.GetStream();
eventLog1.WriteEntry("Size of 1st = "+stream.Read(lengthOfData,0,lengthOfData.Length));
int numBytesToRead = ByteArraySerializer.BytesArrayToInt(lengthOfData);
eventLog1.WriteEntry("number of bytes to read= "+numBytesToRead);
buffer = new byte[numBytesToRead+10];
int numBytesRead = 0;
do
{
int n = stream.Read(buffer, numBytesRead, 10);
numBytesRead += n;
numBytesToRead -= n;
eventLog1.WriteEntry("number of bytes read= " + numBytesRead);
} while (numBytesToRead > 0);
}
catch (Exception e) // Called automatically when Client Diposes or disconnects unexpectedly
{
eventLog1.WriteEntry("Connection Closes: "+e.ToString());
lock (connectedClients)
{
connectedClients.Remove(client);
}
client.Close();
break;
}
Client Side Code
byte[] command = ByteArraySerializer.Serialize<Command>(cmd);
byte[] sizeOfData = ByteArraySerializer.IntToBytesArray(command.Length);
stream.Write(sizeOfData, 0, sizeOfData.Length);
Console.WriteLine("Size of Data = "+command.Length);
stream.Write(command, 0, command.Length);
Change the following line at server
byte[] lengthOfData = new byte[2048];
to
byte[] lengthOfData = new byte[sizeof(int)];
The issue was that the read at server was supposed to read only 4 bytes integer whereas it was reading other data as well which was getting written after writing the length of data. We are supposed to read only 4 bytes if we want to get the length of data(integer).
I've been trying to send a file from a client to a server application using the TCPClient class in C#. Before I send the actual data, I send some additional information like the exact file size and the file name, so the server application knows how much there is to read. The funny thing is everything was fine when I tested it on 127.0.0.1 - as soon as I replaced the IP address with the actual one, the server could only read about 1,5 KByte of the data that was sent. It still gets the filename and the file size, but theres no way it's retrieving the actual data.
For testing purposes, I replaced the image I was going to send with a simple string and the transmission went alright, so I suppose there is a problem with sending and receiving the data chunks, but I'm not getting any exceptions on the client side either.
Anyone got an idea? Cheers!
Edit:
Thanks so far, this is what I have got codewise. For the client:
IPAddress ipAddress = IPAddress.Parse("xx.xx.xx.xx");
int port = 3003;
int bufferSize = 1024;
TcpClient client = new TcpClient();
NetworkStream netStream;
// Connect to server
try
{
client.Connect(new IPEndPoint(ipAddress, port));
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
netStream = client.GetStream();
// Read bytes from image
byte[] data = File.ReadAllBytes("C:\\Users\\Dan\\Desktop\\asdf.jpg");
// Build the package
byte[] dataLength = BitConverter.GetBytes(data.Length);
byte[] package = new byte[4 + data.Length];
dataLength.CopyTo(package, 0);
data.CopyTo(package, 4);
// Send to server
int bytesSent = 0;
int bytesLeft = package.Length;
while (bytesLeft > 0)
{
int nextPacketSize = (bytesLeft > bufferSize) ? bufferSize : bytesLeft;
netStream.Write(package, bytesSent, nextPacketSize);
bytesSent += nextPacketSize;
bytesLeft -= nextPacketSize;
}
// Clean up
netStream.Close();
client.Close();
And the server:
TcpListener listen = new TcpListener(3003);
TcpClient client;
int bufferSize = 1024;
NetworkStream netStream;
int bytesRead = 0;
int allBytesRead = 0;
// Start listening
listen.Start();
// Accept client
client = listen.AcceptTcpClient();
netStream = client.GetStream();
// Read length of incoming data
byte[] length = new byte[4];
bytesRead = netStream.Read(length, 0, 4);
int dataLength = BitConverter.ToInt32(length,0);
// Read the data
int bytesLeft = dataLength;
byte[] data = new byte[dataLength];
while (bytesLeft > 0)
{
int nextPacketSize = (bytesLeft > bufferSize) ? bufferSize : bytesLeft;
bytesRead = netStream.Read(data, allBytesRead, nextPacketSize);
allBytesRead += bytesRead;
bytesLeft -= bytesRead;
}
// Save image to desktop
File.WriteAllBytes("C:\\Users\\Dan\\Desktop\\tcpimage.jpg", data);
// Clean up
netStream.Close();
client.Close();
About 1.5 KiB sounds like 1500 bytes, "the largest allowed by Ethernet at the network layer". This is the maximum transmission unit (mtu) forcing your network stack to split your file into several small packets.
You need to call the NetworkStream.Read in a loop to read every packet arrived. There's example code of this at MSDN.
Combine this with the default behavior of .NET; consolidating smaller packets to reduce the amount of packets sent, and you'll also see this behavior when sending smaller packets. This can be controlled with ServicePointManager.UseNagleAlgorithm or by using smaller scoped socket options.
Ok, don't know what I'm doing here, but in case of anyone uses this as referrence.
I got rid of unnecessary copying and done a very important improvement. Your way to calc nextPacketSize is not complete, since there could be less data, than 1024 avaliable, and you'll get extra nulls in between these chunks (I had a lot of headache with that, now so happy to figure out)
I made them as function, needed for several files, so here's code:
client
this one is almost the same
public void sendData(byte[] data, NetworkStream stream)
{
int bufferSize = 1024;
byte[] dataLength = BitConverter.GetBytes(data.Length);
stream.Write(dataLength, 0, 4);
int bytesSent = 0;
int bytesLeft = data.Length;
while (bytesLeft > 0)
{
int curDataSize = Math.Min(bufferSize, bytesLeft);
stream.Write(data, bytesSent, curDataSize);
bytesSent += curDataSize;
bytesLeft -= curDataSize;
}
}
server
public byte[] getData(TcpClient client)
{
NetworkStream stream = client.GetStream();
byte[] fileSizeBytes = new byte[4];
int bytes = stream.Read(fileSizeBytes, 0, 4);
int dataLength = BitConverter.ToInt32(fileSizeBytes, 0);
int bytesLeft = dataLength;
byte[] data = new byte[dataLength];
int bufferSize = 1024;
int bytesRead = 0;
while (bytesLeft > 0)
{
int curDataSize = Math.Min(bufferSize, bytesLeft);
if (client.Available < curDataSize)
curDataSize = client.Available; //This saved me
bytes = stream.Read(data, bytesRead, curDataSize);
bytesRead += curDataSize;
bytesLeft -= curDataSize;
}
return data;
}
I used some of your code for a test networking project I'm working on. I tweaked a couple things to fit the requirements of my project, but one alteration I needed to make to the original code is to add a "-4" to the line that was setting the bytesLeft variable. After that the code worked. My test file is 52KB & was transmitted successfully.
// Read the data
int bytesLeft = dataLength-4;
byte[] data = new byte[dataLength];
I have the following code:
private string Connect()
{
string responseData;
try
{
TcpClient client = new TcpClient(ServerIp, Port);
client.ReceiveBufferSize = Int32.MaxValue;
Byte[] data = Encoding.GetEncoding(1251).GetBytes(ReadyQuery);
NetworkStream stream = client.GetStream();
// send data
stream.Write(data, 0, data.Length);
// buffer
data = new Byte[65536];
Int32 bytes = stream.Read(data, 0, data.Length);
responseData = Encoding.GetEncoding(1251).GetString(data, 0, bytes);
// close all
stream.Close();
client.Close();
return responseData;
}
I have problem with a big message. The receive message size is 22K chars. I get only part of message.
How can I receive big messages?
PS. In the debugger bytes equal 4096.
You call stream.Read in a loop until you read the entire message. If you know the message size in advance it's relatively easy:
int messageSize = 22000;
int readSoFar = 0;
byte [] msg = new byte[messageSize];
while(readSoFar < messageSize)
{
var read = stream.Read(msg, readSoFar, msg.Length - readSoFar);
readSoFar += read;
if(read==0)
break; // connection was broken
}
If the message size is part of the message (say, encoded in the first 4 bytes), you should read those first and then do as I suggested.