I'm making a chatroom application, and I send TCP packets to communicate between the server and the client.
I have the following code:
string returnMessage = "[EVT]USERSUCCESS";
bytes = Encoding.ASCII.GetBytes(returnMessage);
info.WriteToStream(bytes);
foreach (ConnectionInfo con in connections)
{
info.WriteToStream(bytes);
bytes = Encoding.ASCII.GetBytes("[EVT]USERJOIN;" + username);
con.WriteToStream(bytes);
}
However, when the client reads this, the response is:
[EVT]USERSUCCESS[EVT]USERJOIN;Name
Which seems as though it is receiving both packets at once..?
This is the code I have for receiving:
static void ServerListener()
{
while (true)
{
byte[] bytes = new byte[1024];
int numBytes = stream.Read(bytes, 0, bytes.Length);
string message = Encoding.ASCII.GetString(bytes, 0, numBytes);
if (HandleResponse(message) && !WindowHasFocus())
{
player.Play();
}
}
}
Which is run as a separate thread. HandleResponse() is fully working.
Thanks in advance!
Tcp is a stream protocol not a packet protocol.
You could get the bytes in mutiple packets or a single packet like you are doing.
What you need to do is put a byte that siginifies the end of a packet (such as the null byte)
Psudeo code:
Have a buffer that you add every byte received to.
If their is a null byte in the buffer then split the buffer at the null byte
Handle the left side as a complete packet. Keep the right side in the buffer until you get a new null byte.
Advanced TCP:
You may want to add your own error checking in your "packet"
Related
I have a tcp listener set up where i have an unknown message size coming in. I am not sure what the best practice is when it comes to handling unknown message sizes. Here is the code:
TcpListener _server = new TcpListener(_localAddr, _port);
_server.Start();
while (true)
{
if (_server.Pending())
{
Byte[] bytes = new Byte[256];//Works fine if message under this size
string data = string.Empty;
_client = _server.AcceptTcpClient();
NetworkStream stream = _client.GetStream();
int i;
while ((i = stream.Read(bytes, 0, bytes.Length)) != 0)
{
data = System.Text.Encoding.ASCII.GetString(bytes, 0, i);
data = data.ToUpper();
//Do stuff with data
}
}
So you will notice i can change the size of the array bytes from 256 to something bigger but i don't know if there is a better way than just setting the size to something i know is big enough. If the message is smaller than 256 it works fine.
Thanks for any help in advance
If the messages length is not permanent the service who send you the messages should send you in the beginning of every message a few bytes (usually 4 because Int32 is 32 bits that are 4 bytes) that will indicate the message length.
When you receive the message you should first read this 4 bytes, then after you parse them you will know the message length and you can create array of bytes in the appropriate size.
Now you can read to this array the message itself.
I am relativity new to C#. In my TCP client have the following function which sends data to the server and returns the response:
private static TcpClient tcpint = new TcpClient(); //Already initiated and set up
private static NetworkStream stm; //Already initiated and set up
private static String send(String data)
{
//Send data to the server
ASCIIEncoding asen = new ASCIIEncoding();
byte[] ba = asen.GetBytes(data);
stm.Write(ba, 0, ba.Length);
//Read data from the server
byte[] bb = new byte[100];
int k = stm.Read(bb, 0, 100);
//Construct the response from byte array to string
StringBuilder sb = new StringBuilder();
for (int i = 0; i < k; i++)
{
sb.Append(bb[i].ToString());
}
//Return server response
return sb.ToString();
}
As you can see here, when I am reading the response from the server, I am reading it into a fix byte[] array of length 100 bytes.
byte[] bb = new byte[100];
int k = stm.Read(bb, 0, 100);
What do i do if the response from the server is more than 100 bytes? How can I read the data without me knowing what the max length of data form the server will be?
Typically, where there is not some specific intrinsic size of something, tcp protocols explicitly send the length of objects they are sending. One possible method for illustration:
size_t data_len = strlen(some_data_blob);
char lenstr[32];
sprintf(lenstr, "%zd\n", data_len);
send(socket, lenstr, strlen(lenstr));
send(socket, some_data_blob, data_len);
then when the receiver reads the length string, it knows exactly how mush data should follow (good programming practice is to trust but verify though -- if there is more or less data really sent -- say by an 'evil actor' -- you need to be prepared to handle that).
Not with respect to C# but a general answer on writing TCP application:
TCP is steam based protocol. It does not maintain message boundaries. So, the applications using TCP should take care of choosing the right method of data exchange between server and client. Its becomes more paramount if multiple messages gets sent and received on one connection.
One widely used method is to prepend the data message with the length bytes.
Ex:
[2 byte -length field][Actual Data].
The receiver of such data (be it server or client needs to decode length field, wait for until such event where as many bytes are received or raise an alarm on timeout and give up.
Another protocol that can be used is to have applications maintain message boundaries.
Ex:
`[START-of-MSG][Actual Data][END-of-MSG]
The reciever has to parse the data for Start-byte and End-byte (predefined by application protocol) and treat anything in between as data of interest.
hello i solved it with a list, i don't know the size of the complete package but i can read it in parts
List<byte> bigbuffer = new List<byte>();
byte[] tempbuffer = new byte[254];
//can be in another size like 1024 etc..
//depend of the data as you sending from de client
//i recommend small size for the correct read of the package
NetworkStream stream = client.GetStream();
while (stream.Read(tempbuffer, 0, tempbuffer.Length) > 0) {
bigbuffer.AddRange(tempbuffer);
}
// now you can convert to a native byte array
byte[] completedbuffer = new byte[bigbuffer.Count];
bigbuffer.CopyTo(completedbuffer);
//Do something with the data
string decodedmsg = Encoding.ASCII.GetString(completedbuffer);
I do this whith images and looks good, i thik than you dont know the size of the data if the porpouse is read a complete source with a unknow size
I was looking around for an answer to this, and noticed the Available property was added to TcpClient. It returns the amount of bytes available to read.
I'm assuming it was added after most of the replies, so I wanted to share it for others that may stumble onto this question.
https://learn.microsoft.com/en-us/dotnet/api/system.net.sockets.tcpclient.available?view=netframework-4.8
I am writing a C# client application which will connect to the server written in python. My question is about receiving data in loop. The application structure is all about client asks server -> server responds to client. Everything works fine when the message is lower that actual buffer size (set in server). For example: server side buffer: 1024, client buffer size: 256, data length < 1kb. I run my application with following code:
int datacounter = 0;
byte[] recived = new byte[256];
StringBuilder stb = new StringBuilder();
serverStream.ReadTimeout = 1500;
try
{
while ((datacounter = serverStream.Read(recived, 0, 256)) > 0)
{
Console.WriteLine("RECIVED: " + datacounter.ToString());
stb.append(System.Text.Encoding.UTF8.GetString(recived, 0, datacounter));
}
}
catch { Console.WriteLine("Timeout!"); }
Then the application receives data in 4 loops (256 bytes each):
RECIVED: 256
RECIVED: 256
RECIVED: 256
RECIVED: 96
And then the timeout ticks, that ends the transmission and pass the complete data to later analysis (from stb object). I don't think using timeout is proper, but i don't know any other way to do this.
However, this way it works. Here we go with example, that does not:
server side buffer: 1024, client side buffer: 256, data length ~ 8kbytes (python side sends data in loop).
RECIVED: 256
RECIVED: 256
RECIVED: 256
RECIVED: 256
Then the timeout ticks (and obviosly the data is incomplete - got 1kb of 8kb). Sometimes the loop even ends after 1 run with 28 recived bytes and thats all before timeout. Python says that the data has been send properly. Here's the way i create the socket and serverStream object:
TcpClient clientSocket = new TcpClient();
clientSocket.Connect("x.y.z.x", 1234);
NetworkStream serverStream = clientSocket.GetStream();
Its not the TcpClient fault. Tried the same with clear sockets, created like:
new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)
However that works similar. Is there a way, to make my loop work without timeout, receiving all data? I would like to keep the socket synchronous if possible.
I don't think there's anything wrong with your receive code functionally. I put together a test, and the receiver gets as much as you can send it (eg. 8 MBs), as long as you keep sending without 1.5 seconds pause, before timing out.
So it looks like your server is simply not sending "fast" enough.
To answer your question, timing is not the typical way of knowing when you have received a full message. One common, simple way of determining when a full message is received is to prefix the length of the full message on the sending side (eg. 4-byte int). Then on the receive side, first read 4 bytes, decode to the length, and then read that many more bytes.
You could also consider appending a message termination string, such as Environment.NewLine, to the end of your message. This has the advantage that you could call StreamReader.ReadLine(), which will block until the full message is received. This only works if the termination can NOT be included in the message itself.
If you can't alter the server protocol, is there any other way of knowing you have received a full message? (eg. checking for a NewLine at the end of the message, an XML end tag, or some other pattern.) If not, perhaps you could wait for the server to disconnect, otherwise it looks like you would be forced to find the right timing balance.
I am including the test code below in case you want to play around with it.
Server/Sending Side:
IPAddress localAddr = IPAddress.Parse("127.0.0.1");
TcpListener server = new TcpListener(localAddr, 13579);
server.Start();
TcpClient clientSocket = server.AcceptTcpClient();
NetworkStream stream = clientSocket.GetStream();
int bytesSent = 0;
int bytesToSend = 1 << 25;
int bufferSize = 1024;
string testMessage = new string('X', bufferSize);
byte[] buffer = UTF8Encoding.UTF8.GetBytes(testMessage);
while (bytesSent < bytesToSend)
{
int byteToSendThisRound = Math.Min(bufferSize, bytesToSend - bytesSent);
stream.Write(buffer, 0, byteToSendThisRound);
bytesSent += byteToSendThisRound;
}
Client/Receiving Side:
TcpClient client = new TcpClient("127.0.0.1", 13579);
NetworkStream serverStream = client.GetStream();
int totalBytesReceived = 0;
int datacounter = 0;
byte[] recived = new byte[256];
StringBuilder stb = new StringBuilder();
serverStream.ReadTimeout = 1500;
try
{
while ((datacounter = serverStream.Read(recived, 0, 256)) > 0)
{
totalBytesReceived += 256;
Console.WriteLine("RECIVED: {0}, {1}", datacounter, totalBytesReceived);
stb.Append(System.Text.Encoding.UTF8.GetString(recived, 0, datacounter));
}
}
catch { Console.WriteLine("Timeout!"); }
Why don't you dump the exception that makes your code go into the catch branch and find out? :)
catch (Exception ex) { Console.WriteLine("Timeout because of... " + ex.Message); }
--EDIT
Sorry I didn't see the timout. The question you're asking is if there's a way to do it without timeout. Yes, don't set any timeout and check if received number of bytes is smaller than the buffer size.
That is:
while ((datacounter = serverStream.Read(recived, 0, 256)) > 0)
{
Console.WriteLine("RECIVED: " + datacounter.ToString());
stb.append(System.Text.Encoding.UTF8.GetString(recived, 0, datacounter));
if(datacounter < 256) //you're good to go
break;
}
For anyone else who needs help with this
Just to add to Chi_Town_Don's answer, make sure you use stb.ToString() outside of the loop. And I've found that nothing will print out unless the loop breaks out. To do that if(!serverStream.DataAvailable()){break;} works wonders. That way you don't need to pass in the packet size or some other convoluted condition.
This question already has answers here:
Receiving data in TCP
(10 answers)
Closed 2 years ago.
Here's my code:
private void OnReceive(IAsyncResult result)
{
NetStateObject state = (NetStateObject)result.AsyncState;
Socket client = state.Socket;
int size = client.EndReceive(result);
byte[] data = state.Buffer;
object data = null;
using (MemoryStream stream = new MemoryStream(data))
{
BinaryFormatter formatter = new BinaryFormatter();
data = formatter.Deserialize(stream);
}
//todo: something with data
client.BeginReceive(
state.Buffer,
0,
NetStateObject.BUFFER_SIZE,
SocketFlags.None,
OnReceive,
state
);
}
state.Buffer has a maximum size of NetStateObject.BUFFER_SIZE (1024). Firstly, is this too big or too small? Second, if I send something larger than that, my deserialize messes up because the object it is trying to deserialize doesnt have all the information (because not all the data was sent). How do I make sure that all my data has been received before I try to construct it and do something with it?
Completed Working Code
private void OnReceive(IAsyncResult result)
{
NetStateObject state = (NetStateObject)result.AsyncState;
Socket client = state.Socket;
try
{
//get the read data and see how many bytes we received
int bytesRead = client.EndReceive(result);
//store the data from the buffer
byte[] dataReceived = state.Buffer;
//this will hold the byte data for the number of bytes being received
byte[] totalBytesData = new byte[4];
//load the number byte data from the data received
for (int i = 0; i < 4; i++)
{
totalBytesData[i] = dataReceived[i];
}
//convert the number byte data to a numan readable integer
int totalBytes = BitConverter.ToInt32(totalBytesData, 0);
//create a new array with the length of the total bytes being received
byte[] data = new byte[totalBytes];
//load what is in the buffer into the data[]
for (int i = 0; i < bytesRead - 4; i++)
{
data[i] = state.Buffer[i + 4];
}
//receive packets from the connection until the number of bytes read is no longer less than we need
while (bytesRead < totalBytes + 4)
{
bytesRead += state.Socket.Receive(data, bytesRead - 4, totalBytes + 4 - bytesRead, SocketFlags.None);
}
CommandData commandData;
using (MemoryStream stream = new MemoryStream(data))
{
BinaryFormatter formatter = new BinaryFormatter();
commandData = (CommandData)formatter.Deserialize(stream);
}
ReceivedCommands.Enqueue(commandData);
client.BeginReceive(
state.Buffer,
0,
NetStateObject.BUFFER_SIZE,
SocketFlags.None,
OnReceive,
state
);
dataReceived = null;
totalBytesData = null;
data = null;
}
catch(Exception e)
{
Console.WriteLine("***********************");
Console.WriteLine(e.Source);
Console.WriteLine("***********************");
Console.WriteLine(e.Message);
Console.WriteLine("***********************");
Console.WriteLine(e.InnerException);
Console.WriteLine("***********************");
Console.WriteLine(e.StackTrace);
}
}
TCP is a stream protocol. It has no concept of packets. A single write call can be sent in multiple packets, and multiple write calls can be put into the same packet. So you need to implement your own packetizing logic on top of TCP.
There are two common ways to packetize:
Delimiter characters, this is usually used in text protocols, with the new-line being a common choice
Prefix the length to each packet, usually a good choice with binary protocols.
You store the size of a logical packet at the beginning of that packet. Then you read until you received enough bytes to fill the packet and start deserializing.
How do I make sure that all my data has been received before I try to construct it and do something with it?
You have to implement some protocol so you know.
While TCP is reliable, it does not guarantee that the data from single write at one end of the socket will appear as a single read at the other end: retries, packet fragmentation and MTU can all lead to data being received in different sized units by the receiver. You will get the data in the right order.
So you need to include some information when sending that allows the receiver to know when it has the complete message. I would also recommend including what kind of message and what version of the data (this will form the basis of being able to support different client and server versions together).
So the sender sends:
- Message type
- Message version
- Message size (in bytes)
And the receiver will loop, performing a read with a buffer and appending this to a master buffer (MemoryStream is good for this). Once the complete header is received it knows when the complete data has been received.
(Another route is to include some pattern as an "end of message" marker, but then you need to handle the same sequence of bytes occurring in the content—hard to do if the data is binary rather than text.)
Hey, I'm having an issue seperating packets using a custom binary protocol.
Currently the server side code looks like this.
public void HandleConnection(object state)
{
TcpClient client = threadListener.AcceptTcpClient();
NetworkStream stream = client.GetStream();
byte[] data = new byte[4096];
while (true)
{
int recvCount = stream.Read(data, 0, data.Length);
if (recvCount == 0) break;
LogManager.Debug(Utility.ToHexDump(data, 0, recvCount));
//processPacket(new MemoryStream(data, 0, recvCount));
}
LogManager.Debug("Client disconnected");
client.Close();
Dispose();
}
I've been watching the hex dumps of the packets, and sometimes the entire packet comes in one shot, let's say all 20 bytes. Other times it comes in fragmented, how do I need to buffer this data to be able to pass it to my processPacket() method correctly. I'm attempting to use a single byte opcode header only, should I add something like a (ushort)contentLength to the header aswell? I'm trying to make the protocol as lightweight as possible, and this system won't be sending very large packets(< 128 bytes).
The client side code I'm testing with is, as follows.
public void auth(string user, string password)
{
using (TcpClient client = new TcpClient())
{
client.Connect(IPAddress.Parse("127.0.0.1"), 9032);
NetworkStream networkStream = client.GetStream();
using (BinaryWriter writer = new BinaryWriter(networkStream))
{
writer.Write((byte)0); //opcode
writer.Write(user.ToUpper());
writer.Write(password.ToUpper());
writer.Write(SanitizationMgr.Verify()); //App hash
writer.Write(Program.Seed);
}
}
}
I'm not sure if that could be what's messing it up, and binary protocol doesn't seem to have much info on the web, especially where C# is involved. Any comment's would be helpful. =)
Solved with this, not sure if it's correct, but it seems to give my handlers just what they need.
public void HandleConnection(object state)
{
TcpClient client = threadListener.AcceptTcpClient();
NetworkStream stream = client.GetStream();
byte[] data = new byte[1024];
uint contentLength = 0;
var packet = new MemoryStream();
while (true)
{
int recvCount = stream.Read(data, 0, data.Length);
if (recvCount == 0) break;
if (contentLength == 0 && recvCount < headerSize)
{
LogManager.Error("Got incomplete header!");
Dispose();
}
if(contentLength == 0) //Get the payload length
contentLength = BitConverter.ToUInt16(data, 1);
packet.Write(data, (int) packet.Position, recvCount); //Buffer the data we got into our MemStream
if (packet.Length < contentLength + headerSize) //if it's not enough, continue trying to read
continue;
//We have a full packet, pass it on
//LogManager.Debug(Utility.ToHexDump(packet));
processPacket(packet);
//reset for next packet
contentLength = 0;
packet = new MemoryStream();
}
LogManager.Debug("Client disconnected");
client.Close();
Dispose();
}
You should just treat it as a stream. Don't rely on any particular chunking behaviour.
Is the amount of data you need always the same? If not, you should change the protocol (if you can) to prefix the logical "chunk" of data with the length in bytes.
In this case you're using BinaryWriter on one side, so attaching a BinaryReader to the NetworkStream returned by TcpClient.GetStream() would seem like the easiest approach. If you really want to capture all the data for a chunk at a time though, you should go back to my idea of prefixing the data with its length. Then just loop round until you've got all the data.
(Make sure you've got enough data to read the length though! If your length prefix is 4 bytes, you don't want to read 2 bytes and miss the next 2...)