I have a C# client/server network program I've written using
TCPListener and TCPClient classes. The server is reading everything
from the client (small amounts of xml) just fine until I try to send a
large file (100k) back to the client.
I'm using stream functions for
both client and server with non-blocking socket functions. When I do a
socket.SendFile("filename") back to the client, the file is getting
cut off - I've set the receive buffer size on the client to well past
100k but it still gets cut off around 25k and the communication
between client and server is unreliable afterwords.
My basic question
is what happens if data is somehow left in the pipe ? i.e.. will it be
read by the next socket.Read... Does every Send call require exactly
one and only one Read ? Maybe I'm not giving the client enough time to
read the file but their both on the same machine and I've tried
sleeping for a few seconds in various places w/o success.
It is very possible that you cannot read the entire message through one Read call (perhaps all data has not arrived yet). In network programming, you would often place the call to Read in a while loop and simply Read() until you have received the entire expected message.
1 Send call might take more than one Read call to receive, and 1 Read call might read the data send by several Send call.
TCP just provides a stream, so it's up to you to define how the data or messages you send are partitioned.
In this case, you probably just need to loop ,doing Read until the stream is closed.
You probably want something like this:
socket.Blocking = false;
const int ChunkSize = 1492;
const int ReceiveTimeout = 10000;
const int SendTimeout = 10000;
public void Send(Byte[] data)
{
var sizeBytes = BitConverter.GetBytes(data.Length);
SendInternal(sizeBytes);
SendInternal(data);
}
public Byte[] Receive()
{
var sizeBytes = ReceiveInternal(4);
var size = BitConverter.ToInt32(sizeBytes, 0);
var data = ReceiveInternal(size);
return data;
}
private void SendInternal(Byte[] data)
{
var error = SocketError.Success;
var lastUpdate = Environment.TickCount;
var size = data.Length;
var count = 0;
var sent = 0;
while (sent < size)
{
count = Math.Min(ChunkSize, size - sent);
count = socket.Send(data, sent, count, SocketFlags.None, out error);
if (count > 0)
{
sent += count;
lastUpdate = Environment.TickCount;
}
if (error != SocketError.InProgress && error != SocketError.Success && error != SocketError.WouldBlock)
throw new SocketException((Int32)error);
if (Environment.TickCount - lastUpdate > SendTimeout)
throw new TimeoutException("Send operation timed out.");
if (count == 0 && !socket.Poll(100, SelectMode.SelectWrite))
throw new SocketException((Int32)SocketError.Shutdown);
}
}
private Byte[] ReceiveInternal(Int32 size)
{
var error = SocketError.Success;
var lastUpdate = Environment.TickCount;
var buffer = new Byte[ChunkSize];
var count = 0;
var received = 0;
using (var ms = new MemoryStream(size))
{
while (received < size)
{
count = Math.Min(ChunkSize, size - received);
count = socket.Receive(buffer, 0, count, SocketFlags.None, out error);
if (count > 0)
{
ms.Write(buffer, 0, count);
received += count;
lastUpdate = Environment.TickCount;
}
if (error != SocketError.InProgress && error != SocketError.Success && error != SocketError.WouldBlock)
throw new SocketException((Int32)error);
if (Environment.TickCount - lastUpdate > ReceiveTimeout)
throw new TimeoutException("Receive operation timed out.");
if (count == 0 && socket.Poll(100, SelectMode.SelectRead) && socket.Available == 0)
throw new SocketException((Int32)SocketError.Shutdown);
}
return ms.ToArray();
}
}
What I would usually do is create a header structure that is sent
Header Size (int, 4 bytes)
File Name Offset (int, 4 bytes)
File Name Size (int , 4 bytes)
File Data Offset (int, 4 bytes)
File Data Size (int , 4 bytes)
[ message data here]
and then that header is read using either a BinaryReader or copying the bytes to a struct using Marshal. This way you always know what data is arriving and how many times you need to call Read().
The header size field is also helps with versioning the protocol (keep the structure the same but add to it for later clients so you can keep backwards compatibility). If you define the structure in C# be sure to do it like so:
[StructLayout LayoutKind.Sequential]
struct MessageHeader
{
public int HeaderSize;
public int FileNameOffset;
public int FileNameSize;
public int FileDataOffset;
public int FileDataSize;
}
Then Marshal.PtrToStructure will allow you do create an instance of this struct right from the byte[] you read from the socket
Try sending the chunk from the server side in chunks. Just as the other said, posting the code would be of great help to us.
Related
What I'm doing is taking a user entered string, creating a packet with the data, then sending the string out to a serial port. I am then reading the data I send via a loopback connector. My send is working flawlessly, however my receive is randomly throwing an arithmetic overflow exception.
I say randomly because it is not happening consistently. For example, I send the message "hello" twice. The first time works fine, the second time outputs nothing and throws an exception. I restart my program, run the code again, and send hello only to receive "hell" and then an exception. On rare occasion, I'll receive the packet 3 or 4 times in a row without error before the exception.
Here is my relevant code:
public void receivePacket(object sender, SerialDataReceivedEventArgs e)
{
byte[] tempByte = new byte[2];
int byteCount = 0;
while (serialPort1.BytesToRead > 0)
{
if (byteCount <= 1)
{
tempByte[byteCount] = (byte)serialPort1.ReadByte();
}
if (byteCount == 1)
{
receivedString = new byte[tempByte[byteCount]];
receivedString[0] = tempByte[0];
receivedString[1] = tempByte[1];
}
else if (byteCount > 1)
{
byte b = (byte)serialPort1.ReadByte();
receivedString[byteCount] = b;
}
byteCount++;
}
int strLen = (byteCount - 3);
tempByte = new byte[strLen];
int newBit = 0;
for (int i = 2; i <= strLen+1; i++)
{
tempByte[newBit] = receivedString[i];
newBit++;
}
string receivedText = encoder.GetString(tempByte);
SetText(receivedText.ToString());
}
I'm well aware that my implementation using byteCount (which I use to traverse the byte array) is rather sloppy. When I step through the code, I find that when I get the error byteCount == 1, which is making strLen a negative number (since strLen is byteCount - 3, which is done because the packet contains a header, length, and CRC i.e. byteCount - 3 == # of actual data bytes received). This leads to by tempByte having a size of -2, which throws my exceptions. I, however, am having a very hard time figuring out why byteCount is being set to 1.
The code after this basically just traverses the data section of the array, copies it into the tempByte, then is sent off to a function to append the text in another thread.
I am guessing that byteCount is 1 because you only received one byte - or rather, you processed the first byte before the second one arrived in the buffer.
The ReadByte function will wait for a certain amount of time for a byte to arrive if there isn't one waiting.
Maybe if instead of checking BytesToRead, you did something more like this:
byte headerByte = serialPort1.ReadByte();
byte length = serialPort1.ReadByte();
receivedString = new byte[length];
receivedString[0] = headerByte;
receivedString[1] = length;
for (int i = 2; i < length; i++) {
receivedString[i] = serialPort1.ReadByte();
}
I want to communicate with a DSP using RS232, so I use System.IO.SerialPort to achieve this. Everything goes well except the reading performance.
Every 200ms, the port can received a package of 144 bytes. But in the tests, the applications almost skip every other package. I try to print the system time in the console. It amaze me that the code below (when length = 140) take me over 200ms. It let the application can not handle the data in time.
Does anything wrong I do?
Port Property:
BaudRate = 9600
Parity = None
StopBits = One
private byte[] ReadBytesInSpicifiedLength(int length)
{
byte[] des = new byte[length];
for (int i = 0; i < length; i++)
{
des[i] = (byte)serialPort.ReadByte();
}
return des;
}
You're doing a lot of individual I/O calls, which means a lot of kernel transitions. Those are expensive. Not being able to reach 720 bytes per second is surprising, but you can make the data handling an order of magnitude faster by doing block reads:
private byte[] ReadBytesWithSpecifiedLength(int length)
{
byte[] des = new byte[length];
serialPort.BaseStream.Read(des, 0, des.Length);
return des;
}
If you have timeouts enabled, you could get partial reads. Then you need to do something like:
private byte[] ReadBytesWithSpecifiedLength(int length)
{
byte[] des = new byte[length];
int recd = 0;
do {
int partial = serialPort.BaseStream.Read(des, recd, length - recd);
if (partial == 0) throw new IOException("Transfer Interrupted");
recd += partial;
} while (recd < length);
return des;
}
The nice thing about BaseStream is that it also has async support (via ReadAsync). That's what new C# code should be using.
I'm writing an interface for talking to a piece of test equipment. The equipment talks over a serial port and responds with a known number of bytes to each command I send it.
My current structure is:
Send command
Read number of specified bytes back
Proceed with application
However, when I used SerialPort.Read(byte[], int32, int32), the function is not blocking. So, for example, if I call MySerialPort.Read(byteBuffer, 0, bytesExpected);, the function returns with less than the specified number of bytesExpected. Here is my code:
public bool ReadData(byte[] responseBytes, int bytesExpected, int timeOut)
{
MySerialPort.ReadTimeout = timeOut;
int bytesRead = MySerialPort.Read(responseBytes, 0, bytesExpected);
return bytesRead == bytesExpected;
}
And I call this method like this:
byte[] responseBytes = new byte[13];
if (Connection.ReadData(responseBytes, 13, 5000))
ProduceError();
My problem is that I can't ever seem to get it to read the full 13 bytes like I am telling it. If I put a Thread.Sleep(1000) right before my SerialPort.Read(...) everything works fine.
How can I force the Read method to block until either the timeOut is exceeded or the specified number of bytes are read?
That is expected; most IO APIs allow you to specify the upper bound only - they are simply required to return at-least-one byte, unless it is an EOF in which case they can return a non-positive value. To compensate, you loop:
public bool ReadData(byte[] responseBytes, int bytesExpected, int timeOut)
{
MySerialPort.ReadTimeout = timeOut;
int offset = 0, bytesRead;
while(bytesExpected > 0 &&
(bytesRead = MySerialPort.Read(responseBytes, offset, bytesExpected)) > 0)
{
offset += bytesRead;
bytesExpected -= bytesRead;
}
return bytesExpected == 0;
}
The only problem is you might need to reduce the timeout per iteration, by using a Stopwatch or similar to see how much time has passed.
Note that I also removed the ref on responseBytes - you don't need that (you don't re-assign that value).
Try changing the timeout to InfiniteTimeout.
SerialPort.Read is expected to throw a TimeoutException in case no bytes are available before SerialPort.ReadTimeout.
So this method reads exactly the desired number or bytes, or throws an exception:
public byte[] ReadBytes(int byteCount) {
try
{
int totBytesRead = 0;
byte[] rxBytes = new byte[byteCount];
while (totBytesRead < byteCount) {
int bytesRead = comPort.Read(rxBytes, totBytesRead, byteCount - totBytesRead);
totBytesRead += bytesRead;
}
return rxBytes;
}
catch (Exception ex){
throw new MySerialComPortException("SerialComPort.ReadBytes error", ex);
}
}
I'm working on Serialport. I'm facing a new problem that once I receive data my data are incomplete. How can I check if my data are complete then process them, and if not, don't process them?
Here are my data receive and my send function:
private void Send(byte[] cmd)
{
bResponse = new byte[0];
Write(cmd);
}
void comPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
int iCount = comPort.BytesToRead;
byte[] bBuffer = new byte[iCount];
comPort.Read(bBuffer, 0, iCount)
if (bBuffer.Length == 1 && bBuffer[0] == ACK)
Write(new byte[] { ENQ });
else if (bBuffer.Length == 1 && bBuffer[0] == NAK)
{
Debug.WriteLine("Incomplete Message detected!");
}
else
{
bResponse = bResponse.Concat(bBuffer).ToArray();
rResponse = Decode(bResponse);
Write(new byte[] { ACK });
}
}
I know my data are received in a few packages and I need to wait until the response is complete, but I don't know based on the code above. How should I check whether the data are complete to determine whether to wait? (P.S: The size of the received response varies.)
There is no built-in concept of completeness or packet size.
You'll have to append to a buffer until you see some recognizable end-of-packet pattern that you (or someone else) defined as part of the protocol specification. - And then probably time out after a while if you haven't seen what you are looking for.
Example of old project, notice the firstindex, lastindex, you put in a character to know the length, the start/end character is predefined and can be any character you choose, just be sure not to take any common characters
This is for tcp/ip, but same principle can be used for serialport
public void ReceiveMessage(IAsyncResult ar)
{
int bytesRead;
try
{
lock (client1.GetStream())
{
bytesRead = client1.GetStream().EndRead(ar);
}
string messageReceived = System.Text.Encoding.ASCII.GetString(data, 0, bytesRead);
received = messageReceived;
int firstindex = received.IndexOf((char)2);
int lastindex = received.IndexOf((char)3);
if (firstindex > 0 && lastindex > 0)
{
string first = received.Substring(firstindex, 1);
string last = received.Substring(lastindex, 1);
}
lock (client1.GetStream())
{
client1.GetStream().BeginRead(data, 0, System.Convert.ToInt32(client1.ReceiveBufferSize), ReceiveMessage, null);
}
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
}
I have some code for you.
First, you implement the DataReceived Event (as you have done already). This event is only called when there is data to process. While I would not call it interrupt-based (as in "realtime capable") is is definitely not polling. Which is good.
Second: When the event is called you may have only one byte, but there may be more bytes. To capture each packet you need to implement an custom buffer.
Third: After you append one byte to your buffer, you check whether the buffer contains a valid packet. If so, process it. If not: Append another one. If no bytes are left, wait for the event to be called again.
In code it looks like this:
const BUFFERLENGTH = 17; // Bytes
byte[] buffer = new byte[BUFFERLENGTH];
private void COM_Port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
var port = (SerialPort)sender;
while (port.BytesToRead > 0)
{
var data = (byte)port.ReadByte();
Read(data);
}
}
private void Read(byte value)
{
// Append Byte to buffer
System.Buffer.BlockCopy(buffer, 1, buffer, 0, BUFFERLENGTH- 1);
buffer[BUFFERLENGTH - 1] = value;
// Check for valid Packet
if (IsValidPacket(buffer))
{
// Yeah! Gotcha :-)
// Now copy your Packet from the Buffer to your struct/whatever
}
}
private bool IsValidPacket(byte[] buffer)
{
// Todo: Check whether buffer contains a valid Packet.
// Some think like:
// return buffer != null && buffer[0] == 0xF4 && buffer[2] == buffer.length
throw new NotImplementedException();
}
Note that I did not "append the byte to the buffer". I discarded the first byte, shifted every byte by one position and inserted the new byte at the end. If a valid Packet was found I could just copy it in one block into a struct. So the buffer size is always constant and exactly as long as one packet.
This may not be the fastest code out there (because it's reading each byte separately) but it works well for me :-)
Oh, and remember to use Begininvoke() if you want to display that stuff in your GUI.
After enough playing with asynchronous socket programming I noticed that the server was receiving chunked payloads (ie: more than one complete payload sitting in the same buffer). So I came up with the following:
if (bytes_to_read > 0)
{
while (bytes_to_read > 0)
// Get payload size as int.
// Get payload in byte format.
// Do something with payload.
// Decrease the amount of bytes to read.
}
// Wait for more data.
}
And then I noticed packet fragmentation (ie: what I thought were complete payloads chunked together wasn't always so) which changed the previous code to something like:
if (bytes_to_read > 0)
{
while (bytes_to_read > 0)
{
// Get payload size as int.
// Check if the payload size is less than or equal to the amount of bytes left to read.
if (payload_size <= bytes_to_read)
{
// Get payload in byte format.
// Do something with payload.
// Decrease the amount of bytes to read.
}
else
{
// We received a fragmented payload.
break;
}
}
if (bytes_to_read == 0)
{
// Wait for more data.
}
else if (bytes_to_read > 0)
{
// Wait for more data where we left off. ***
}
else
{
// Something awful happened.
}
}
*** I don't even know how to go about this and would like to see code for it. I had an idea that it involved copying the in-completed payload to the beginning of the buffer and then picking up from there.
The pseudo code I included is based on the Begin* End* method I am using (I'm aware that I should be using the *Async set of methods found here -> http://msdn.microsoft.com/en-us/library/system.net.sockets.socketasynceventargs.aspx <- but I think my overall question still applies).
I am seeking the answers to 2 questions--namely:
Is this approach correct or am I
missing something?
What does a working example of
dealing with packet fragmentation in
C# look like?
EDIT: I'm using raw sockets.
Thanks in advance for all your help.
EDIT: John Saunders and Greg Hewgill have brought up the point of treating data as a stream but that does not provide me with a concrete example of how to deal with the last chunked payload sometimes being fragmented.
EDIT: I have read Jon Skeet's answer here which is basically along the same lines as the other answers I have seen but it doesn't help me much as I already get what I have to do but not how to do it.
EDIT: To elaborate on what I mean by fragmentation, consider the following the receive buffers:
224TEST3foo3bar
224TEST3foo3bar224TEST3foo3bar
224TEST3foo3bar224TEST3foo
3bar224TEST3foo3bar
EDIT: I found this and this which lead me here. Vadym Stetsiak has cleared nearly everything up (his was one of the answers I was looking for).
This may or may not have anything to do with fragmentation.
In general, the socket will pass you as many bytes at a time as it feels like. Your job is to know how many bytes are in your overall message, and to read them all. Just keep looping until you have all the bytes you need, or until there's an exception.
The following code is untested right now. I thought I'd post it before writing the server side of it and testing both.
private static string ReceiveMessage(Socket socket)
{
const int BUFFER_SIZE = 1024;
var inputBuffer = new byte[BUFFER_SIZE];
var offset = 0;
var bytesReceived = socket.Receive(
inputBuffer, offset, BUFFER_SIZE - offset, SocketFlags.None);
if (bytesReceived < 2)
{
throw new InvalidOperationException("Receive error");
}
var inputMessageLength = inputBuffer[0]*256 + inputBuffer[1];
offset += bytesReceived;
var totalBytesReceived = bytesReceived;
while (bytesReceived > 0 &&
totalBytesReceived < inputMessageLength + 2)
{
bytesReceived = socket.Receive(
inputBuffer, offset, BUFFER_SIZE - offset, SocketFlags.None);
offset += bytesReceived;
totalBytesReceived += bytesReceived;
}
return Encoding.UTF8.GetString(
inputBuffer, 2, totalBytesReceived - 2);
}
Note that the receipt of the message length is wrong. The socket layer could give it to me a byte at a time. I'm going to revisit that as part of a refactoring that will receive the count into a separate two-byte buffer, and change the loop into a single do/while.
When you have to do it yourself, it can be done like so (reference here):
///
/// Server state holds current state of the client socket
///
class AsyncServerState
{
public byte[] Buffer = new byte[512]; //buffer for network i/o
public int DataSize = 0; //data size to be received by the server
//flag that indicates whether prefix was received
public bool DataSizeReceived = false;
public MemoryStream Data = new MemoryStream(); //place where data is stored
public SocketAsyncEventArgs ReadEventArgs = new SocketAsyncEventArgs();
public Socket Client;
}
///
/// Implements server receive logic
///
private void ProcessReceive(SocketAsyncEventArgs e)
{
//single message can be received using several receive operation
AsyncServerState state = e.UserToken as AsyncServerState;
if (e.BytesTransferred <= 0 || e.SocketError != SocketError.Success)
{
CloseConnection(e);
}
int dataRead = e.BytesTransferred;
int dataOffset = 0;
int restOfData = 0;
while (dataRead > 0)
{
if (!state.DataSizeReceived)
{
//there is already some data in the buffer
if (state.Data.Length > 0)
{
restOfData = PrefixSize - (int)state.Data.Length;
state.Data.Write(state.Buffer, dataOffset, restOfData);
dataRead -= restOfData;
dataOffset += restOfData;
}
else if (dataRead >= PrefixSize)
{ //store whole data size prefix
state.Data.Write(state.Buffer, dataOffset, PrefixSize);
dataRead -= PrefixSize;
dataOffset += PrefixSize;
}
else
{ // store only part of the size prefix
state.Data.Write(state.Buffer, dataOffset, dataRead);
dataOffset += dataRead;
dataRead = 0;
}
if (state.Data.Length == PrefixSize)
{ //we received data size prefix
state.DataSize = BitConverter.ToInt32(state.Data.GetBuffer(), 0);
state.DataSizeReceived = true;
state.Data.Position = 0;
state.Data.SetLength(0);
}
else
{ //we received just part of the headers information
//issue another read
if (!state.Client.ReceiveAsync(state.ReadEventArgs))
ProcessReceive(state.ReadEventArgs);
return;
}
}
//at this point we know the size of the pending data
if ((state.Data.Length + dataRead) >= state.DataSize)
{ //we have all the data for this message
restOfData = state.DataSize - (int)state.Data.Length;
state.Data.Write(state.Buffer, dataOffset, restOfData);
Console.WriteLine("Data message received. Size: {0}",
state.DataSize);
dataOffset += restOfData;
dataRead -= restOfData;
state.Data.SetLength(0);
state.Data.Position = 0;
state.DataSizeReceived = false;
state.DataSize = 0;
if (dataRead == 0)
{
if (!state.Client.ReceiveAsync(state.ReadEventArgs))
ProcessReceive(state.ReadEventArgs);
return;
}
else
continue;
}
else
{ //there is still data pending, store what we've
//received and issue another BeginReceive
state.Data.Write(state.Buffer, dataOffset, dataRead);
if (!state.Client.ReceiveAsync(state.ReadEventArgs))
ProcessReceive(state.ReadEventArgs);
dataRead = 0;
}
}
}
I did not do it exactly this way myself but it helped.