I want to communicate with a DSP using RS232, so I use System.IO.SerialPort to achieve this. Everything goes well except the reading performance.
Every 200ms, the port can received a package of 144 bytes. But in the tests, the applications almost skip every other package. I try to print the system time in the console. It amaze me that the code below (when length = 140) take me over 200ms. It let the application can not handle the data in time.
Does anything wrong I do?
Port Property:
BaudRate = 9600
Parity = None
StopBits = One
private byte[] ReadBytesInSpicifiedLength(int length)
{
byte[] des = new byte[length];
for (int i = 0; i < length; i++)
{
des[i] = (byte)serialPort.ReadByte();
}
return des;
}
You're doing a lot of individual I/O calls, which means a lot of kernel transitions. Those are expensive. Not being able to reach 720 bytes per second is surprising, but you can make the data handling an order of magnitude faster by doing block reads:
private byte[] ReadBytesWithSpecifiedLength(int length)
{
byte[] des = new byte[length];
serialPort.BaseStream.Read(des, 0, des.Length);
return des;
}
If you have timeouts enabled, you could get partial reads. Then you need to do something like:
private byte[] ReadBytesWithSpecifiedLength(int length)
{
byte[] des = new byte[length];
int recd = 0;
do {
int partial = serialPort.BaseStream.Read(des, recd, length - recd);
if (partial == 0) throw new IOException("Transfer Interrupted");
recd += partial;
} while (recd < length);
return des;
}
The nice thing about BaseStream is that it also has async support (via ReadAsync). That's what new C# code should be using.
Related
What I'm doing is taking a user entered string, creating a packet with the data, then sending the string out to a serial port. I am then reading the data I send via a loopback connector. My send is working flawlessly, however my receive is randomly throwing an arithmetic overflow exception.
I say randomly because it is not happening consistently. For example, I send the message "hello" twice. The first time works fine, the second time outputs nothing and throws an exception. I restart my program, run the code again, and send hello only to receive "hell" and then an exception. On rare occasion, I'll receive the packet 3 or 4 times in a row without error before the exception.
Here is my relevant code:
public void receivePacket(object sender, SerialDataReceivedEventArgs e)
{
byte[] tempByte = new byte[2];
int byteCount = 0;
while (serialPort1.BytesToRead > 0)
{
if (byteCount <= 1)
{
tempByte[byteCount] = (byte)serialPort1.ReadByte();
}
if (byteCount == 1)
{
receivedString = new byte[tempByte[byteCount]];
receivedString[0] = tempByte[0];
receivedString[1] = tempByte[1];
}
else if (byteCount > 1)
{
byte b = (byte)serialPort1.ReadByte();
receivedString[byteCount] = b;
}
byteCount++;
}
int strLen = (byteCount - 3);
tempByte = new byte[strLen];
int newBit = 0;
for (int i = 2; i <= strLen+1; i++)
{
tempByte[newBit] = receivedString[i];
newBit++;
}
string receivedText = encoder.GetString(tempByte);
SetText(receivedText.ToString());
}
I'm well aware that my implementation using byteCount (which I use to traverse the byte array) is rather sloppy. When I step through the code, I find that when I get the error byteCount == 1, which is making strLen a negative number (since strLen is byteCount - 3, which is done because the packet contains a header, length, and CRC i.e. byteCount - 3 == # of actual data bytes received). This leads to by tempByte having a size of -2, which throws my exceptions. I, however, am having a very hard time figuring out why byteCount is being set to 1.
The code after this basically just traverses the data section of the array, copies it into the tempByte, then is sent off to a function to append the text in another thread.
I am guessing that byteCount is 1 because you only received one byte - or rather, you processed the first byte before the second one arrived in the buffer.
The ReadByte function will wait for a certain amount of time for a byte to arrive if there isn't one waiting.
Maybe if instead of checking BytesToRead, you did something more like this:
byte headerByte = serialPort1.ReadByte();
byte length = serialPort1.ReadByte();
receivedString = new byte[length];
receivedString[0] = headerByte;
receivedString[1] = length;
for (int i = 2; i < length; i++) {
receivedString[i] = serialPort1.ReadByte();
}
I am doing some data chunking and I'm seeing an interesting issue when sending binary data in my response. I can confirm that the length of the byte array is below my data limit of 4 megabytes, but when I receive the message, it's total size is over 4 megabytes.
For the example below, I used the largest chunk size I could so I could illustrate the issue while still receiving a usable chunk.
The size of the binary data is 3,040,870 on the service side and the client (once the message is deserialized). However, I can also confirm that the byte array is actually just under 4 megabytes (this was done by actually copying the binary data from the message and pasting it into a text file).
So, is WCF causing these issues and, if so, is there anything I can do to prevent it? If not, what might be causing this inflation on my side?
Thanks!
The usual way of sending byte[]s in SOAP messages is to base64-encode the data. This encoding takes 33% more space than binary encoding, which accounts for the size difference almost precisely.
You could adjust the max size or chunk size slightly so that the end result is within the right range, or use another encoding, e.g. MTOM, to eliminate this 33% overhead.
If you're stuck with soap, you can offset the buffer overhead Tim S. talked about using the System.IO.Compression library in .Net - You'd use the compress function first, before building and sending the soap message.
You'd compress with this:
public static byte[] Compress(byte[] data)
{
MemoryStream ms = new MemoryStream();
DeflateStream ds = new DeflateStream(ms, CompressionMode.Compress);
ds.Write(data, 0, data.Length);
ds.Flush();
ds.Close();
return ms.ToArray();
}
On the receiving end, you'd use this to decompress:
public static byte[] Decompress(byte[] data)
{
const int BUFFER_SIZE = 256;
byte[] tempArray = new byte[BUFFER_SIZE];
List<byte[]> tempList = new List<byte[]>();
int count = 0;
int length = 0;
MemoryStream ms = new MemoryStream(data);
DeflateStream ds = new DeflateStream(ms, CompressionMode.Decompress);
while ((InlineAssignHelper(count, ds.Read(tempArray, 0, BUFFER_SIZE))) > 0) {
if (count == BUFFER_SIZE) {
tempList.Add(tempArray);
tempArray = new byte[BUFFER_SIZE];
} else {
byte[] temp = new byte[count];
Array.Copy(tempArray, 0, temp, 0, count);
tempList.Add(temp);
}
length += count;
}
byte[] retVal = new byte[length];
count = 0;
foreach (byte[] temp in tempList) {
Array.Copy(temp, 0, retVal, count, temp.Length);
count += temp.Length;
}
return retVal;
}
I have to trnasfer large file, at this moment I use TcpClient and NetworkStream class to do that, but it isn't enought. When I use NetworkStream.Write and Read it lose bytes I don't know how can I ensure that TCP will stop reading when all bytes will be ok.
Then I found Socket Class, and now my question, "Socket will be perfectly sure about all bytes before it end read?"
How I recive file:
byte[] VideoFrom = new byte[FizeSizeSendedBefore];
byte[] PartData = new byte[clientSocket.ReceiveBufferSize];
int PartNumbers = (VideoSize / clientSocket.ReceiveBufferSize) + 1;
int lastPackageSize = VideoSize - ((PartNumbers - 1) * clientSocket.ReceiveBufferSize);
int i;
int bytesToRead = 0;
int ActualSize = 0;
for (i = 0; i < PartNumbers; i++)
{
if (i < PartNumbers - 1)
{
bytesToRead = clientSocket.ReceiveBufferSize;
}
else
{
bytesToRead = lastPackageSize;
}
ActualSize += bytesToRead;
PartData = new byte[bytesToRead];
System.Threading.Thread.Sleep(2);
networkStream.Read(PartData, 0, bytesToRead);
Buffer.BlockCopy(PartData, 0, VideoFrom, i * clientSocket.ReceiveBufferSize, bytesToRead);
}
if (!Directory.Exists("./temp/anwsers/" + AnwserXML.Attribute("number").Value)) { Directory.CreateDirectory("./temp/anwsers/" + AnwserXML.Attribute("number").Value); }
File.WriteAllBytes("./temp/anwsers/" + AnwserXML.Attribute("number").Value + "/" + AnwserXML.Attribute("client").Value + ".mov", VideoFrom);
}
How I send file
int PartNumber = (FizeSizeSendedBefore/ clientSocket.ReceiveBufferSize) + 1;
int lastPackageSize = FileSize - ((PartNumber - 1) * clientSocket.ReceiveBufferSize);
int i;
for (i = 0; i < PartNumber; i++)
{
if (i < PartNumber - 1)
{
while (!serverStream.CanRead) { }
serverStream.Write(outStream, i * clientSocket.ReceiveBufferSize, clientSocket.ReceiveBufferSize);
}
else
{
while (!serverStream.CanRead) { }
serverStream.Write(outStream, i * clientSocket.ReceiveBufferSize, lastPackageSize);
}
}
This is the problem:
networkStream.Read(PartData, 0, bytesToRead);
Never ignore the return value of Read. Never assume that a single call to read will read all of the data. You need to loop round, reading until it's all "in". The number of calls to Read which are required to read all data is pretty much unrelated to the number of calls to Write. TCP is a stream protocol - treat it that way.
It's not clear whether you know exactly how much data you're expecting to read - do you? Will the server close the connection at the end? If so, and if you're using .NET 4, then you can get rid of a huge amount of this code:
using (Stream output = File.Create(filename))
{
networkStream.CopyTo(output);
}
TCP will receive a chunk of data at a time - it will not lose data, but if you do not cater for the fact that data will be spilt into packets and will arrive one after the other and (often) not as one big "file", then you may not be listening for the rest of the data - this will seem like it is missing data, but really it is the code ignoring the rest after the first packet is received. Just because you sedn it in one lumnp, this does not mean it will arrive in one lump.
There are many examples of TCP server/client apps (and chat apps) out there in the wonderful world of google for you to compare notes on - alternatively post your code and we can then see where the error is.
How large is your file? Maybe it will be enough WebClient.DownloadDataAsync(...)?
UPDATED
If file is about 100Mb use simple WebClient :)
WebClient wc = new WebClient();
wc.DownloadDataCompleted += new DownloadDataCompletedEventHandler(OnDownloadDataCompleted);
wc.DownloadProgressChanged += new DownloadProgressChangedEventHandler(OnDownloadProgressChanged);
wc.DownloadDataAsync(new Uri("http://www.xxxx.com/t.tar.gz"));
I have a C# client/server network program I've written using
TCPListener and TCPClient classes. The server is reading everything
from the client (small amounts of xml) just fine until I try to send a
large file (100k) back to the client.
I'm using stream functions for
both client and server with non-blocking socket functions. When I do a
socket.SendFile("filename") back to the client, the file is getting
cut off - I've set the receive buffer size on the client to well past
100k but it still gets cut off around 25k and the communication
between client and server is unreliable afterwords.
My basic question
is what happens if data is somehow left in the pipe ? i.e.. will it be
read by the next socket.Read... Does every Send call require exactly
one and only one Read ? Maybe I'm not giving the client enough time to
read the file but their both on the same machine and I've tried
sleeping for a few seconds in various places w/o success.
It is very possible that you cannot read the entire message through one Read call (perhaps all data has not arrived yet). In network programming, you would often place the call to Read in a while loop and simply Read() until you have received the entire expected message.
1 Send call might take more than one Read call to receive, and 1 Read call might read the data send by several Send call.
TCP just provides a stream, so it's up to you to define how the data or messages you send are partitioned.
In this case, you probably just need to loop ,doing Read until the stream is closed.
You probably want something like this:
socket.Blocking = false;
const int ChunkSize = 1492;
const int ReceiveTimeout = 10000;
const int SendTimeout = 10000;
public void Send(Byte[] data)
{
var sizeBytes = BitConverter.GetBytes(data.Length);
SendInternal(sizeBytes);
SendInternal(data);
}
public Byte[] Receive()
{
var sizeBytes = ReceiveInternal(4);
var size = BitConverter.ToInt32(sizeBytes, 0);
var data = ReceiveInternal(size);
return data;
}
private void SendInternal(Byte[] data)
{
var error = SocketError.Success;
var lastUpdate = Environment.TickCount;
var size = data.Length;
var count = 0;
var sent = 0;
while (sent < size)
{
count = Math.Min(ChunkSize, size - sent);
count = socket.Send(data, sent, count, SocketFlags.None, out error);
if (count > 0)
{
sent += count;
lastUpdate = Environment.TickCount;
}
if (error != SocketError.InProgress && error != SocketError.Success && error != SocketError.WouldBlock)
throw new SocketException((Int32)error);
if (Environment.TickCount - lastUpdate > SendTimeout)
throw new TimeoutException("Send operation timed out.");
if (count == 0 && !socket.Poll(100, SelectMode.SelectWrite))
throw new SocketException((Int32)SocketError.Shutdown);
}
}
private Byte[] ReceiveInternal(Int32 size)
{
var error = SocketError.Success;
var lastUpdate = Environment.TickCount;
var buffer = new Byte[ChunkSize];
var count = 0;
var received = 0;
using (var ms = new MemoryStream(size))
{
while (received < size)
{
count = Math.Min(ChunkSize, size - received);
count = socket.Receive(buffer, 0, count, SocketFlags.None, out error);
if (count > 0)
{
ms.Write(buffer, 0, count);
received += count;
lastUpdate = Environment.TickCount;
}
if (error != SocketError.InProgress && error != SocketError.Success && error != SocketError.WouldBlock)
throw new SocketException((Int32)error);
if (Environment.TickCount - lastUpdate > ReceiveTimeout)
throw new TimeoutException("Receive operation timed out.");
if (count == 0 && socket.Poll(100, SelectMode.SelectRead) && socket.Available == 0)
throw new SocketException((Int32)SocketError.Shutdown);
}
return ms.ToArray();
}
}
What I would usually do is create a header structure that is sent
Header Size (int, 4 bytes)
File Name Offset (int, 4 bytes)
File Name Size (int , 4 bytes)
File Data Offset (int, 4 bytes)
File Data Size (int , 4 bytes)
[ message data here]
and then that header is read using either a BinaryReader or copying the bytes to a struct using Marshal. This way you always know what data is arriving and how many times you need to call Read().
The header size field is also helps with versioning the protocol (keep the structure the same but add to it for later clients so you can keep backwards compatibility). If you define the structure in C# be sure to do it like so:
[StructLayout LayoutKind.Sequential]
struct MessageHeader
{
public int HeaderSize;
public int FileNameOffset;
public int FileNameSize;
public int FileDataOffset;
public int FileDataSize;
}
Then Marshal.PtrToStructure will allow you do create an instance of this struct right from the byte[] you read from the socket
Try sending the chunk from the server side in chunks. Just as the other said, posting the code would be of great help to us.
After enough playing with asynchronous socket programming I noticed that the server was receiving chunked payloads (ie: more than one complete payload sitting in the same buffer). So I came up with the following:
if (bytes_to_read > 0)
{
while (bytes_to_read > 0)
// Get payload size as int.
// Get payload in byte format.
// Do something with payload.
// Decrease the amount of bytes to read.
}
// Wait for more data.
}
And then I noticed packet fragmentation (ie: what I thought were complete payloads chunked together wasn't always so) which changed the previous code to something like:
if (bytes_to_read > 0)
{
while (bytes_to_read > 0)
{
// Get payload size as int.
// Check if the payload size is less than or equal to the amount of bytes left to read.
if (payload_size <= bytes_to_read)
{
// Get payload in byte format.
// Do something with payload.
// Decrease the amount of bytes to read.
}
else
{
// We received a fragmented payload.
break;
}
}
if (bytes_to_read == 0)
{
// Wait for more data.
}
else if (bytes_to_read > 0)
{
// Wait for more data where we left off. ***
}
else
{
// Something awful happened.
}
}
*** I don't even know how to go about this and would like to see code for it. I had an idea that it involved copying the in-completed payload to the beginning of the buffer and then picking up from there.
The pseudo code I included is based on the Begin* End* method I am using (I'm aware that I should be using the *Async set of methods found here -> http://msdn.microsoft.com/en-us/library/system.net.sockets.socketasynceventargs.aspx <- but I think my overall question still applies).
I am seeking the answers to 2 questions--namely:
Is this approach correct or am I
missing something?
What does a working example of
dealing with packet fragmentation in
C# look like?
EDIT: I'm using raw sockets.
Thanks in advance for all your help.
EDIT: John Saunders and Greg Hewgill have brought up the point of treating data as a stream but that does not provide me with a concrete example of how to deal with the last chunked payload sometimes being fragmented.
EDIT: I have read Jon Skeet's answer here which is basically along the same lines as the other answers I have seen but it doesn't help me much as I already get what I have to do but not how to do it.
EDIT: To elaborate on what I mean by fragmentation, consider the following the receive buffers:
224TEST3foo3bar
224TEST3foo3bar224TEST3foo3bar
224TEST3foo3bar224TEST3foo
3bar224TEST3foo3bar
EDIT: I found this and this which lead me here. Vadym Stetsiak has cleared nearly everything up (his was one of the answers I was looking for).
This may or may not have anything to do with fragmentation.
In general, the socket will pass you as many bytes at a time as it feels like. Your job is to know how many bytes are in your overall message, and to read them all. Just keep looping until you have all the bytes you need, or until there's an exception.
The following code is untested right now. I thought I'd post it before writing the server side of it and testing both.
private static string ReceiveMessage(Socket socket)
{
const int BUFFER_SIZE = 1024;
var inputBuffer = new byte[BUFFER_SIZE];
var offset = 0;
var bytesReceived = socket.Receive(
inputBuffer, offset, BUFFER_SIZE - offset, SocketFlags.None);
if (bytesReceived < 2)
{
throw new InvalidOperationException("Receive error");
}
var inputMessageLength = inputBuffer[0]*256 + inputBuffer[1];
offset += bytesReceived;
var totalBytesReceived = bytesReceived;
while (bytesReceived > 0 &&
totalBytesReceived < inputMessageLength + 2)
{
bytesReceived = socket.Receive(
inputBuffer, offset, BUFFER_SIZE - offset, SocketFlags.None);
offset += bytesReceived;
totalBytesReceived += bytesReceived;
}
return Encoding.UTF8.GetString(
inputBuffer, 2, totalBytesReceived - 2);
}
Note that the receipt of the message length is wrong. The socket layer could give it to me a byte at a time. I'm going to revisit that as part of a refactoring that will receive the count into a separate two-byte buffer, and change the loop into a single do/while.
When you have to do it yourself, it can be done like so (reference here):
///
/// Server state holds current state of the client socket
///
class AsyncServerState
{
public byte[] Buffer = new byte[512]; //buffer for network i/o
public int DataSize = 0; //data size to be received by the server
//flag that indicates whether prefix was received
public bool DataSizeReceived = false;
public MemoryStream Data = new MemoryStream(); //place where data is stored
public SocketAsyncEventArgs ReadEventArgs = new SocketAsyncEventArgs();
public Socket Client;
}
///
/// Implements server receive logic
///
private void ProcessReceive(SocketAsyncEventArgs e)
{
//single message can be received using several receive operation
AsyncServerState state = e.UserToken as AsyncServerState;
if (e.BytesTransferred <= 0 || e.SocketError != SocketError.Success)
{
CloseConnection(e);
}
int dataRead = e.BytesTransferred;
int dataOffset = 0;
int restOfData = 0;
while (dataRead > 0)
{
if (!state.DataSizeReceived)
{
//there is already some data in the buffer
if (state.Data.Length > 0)
{
restOfData = PrefixSize - (int)state.Data.Length;
state.Data.Write(state.Buffer, dataOffset, restOfData);
dataRead -= restOfData;
dataOffset += restOfData;
}
else if (dataRead >= PrefixSize)
{ //store whole data size prefix
state.Data.Write(state.Buffer, dataOffset, PrefixSize);
dataRead -= PrefixSize;
dataOffset += PrefixSize;
}
else
{ // store only part of the size prefix
state.Data.Write(state.Buffer, dataOffset, dataRead);
dataOffset += dataRead;
dataRead = 0;
}
if (state.Data.Length == PrefixSize)
{ //we received data size prefix
state.DataSize = BitConverter.ToInt32(state.Data.GetBuffer(), 0);
state.DataSizeReceived = true;
state.Data.Position = 0;
state.Data.SetLength(0);
}
else
{ //we received just part of the headers information
//issue another read
if (!state.Client.ReceiveAsync(state.ReadEventArgs))
ProcessReceive(state.ReadEventArgs);
return;
}
}
//at this point we know the size of the pending data
if ((state.Data.Length + dataRead) >= state.DataSize)
{ //we have all the data for this message
restOfData = state.DataSize - (int)state.Data.Length;
state.Data.Write(state.Buffer, dataOffset, restOfData);
Console.WriteLine("Data message received. Size: {0}",
state.DataSize);
dataOffset += restOfData;
dataRead -= restOfData;
state.Data.SetLength(0);
state.Data.Position = 0;
state.DataSizeReceived = false;
state.DataSize = 0;
if (dataRead == 0)
{
if (!state.Client.ReceiveAsync(state.ReadEventArgs))
ProcessReceive(state.ReadEventArgs);
return;
}
else
continue;
}
else
{ //there is still data pending, store what we've
//received and issue another BeginReceive
state.Data.Write(state.Buffer, dataOffset, dataRead);
if (!state.Client.ReceiveAsync(state.ReadEventArgs))
ProcessReceive(state.ReadEventArgs);
dataRead = 0;
}
}
}
I did not do it exactly this way myself but it helped.