Develop Reference

tcp server failing after first loop

As above really, I'm trying to create a tcp linux server in c to accept data, perform some processing then send it back.
The code I'm trying to use on the client side to send the data and then read it back:
TcpClient tcpclnt = new TcpClient();
tcpclnt.Connect("192.168.0.14", 1235);
NetworkStream stm = tcpclnt.GetStream();
_signal.WaitOne();
Image<Bgr, Byte> frame = null;
while (_queue.TryDequeue(out frame))
{
if (frame != null)
{
resizedBMPFrame = frame.Resize(0.5, Emgu.CV.CvEnum.INTER.CV_INTER_LINEAR).ToBitmap();
using (MemoryStream ms = new MemoryStream())
{
resizedBMPFrame.Save(ms, ImageFormat.Bmp);
byte[] byteFrame = ms.ToArray();
l = byteFrame.Length;
byte[] buf = Encoding.UTF8.GetBytes(l.ToString());
stm.Write(buf, 0, buf.Length);
stm.Write(byteFrame, 0, byteFrame.Length);
}
}
else
{
Reading = false;
}
int i;
Bitmap receivedBMPFrame;
byte[] receivedFramesize = new byte[4];
int j = stm.Read(receivedFramesize, 0, receivedFramesize.Length);
int receivedFramesizeint = BitConverter.ToInt32(receivedFramesize, 0);
byte[] receivedFrame = new byte[receivedFramesizeint];
j = stm.Read(receivedFrame, 0, receivedFrame.Length);
using (MemoryStream ms = new MemoryStream(receivedFrame))
{
receivedBMPFrame = new Bitmap(ms);
if (receivedBMPFrame != null)
{
outputVideoPlayer.Image = receivedBMPFrame;
}
else
{
Reading = false;
}
}
}
}
stm.Close();
tcpclnt.Close();
So the idea is it waits for the display thread to send the current frame it's displaying using a concurrentqueue, it then takes it, and makes it a quarter of the size, converts it to a byte array and then sends its length and then it itself over the tcp socket.
In theory the server gets it, performs some processing then sends it back, so it reads the length of it then the new frame itself.
The server code is below:
while (1)
{
int incomingframesize;
int n;
n = read(conn_desc, framesizebuff, 6);
if ( n > 0)
{
printf("Length of incoming frame is %s\n", framesizebuff);
}
else
{
printf("Failed receiving length\n");
return -1;
}
char framebuff[atoi(framesizebuff)];
n = read(conn_desc, framebuff, atoi(framesizebuff));
if ( n > 0)
{
printf("Received frame\n");
}
else
{
printf("Failed receiving frame\n");
return -1;
}
printf("Ready to write\n");
int k = sizeof(framebuff);
n = write(conn_desc, &k, sizeof(int));
if (n <0)
{
printf("ERROR writing to socket\n");
}
else
{
printf("Return frame size is %d\n", k);
}
n = write(conn_desc, &framebuff, sizeof(char)*k);
if (n <0)
{
printf("ERROR writing to socket\n");
}
frameno++;
printf("Frames sent: %d\n", frameno);
}
So it reads the length, then the actual frame, which seems to work, and at the moment then just sends it straight back without doing any processing. However it only works for one loop seemingly, if I step through the client code line by line the server code runs through once, but on the 2nd read by the client, receiving the frame from the server, the server then runs the two reads of the loop straight away, without waiting for another write. Failing on the 2nd having seemingly read in nothing as it outputs:
Length of incoming frame is
Failed receiving frame
With no number, which to me makes sense as I haven't sent another write with the length of the next frame. I'm just wondering what I'm missing/why it's acting like this? As on the first loop it waits until the write commands from the client. I'm wondering if it means there is left over data in the write stream, so when it goes back to the top it immediately reads it again? Although it then doesn't print any form of number which to me implies there's nothing there...
Any help would be greatly appreciated.
EDIT/UPDATE:
Changed the read/write sections on the server to do a single byte at a time like this:
while (ntotal != incomingframesize)
{
n = read(conn_desc, &framebuff[ntotal], sizeof(char));
ntotal = ntotal + n;
while (i < k)
{
m = write(conn_desc, &framebuff[i], sizeof(char));
Which seems to have solved the problems I was having and now the correct data is being transferred :)

When the client writes the frame size it uses the length of some object, but when the server reads it always tries to read 6 characters. You need to use a fixed length for the frame size!
When reading, you cannot assume that you get as many bytes as you ask for. The return value, if >0, is the number of bytes actually read. If you get less than you asked for, you need to keep reading until you have received the number of bytes you expect.
First read until you've got 6 bytes (frame size).
Next read until you've got the number of bytes indicated by the frame size.
Make sure you use the same number of bytes for the frame size in all places.
Edit:
I also noted a bug in the call to write in the server:
n = write(conn_desc, &framebuff, sizeof(char)*k);
framebuff is a pointer to the data, so you probably mean:
n = write(conn_desc, &framebuff[0], k);

Network Streams - Amount to read per Read

Im currently a bit stuck with my c# project.
I have 2 applications, they both have a common class definition I call a NetMessage
a NetMessage contains a MessageType string property, as well as 2 List lists.
The idea is that I can pack this class with classes, and data to send across the network as a byte[].
Because Network Streams do not advertise the amount of data they are receiving, I modified my Send method to send the size of the NetMessage byte[] ahead of the actual byte[].
private static byte[] ReceivedBytes(NetworkStream MainStream)
{
try
{
//byte[] myReadBuffer = new byte[1024];
int receivedDataLength = 0;
byte[] data = { };
long len = 0;
int i = 0;
MainStream.ReadTimeout = 60000;
//MainStream.CanTimeout = false;
if (MainStream.CanRead)
{
//Read the length of the incoming message
byte[] byteLen = new byte[8];
MainStream.Read(byteLen, 0, 8);
len = BitConverter.ToInt64(byteLen, 0);
data = new byte[len];
//data is now set to the appropriate size for the expected message
//While we have not got the full message
//Read each individual byte and append to data.
//This method, seems to work, but is ridiculously slow,
while (receivedDataLength < data.Length)
{
receivedDataLength += MainStream.Read(data, receivedDataLength, 1);
}
//receivedDataLength += MainStream.Read(data, receivedDataLength, data.Length);
return data;
}
}
catch (Exception E)
{
//System.Windows.Forms.MessageBox.Show("Exception:" + E.ToString());
}
return null;
}
I have tried to change the size argument below to something like 1024 or to be the data.Length, but I get funky results.
receivedDataLength += MainStream.Read(data, receivedDataLength, 1);
setting it to data.Length seems to cause problems when the Class being sent is a few mb in size.
Setting the buffer size to be 1024 like I have seen in other examples, causes failures when the size of the incoming message is small, like 843 bytes, it errors out saying that I tried to read out of bounds or something.
Below is the type of method being used to send the data in the first place.
public static void SendBytesToStream(NetworkStream TheStream, byte[] TheMessage)
{
//IAsyncResult r = TheStream.BeginWrite(TheMessage, 0, TheMessage.Length, null, null);
// r.AsyncWaitHandle.WaitOne(10000);
//TheStream.EndWrite(r);
try
{
long len = TheMessage.Length;
byte[] Bytelen = BitConverter.GetBytes(len);
TheStream.Write(Bytelen, 0, Bytelen.Length);
TheStream.Flush();
// <-- I've tried putting thread sleeps in this spot to see if it helps
//I've also tried writing each byte of the message individually
//takes longer, but seems more accurate as far as network transmission goes?
TheStream.Write(TheMessage, 0, TheMessage.Length);
TheStream.Flush();
}
catch (Exception e)
{
//System.Windows.Forms.MessageBox.Show(e.ToString());
}
}
I'd like to get these two methods setup to the point where they are reliably sending and receiving data.
The application I am using this for, monitors a screenshots folder in a game directory,
when it detects a screenshot in TGA format, it converts it to PNG, then takes its byte[] and sends it to the receiver.
The receiver then posts it to Facebook (I don't want my FB tokens distributed in my client application), hence the server / proxy idea.
Its strange, but when I step through the code, the transfer is invariably successful.
But if I run it full speed, no breakpoint, it typically tells me that the connection was closed by the remote host etc.
The client typically finishes sending the data almost instantly, even though its a 4mb file.
The receiver spends about 2 minutes reading from the Network Stream, which doesnt make sense, if the client finished sending the data, does that mean the data is just floating in cyber space, and being pulled down?
Surely it should be synchronous?

I suspect I know where my code was going wrong.
It turns out that the scope I was creating my TCPClient that was doing the sending, was declared and instantiated within a method.
This being the case, the GAC was disposing of it, even though the Receiving Server had not finished downloading the stream.
I managed to resolve it by creating a method that can detect when the Client has disconnected on the server end, and until it has actually disconnected, it will keep looping/waiting until disconnected.
This way, we are waiting until the server lets go of us.

Proper way to receive network data in loop

I am writing a C# client application which will connect to the server written in python. My question is about receiving data in loop. The application structure is all about client asks server -> server responds to client. Everything works fine when the message is lower that actual buffer size (set in server). For example: server side buffer: 1024, client buffer size: 256, data length < 1kb. I run my application with following code:
int datacounter = 0;
byte[] recived = new byte[256];
StringBuilder stb = new StringBuilder();
serverStream.ReadTimeout = 1500;
try
{
while ((datacounter = serverStream.Read(recived, 0, 256)) > 0)
{
Console.WriteLine("RECIVED: " + datacounter.ToString());
stb.append(System.Text.Encoding.UTF8.GetString(recived, 0, datacounter));
}
}
catch { Console.WriteLine("Timeout!"); }
Then the application receives data in 4 loops (256 bytes each):
RECIVED: 256
RECIVED: 256
RECIVED: 256
RECIVED: 96
And then the timeout ticks, that ends the transmission and pass the complete data to later analysis (from stb object). I don't think using timeout is proper, but i don't know any other way to do this.
However, this way it works. Here we go with example, that does not:
server side buffer: 1024, client side buffer: 256, data length ~ 8kbytes (python side sends data in loop).
RECIVED: 256
RECIVED: 256
RECIVED: 256
RECIVED: 256
Then the timeout ticks (and obviosly the data is incomplete - got 1kb of 8kb). Sometimes the loop even ends after 1 run with 28 recived bytes and thats all before timeout. Python says that the data has been send properly. Here's the way i create the socket and serverStream object:
TcpClient clientSocket = new TcpClient();
clientSocket.Connect("x.y.z.x", 1234);
NetworkStream serverStream = clientSocket.GetStream();
Its not the TcpClient fault. Tried the same with clear sockets, created like:
new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)
However that works similar. Is there a way, to make my loop work without timeout, receiving all data? I would like to keep the socket synchronous if possible.

I don't think there's anything wrong with your receive code functionally. I put together a test, and the receiver gets as much as you can send it (eg. 8 MBs), as long as you keep sending without 1.5 seconds pause, before timing out.
So it looks like your server is simply not sending "fast" enough.
To answer your question, timing is not the typical way of knowing when you have received a full message. One common, simple way of determining when a full message is received is to prefix the length of the full message on the sending side (eg. 4-byte int). Then on the receive side, first read 4 bytes, decode to the length, and then read that many more bytes.
You could also consider appending a message termination string, such as Environment.NewLine, to the end of your message. This has the advantage that you could call StreamReader.ReadLine(), which will block until the full message is received. This only works if the termination can NOT be included in the message itself.
If you can't alter the server protocol, is there any other way of knowing you have received a full message? (eg. checking for a NewLine at the end of the message, an XML end tag, or some other pattern.) If not, perhaps you could wait for the server to disconnect, otherwise it looks like you would be forced to find the right timing balance.
I am including the test code below in case you want to play around with it.
Server/Sending Side:
IPAddress localAddr = IPAddress.Parse("127.0.0.1");
TcpListener server = new TcpListener(localAddr, 13579);
server.Start();
TcpClient clientSocket = server.AcceptTcpClient();
NetworkStream stream = clientSocket.GetStream();
int bytesSent = 0;
int bytesToSend = 1 << 25;
int bufferSize = 1024;
string testMessage = new string('X', bufferSize);
byte[] buffer = UTF8Encoding.UTF8.GetBytes(testMessage);
while (bytesSent < bytesToSend)
{
int byteToSendThisRound = Math.Min(bufferSize, bytesToSend - bytesSent);
stream.Write(buffer, 0, byteToSendThisRound);
bytesSent += byteToSendThisRound;
}
Client/Receiving Side:
TcpClient client = new TcpClient("127.0.0.1", 13579);
NetworkStream serverStream = client.GetStream();
int totalBytesReceived = 0;
int datacounter = 0;
byte[] recived = new byte[256];
StringBuilder stb = new StringBuilder();
serverStream.ReadTimeout = 1500;
try
{
while ((datacounter = serverStream.Read(recived, 0, 256)) > 0)
{
totalBytesReceived += 256;
Console.WriteLine("RECIVED: {0}, {1}", datacounter, totalBytesReceived);
stb.Append(System.Text.Encoding.UTF8.GetString(recived, 0, datacounter));
}
}
catch { Console.WriteLine("Timeout!"); }

Why don't you dump the exception that makes your code go into the catch branch and find out? :)
catch (Exception ex) { Console.WriteLine("Timeout because of... " + ex.Message); }
--EDIT
Sorry I didn't see the timout. The question you're asking is if there's a way to do it without timeout. Yes, don't set any timeout and check if received number of bytes is smaller than the buffer size.
That is:
while ((datacounter = serverStream.Read(recived, 0, 256)) > 0)
{
Console.WriteLine("RECIVED: " + datacounter.ToString());
stb.append(System.Text.Encoding.UTF8.GetString(recived, 0, datacounter));
if(datacounter < 256) //you're good to go
break;
}

For anyone else who needs help with this
Just to add to Chi_Town_Don's answer, make sure you use stb.ToString() outside of the loop. And I've found that nothing will print out unless the loop breaks out. To do that if(!serverStream.DataAvailable()){break;} works wonders. That way you don't need to pass in the packet size or some other convoluted condition.

C# TCP file transfer - Images semi-transferred

I am developing a TCP file transfer client-server program. At the moment I am able to send text files and other file formats perfectly fine, such as .zip with all contents intact on the server end. However, when I transfer a .gif the end result is a gif with same size as the original but with only part of the image showing as if most of the bytes were lost or not written correctly on the server end.
The client sends a 1KB header packet with the name and size of the file to the server. The server then responds with OK if ready and then creates a fileBuffer as large as the file to be sent is.
Here is some code to demonstrate my problem:
// Serverside method snippet dealing with data being sent
while (true)
{
// Spin the data in
if (streams[0].DataAvailable)
{
streams[0].Read(fileBuffer, 0, fileBuffer.Length);
break;
}
}
// Finished receiving file, write from buffer to created file
FileStream fs = File.Open(LOCAL_FOLDER + fileName, FileMode.CreateNew, FileAccess.Write);
fs.Write(fileBuffer, 0, fileBuffer.Length);
fs.Close();
Print("File successfully received.");
// Clientside method snippet dealing with a file send
while(true)
{
con.Read(ackBuffer, 0, ackBuffer.Length);
// Wait for OK response to start sending
if (Encoding.ASCII.GetString(ackBuffer) == "OK")
{
// Convert file to bytes
FileStream fs = new FileStream(inPath, FileMode.Open, FileAccess.Read);
fileBuffer = new byte[fs.Length];
fs.Read(fileBuffer, 0, (int)fs.Length);
fs.Close();
con.Write(fileBuffer, 0, fileBuffer.Length);
con.Flush();
break;
}
}
I've tried a binary writer instead of just using the filestream with the same result.
Am I incorrect in believing successful file transfer to be as simple as conversion to bytes, transportation and then conversion back to filename/type?
All help/advice much appreciated.

Its not about your image .. It's about your code.
if your image bytes were lost or not written correctly that's mean your file transfer code is wrong and even the .zip file or any other file would be received .. It's gonna be correpted.
It's a huge mistake to set the byte buffer length to the file size. imagine that you're going to send a large a file about 1GB .. then it's gonna take 1GB of RAM .. for an Idle transfering you should loop over the file to send.
This's a way to send/receive files nicely with no size limitation.
Send File
using (FileStream fs = new FileStream(srcPath, FileMode.Open, FileAccess.Read))
{
long fileSize = fs.Length;
long sum = 0; //sum here is the total of sent bytes.
int count = 0;
data = new byte[1024]; //8Kb buffer .. you might use a smaller size also.
while (sum < fileSize)
{
count = fs.Read(data, 0, data.Length);
network.Write(data, 0, count);
sum += count;
}
network.Flush();
}
Receive File
long fileSize = // your file size that you are going to receive it.
using (FileStream fs = new FileStream(destPath, FileMode.Create, FileAccess.Write))
{
int count = 0;
long sum = 0; //sum here is the total of received bytes.
data = new byte[1024 * 8]; //8Kb buffer .. you might use a smaller size also.
while (sum < fileSize)
{
if (network.DataAvailable)
{
{
count = network.Read(data, 0, data.Length);
fs.Write(data, 0, count);
sum += count;
}
}
}
}
happy coding :)

When you write over TCP, the data can arrive in a number of packets. I think your early tests happened to fit into one packet, but this gif file is arriving in 2 or more. So when you call Read, you'll only get what's arrived so far - you'll need to check repeatedly until you've got as many bytes as the header told you to expect.
I found Beej's guide to network programming a big help when doing some work with TCP.

As others have pointed out, the data doesn't necessarily all arrive at once, and your code is overwriting the beginning of the buffer each time through the loop. The more robust way to write your reading loop is to read as many bytes as are available and increment a counter to keep track of how many bytes have been read so far so that you know where to put them in the buffer. Something like this works well:
int totalBytesRead = 0;
int bytesRead;
do
{
bytesRead = streams[0].Read(fileBuffer, totalBytesRead, fileBuffer.Length - totalBytesRead);
totalBytesRead += bytesRead;
} while (bytesRead != 0);
Stream.Read will return 0 when there's no data left to read.
Doing things this way will perform better than reading a byte at a time. It also gives you a way to ensure that you read the proper number of bytes. If totalBytesRead is not equal to the number of bytes you expected when the loop is finished, then something bad happened.

Thanks for your input Tvanfosson. I tinkered around with my code and managed to get it working. The synchronicity between my client and server was off. I took your advice though and replaced read with reading a byte one at a time.

In c# client/server program Data that sent from server is changed when received at client

Server side code:
byte[] size = new byte[4];
size = BitConverter.GetBytes(fileData.Length);
stream.Write(size, 0, 4);
Client side code:
byte[] size = new byte[4];
ReadWholeArray(s, size);
int fileSize = BitConverter.ToInt32(size, 0);
Definition of ReadWholeArray method:
public static void ReadWholeArray(Stream stream, byte[] data)
{
int offset = 0;
int remaining = data.Length;
while (remaining > 0)
{
int read = stream.Read(data, offset, remaining);
if (read <= 0)
throw new EndOfStreamException
(String.Format("End of stream reached with {0} bytes left to read", remaining));
remaining -= read;
offset += read;
}
}
The Program sends fileData.Length (the value for this instance is 2422) from server. On receiving this data at client side the value of received data is -772097985
Why the sent data is not received without alteration in value? What is the problem?

Okay, simple diagnostics to start with: log the individual contents of the byte array at both ends, so you can see what's going on there.
That way you can see if it's the binary data which is getting corrupted in your communication protocol, or whether BitConverter is causing your problem. For example, you could have a big-endian BitConverter at one end, and a little-endian BitConverter at the other. That seems unlikely, but it's possible - particularly if one of your server or client is running Mono rather than .NET itself.
If that does turn out to be the problem, you might want to use my EndianBitConverter class from MiscUtil, which lets you specify the endianness.
If the problem is in the communications layer, you quite possibly want to install Wireshark to see what's happening at the network level. Are you sure you've read all the data you're meant to have read so far, for example? (If you've only read 15 bytes before this, and the size is written at offset 16, then obviously you'll get the "extra" byte first.)

This works fine:
private void button2_Click(object sender, RoutedEventArgs e)
{
MemoryStream ms = new MemoryStream();
byte [] original = BitConverter.GetBytes((int)2224); // 176, 8, 0, 0
ms.Write(original, 0, original.Length);
ms.Seek(0, SeekOrigin.Begin);
byte [] data = new byte[4];
int count = ms.Read(data, 0, 4); // count is 4, data is 176, 8, 0, 0
int fileSize = BitConverter.ToInt32(data, 0); // is 2224
return;
}
Can you use WireShark or something to intercept the bytes? What kind of connection are you using? Could more data be being sent (i.e. telnet control characters at the start of the stream)? Can you debug each end and verify these values or write the byte array contents to a log file? By the way calling this: "byte[] size = new byte[4];" is wasteful because BitConverter.GetBytes() returns a new array.