Develop Reference

C# Socket Receive: Taking in data until buffer is full dilemma

Here is my code I'm having an issue with:
// read the file in chunks of 5KB
var buffer = new byte[1024*5];
int bytesRead = 0;
do
{
bytesRead = host.Receive(buffer, buffer.Length, 0);
output.Write(buffer, 0, bytesRead);
}
while (bytesRead == buffer.Length);
So here's my problem: I want to read in data as long as the buffer is full. However, even if there is more data, the buffer isn't guaranteed to be filled when sending. This causes the Receive to prematurely exit. If I change the while condition to bytesRead > 0 then it reaches the end of data and Receive blocks until more data is available (which it won't be). How do I solve this?

I think you need to think about how the protocol works and make a more robust solution. Either you go with blocking and then you could either wait until you have no more data to read and then do something else and read again when you know you have more data to read.
Or you add threading and have a seperate thread that just reads and then it doesn't matter if it blocks since it will be on a seperate thread.
Another solution that might be simpler is to use asynchronous reads.
You can read more about it here: https://msdn.microsoft.com/en-us/library/bbx2eya8(v=vs.110).aspx
A simple example from the site above is to start with the read:
private static void Receive(Socket client) {
try {
// Create the state object.
StateObject state = new StateObject();
state.workSocket = client;
// Begin receiving the data from the remote device.
client.BeginReceive( state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
} catch (Exception e) {
Console.WriteLine(e.ToString());
}
}
Then you will get a callback to ReceiveCallback when there is data to be handled:
private static void ReceiveCallback( IAsyncResult ar ) {
try {
// Retrieve the state object and the client socket
// from the asynchronous state object.
StateObject state = (StateObject) ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
if (bytesRead > 0) {
// There might be more data, so store the data received so far.
state.sb.Append(Encoding.ASCII.GetString(state.buffer,0,bytesRead));
// Get the rest of the data.
client.BeginReceive(state.buffer,0,StateObject.BufferSize,0,
new AsyncCallback(ReceiveCallback), state);
} else {
// All the data has arrived; put it in response.
if (state.sb.Length > 1) {
response = state.sb.ToString();
}
// Signal that all bytes have been received.
receiveDone.Set();
}
} catch (Exception e) {
Console.WriteLine(e.ToString());
}
}
I hope this works if not we need to know more about what protocol you use and if there is some reason you need blocking reads so we can give more specific information and help.

Socket read freezes

I'm working on small program for learning purpose.
My program is simple Telnet console.
Using visual studio 2010 Express I created pretty cool UI now I'm trying to establish communication with remote server(Creston control processor). I copied and pasted class from this Microsoft article and when I execute the class the program freezes. I'm not sure how to properly describe what happens but in basic word all controls(including close button) stop working.
Here code for my class:(I added few debug lines);
public class StateObject
{
// Client socket.
public Socket workSocket = null;
// Size of receive buffer.
public const int BufferSize = 256;
// Receive buffer.
public byte[] buffer = new byte[BufferSize];
// Received data string.
public StringBuilder sb = new StringBuilder();
public string TempString = string.Empty;
public int TotalBytesRead = 0;
public char[] charBuffer = new char[1000];
}
public class AsynchronousClient
{
// The port number for the remote device.
private const int port = 23;
// ManualResetEvent instances signal completion.
private static ManualResetEvent connectDone =
new ManualResetEvent(false);
private static ManualResetEvent sendDone =
new ManualResetEvent(false);
private static ManualResetEvent receiveDone =
new ManualResetEvent(false);
// The response from the remote device.
private static String response = String.Empty;
public AsynchronousClient()
{
}
public void New()
{
StartClient();
}
private static void StartClient()
{
// Connect to a remote device.
try
{
// Establish the remote endpoint for the socket.
// The name of the
// remote device is "host.contoso.com".
IPEndPoint remoteEP = new IPEndPoint(IPAddress.Parse("10.106.6.60"), port);
// Create a TCP/IP socket.
Socket client = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);
// Connect to the remote endpoint.
client.BeginConnect(remoteEP,
new AsyncCallback(ConnectCallback), client);
connectDone.WaitOne();
Console.WriteLine("StartSend");
// Send test data to the remote device.
//Send(client, "hostname\n");
//sendDone.WaitOne();
Console.WriteLine("WaitForResponse");
// Receive the response from the remote device.
Receive(client);
receiveDone.WaitOne();
// Write the response to the console.
Console.WriteLine("Read From Socket : {0}", response);
Console.WriteLine("ReleaseSocket");
// Release the socket.
client.Shutdown(SocketShutdown.Both);
client.Disconnect(true);
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
private static void ConnectCallback(IAsyncResult ar)
{
try
{
// Retrieve the socket from the state object.
Socket client = (Socket)ar.AsyncState;
// Complete the connection.
client.EndConnect(ar);
Console.WriteLine("Socket connected to {0}",
client.RemoteEndPoint.ToString());
// Signal that the connection has been made.
connectDone.Set();
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
private static void Receive(Socket client)
{
try
{
// Create the state object.
Console.WriteLine("Receive");
StateObject state = new StateObject();
state.workSocket = client;
// Begin receiving the data from the remote device.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0, new AsyncCallback(ReceiveCallback), state);
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
private static void ReceiveCallback(IAsyncResult ar)
{
try
{
// Retrieve the state object and the client socket
// from the asynchronous state object.
Console.WriteLine("Start Read");
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
Console.WriteLine("Bytes read: {0}", bytesRead);
if (bytesRead > 0)
{
// There might be more data, so store the data received so far.
string tsTempString = Encoding.ASCII.GetString(state.buffer, 0, bytesRead);
state.TempString += tsTempString;
Console.WriteLine("String {0}", tsTempString);
state.sb.Append(Encoding.ASCII.GetString(state.buffer, 0, bytesRead));
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0, new AsyncCallback(ReceiveCallback), state);
}
else
{
// All the data has arrived; put it in response.
if (state.sb.Length > 1)
{
response = state.sb.ToString();
}
// Signal that all bytes have been received.
receiveDone.Set();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
private static void Send(Socket client, String data)
{
Console.WriteLine("Send");
// Convert the string data to byte data using ASCII encoding.
byte[] byteData = Encoding.ASCII.GetBytes(data);
// Begin sending the data to the remote device.
client.BeginSend(byteData, 0, byteData.Length, 0,
new AsyncCallback(SendCallback), client);
}
private static void SendCallback(IAsyncResult ar)
{
try
{
Console.WriteLine("SendCallBack");
// Retrieve the socket from the state object.
Socket client = (Socket)ar.AsyncState;
// Complete sending the data to the remote device.
int bytesSent = client.EndSend(ar);
Console.WriteLine("Sent {0} bytes to server.", bytesSent);
// Signal that all bytes have been sent.
sendDone.Set();
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
}
When connecting to the server it responds with following lines:
CP3 Console
Warning: Another console session is open
CP3>
When I run the program i get following output:
Socket connected to 10.106.6.60:23
StartSend
WaitForResponse
Receive
Start Read
Bytes read: 13
String CP3 Console
Start Read
Bytes read: 43
String Warning: Another console session is open
Start Read
Bytes read: 6
String
CP3>
The thread '<No Name>' (0x1d80) has exited with code 0 (0x0).
The thread '<No Name>' (0x24bc) has exited with code 0 (0x0).
I tried different methods of reading stream but no successes. The program still hangs up at the same spot.
It seems as after reading last byte ">" the program does not re-executes method "ReceiveCallback" to exit out of the loop.
I get feeling it has something with the way "ReceiveCallback" is being called but I could not figure out what client.beginReceive() parameters actually do.

To expand on Roemer's answer: EndReceive will return 0 only if the conversation has terminated. That usually means a TCP packet with the FIN or RST flag has been received, most often because the peer - in your case, the server - closed its endpoint or exited. (There are various other cases; for example, a firewall or intermediate node could generate an RST.)
(Roemer wrote "BeginReceive never returns zero...", but clearly meant EndReceive, since BeginReceive returns the IAsyncResult object.)
If the peer doesn't close its end of the conversation, EndReceive won't return 0, and you'll never enter the else branch of your callback that invokes receiveDone.Set().
It's not entirely clear from your description exactly how the server behaves, but its last send of 6 bytes was clearly a newline followed by the "CP3>" prompt. At that point it's waiting for your client to send something. It's not going to close the connection.
TCP is a bytestream service - it doesn't provide record boundaries. So there's no way for the socket layer to know when the server is "done sending", except when the server closes the connection. You have three choices for your client:
Parse the server's output as you receive it and recognize when it's waiting for input (in this case, that means recognizing the "CP3>" prompt)
Decide the server's done after some arbitrary time has passed with no further data from the server
Don't worry about it, and simply output data from the server as you receive it
That third option is the simplest one. Do your first BeginReceive after making the connection. Then, in your callback method, if the conversation is still open and no errors have occurred, process the data you've received and invoke BeginReceive again.

To be honest, this example from microsoft is really bad.
In general, client.EndReceive never returns 0 except when the connection is closed. So receiveDone.Set will never be called in your example. In a more real-live example, you should check the received data in ReceiveCallback and when a keyword (like "Login: ") occurs, set the receiveDone and start sending (the login data for example) and the start receiving again and so on. You might also want to look at the async/await keyword (.net 4.5) and maybe upgrade to Visual Studio community edition :)
There are tons of examples on codeproject or found via google (like http://blogs.msdn.com/b/pfxteam/archive/2011/12/15/10248293.aspx).

Ensure all data was received

I am following this example but I am running into a problem. The sequence of my client/server connection goes like this:
Server is always listening for new clients
Client initiates connection
Server sends a small packet with connection information (9 bytes), client receives
Client sends response connection information, server receives
More back and forth with packets
My code is getting stuck during step 3 because the first call to BeginReceive is actually receiving all the data at once, but it goes into another call to BeginReceive even though there is no more data to get. I am using a custom server implementation that already functions communicating in this same fashion with an iOS application.
Is there a simple check I can perform to see if there is indeed more data to be received on the socket?
Here is my implementation of the receive callback function, the start receive function is identical:
private static void ReceiveCallback(IAsyncResult ar)
{
try
{
Console.WriteLine ( "Received data" );
// Retrieve the state object and the client socket
// from the asynchronous state object.
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
if (bytesRead > 0)
{
//First pass reads all 9 bytes the server sent
Console.WriteLine ( "Received bytes: {0}", bytesRead );
// There might be more data, so store the data received so far.
state.ms.Write(state.buffer, 0, bytesRead);
// Get the rest of the data.
//Gets stuck in this call because there is no more data to receive
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
}
else
{
Console.WriteLine ( "Received all data? {0}", state.ms.Length );
// All the data has arrived; put it in response.
if ( state.ms.Length > 0 )
{
Console.WriteLine ( "Received whole packet" );
MyPacketInfo.MyPacket receivedPacket = MyPacketInfo.MyPacket.packetFromMemoryStream ( state.ms );
if ( receivedPacket != null )
{
state.caller.ReceivedPacket ( receivedPacket, client );
}
else
{
Console.WriteLine ( "Received bad packet!" );
}
}
// Signal that all bytes have been received.
receiveDone.Set();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}

Socket Programming multiple client one server

I am just starting out Socket Programming in C# and am now a bit stuck with this problem.
How do you handle multiple clients on a single server without creating a thread for each client?
The one thread for each client works fine when there are say 10 clients but if the client number goes up to a 1000 clients is creating a thread for every single one of them advisable? If there is any other method to do this can someone please tel me?

Try to use asynchronous server.
The following example program creates a server that receives connection requests from clients. The server is built with an asynchronous socket, so execution of the server application is not suspended while it waits for a connection from a client. The application receives a string from the client, displays the string on the console, and then echoes the string back to the client. The string from the client must contain the string <EOF> to signal the end of the message.
using System;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
// State object for reading client data asynchronously
public class StateObject {
// Client socket.
public Socket workSocket = null;
// Size of receive buffer.
public const int BufferSize = 1024;
// Receive buffer.
public byte[] buffer = new byte[BufferSize];
// Received data string.
public StringBuilder sb = new StringBuilder();
}
public class AsynchronousSocketListener {
// Thread signal.
public static ManualResetEvent allDone = new ManualResetEvent(false);
public AsynchronousSocketListener() {
}
public static void StartListening() {
// Data buffer for incoming data.
byte[] bytes = new Byte[1024];
// Establish the local endpoint for the socket.
// The DNS name of the computer
// running the listener is "host.contoso.com".
IPHostEntry ipHostInfo = Dns.Resolve(Dns.GetHostName());
IPAddress ipAddress = ipHostInfo.AddressList[0];
IPEndPoint localEndPoint = new IPEndPoint(ipAddress, 11000);
// Create a TCP/IP socket.
Socket listener = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp );
// Bind the socket to the local endpoint and listen for incoming connections.
try {
listener.Bind(localEndPoint);
listener.Listen(100);
while (true) {
// Set the event to nonsignaled state.
allDone.Reset();
// Start an asynchronous socket to listen for connections.
Console.WriteLine("Waiting for a connection...");
listener.BeginAccept(
new AsyncCallback(AcceptCallback),
listener );
// Wait until a connection is made before continuing.
allDone.WaitOne();
}
} catch (Exception e) {
Console.WriteLine(e.ToString());
}
Console.WriteLine("\nPress ENTER to continue...");
Console.Read();
}
public static void AcceptCallback(IAsyncResult ar) {
// Signal the main thread to continue.
allDone.Set();
// Get the socket that handles the client request.
Socket listener = (Socket) ar.AsyncState;
Socket handler = listener.EndAccept(ar);
// Create the state object.
StateObject state = new StateObject();
state.workSocket = handler;
handler.BeginReceive( state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
public static void ReadCallback(IAsyncResult ar) {
String content = String.Empty;
// Retrieve the state object and the handler socket
// from the asynchronous state object.
StateObject state = (StateObject) ar.AsyncState;
Socket handler = state.workSocket;
// Read data from the client socket.
int bytesRead = handler.EndReceive(ar);
if (bytesRead > 0) {
// There might be more data, so store the data received so far.
state.sb.Append(Encoding.ASCII.GetString(
state.buffer,0,bytesRead));
// Check for end-of-file tag. If it is not there, read
// more data.
content = state.sb.ToString();
if (content.IndexOf("<EOF>") > -1) {
// All the data has been read from the
// client. Display it on the console.
Console.WriteLine("Read {0} bytes from socket. \n Data : {1}",
content.Length, content );
// Echo the data back to the client.
Send(handler, content);
} else {
// Not all data received. Get more.
handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
}
}
private static void Send(Socket handler, String data) {
// Convert the string data to byte data using ASCII encoding.
byte[] byteData = Encoding.ASCII.GetBytes(data);
// Begin sending the data to the remote device.
handler.BeginSend(byteData, 0, byteData.Length, 0,
new AsyncCallback(SendCallback), handler);
}
private static void SendCallback(IAsyncResult ar) {
try {
// Retrieve the socket from the state object.
Socket handler = (Socket) ar.AsyncState;
// Complete sending the data to the remote device.
int bytesSent = handler.EndSend(ar);
Console.WriteLine("Sent {0} bytes to client.", bytesSent);
handler.Shutdown(SocketShutdown.Both);
handler.Close();
} catch (Exception e) {
Console.WriteLine(e.ToString());
}
}
public static int Main(String[] args) {
StartListening();
return 0;
}
}
That's will be the best solution.

Threads can work fine but rarely scales well to that many clients. There are two easy ways and lots of more complex ways to handle that, here's some pseudocode for how the easier two are usually structured to give you an overview.
select()
This is a call to check which sockets have new clients or data waiting on them, a typical program looks something like this.
server = socket(), bind(), listen()
while(run)
status = select(server)
if has new client
newclient = server.accept()
handle add client
if has new data
read and handle data
Which means no threads are needed to handle multiple clients, but it doesn't really scale well either if handle data take a long time, then you won't read new data or accept new clients until that's done.
Async sockets
This is another way of handling sockets which is kind of abstracted above select. You just set up callbacks for common events and let the framework do the not-so-heavy lifting.
function handleNewClient() { do stuff and then beginReceive(handleNewData) }
function handleNewData() { do stuff and then beginReceive(handleNewData) }
server = create, bind, listen etc
server.beginAddNewClientHandler(handleNewClient)
server.start()
I think this should scale better if your data handling take a long time. What kind of data handling will you be doing?

This could be a good starting point. If you want to avoid 1 thread <-> 1 client; then you should use async socket facilities provided in .NET. Core object to use here is SocketAsyncEventArgs.

How to asynchronously receive complex object in C#?

EDIT:
A more concise explanation of what I was trying to do here and with answers
I'm using c# asynchronous sockets to receive data from a source.
My problem is how and where to store received data if there are more to be received?
When a string is received, I can just use a string builder to receive and store just like this from msdn:
private void ReceiveCallback_onQuery(IAsyncResult ar)
{
try
{
// Retrieve the state object and the client socket
// from the asynchronous state object.
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
if (bytesRead > 0)
{
// There might be more data, so store the data received so far.
dataReceived += state.buffer; //Does not work (dataReceived is byte[] type)
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback_onQuery), state);
}
else
{
// All the data has arrived; put it in response.
if (dataReceived > 1)
{
response_onQueryHistory = ByteArrayToObject(dataReceived)
}
// Signal that all bytes have been received.
receiveDoneQuery.Set();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
I dont want to convert my received data into a string as in my case, I am receiving a complex object.
The data sent was serialized and I'm also fine with deserializing.
My question how to 'continually' receive data from the socket without using the string builder to store it.
Thanks!

This depends on how the complex thing is serialized prior to being pushed down the wire in the broken down bytes, you will receive those bytes and use the same algorithm/technique used to serialize the thing to deserialize it back to its original state.
For a more specific answer I'd ask that you be more specific yourself.

My problem is how and where to store received data if there are more to be received?
Can use Buffer.BlockCopy and queue it up, for eg,
int rbytes = client.EndReceive(ar);
if (rbytes > state.buffer)
{
byte[] bytesReceived = new byte[rbytes];
Buffer.BlockCopy(state.buffer, 0, bytesReceived, 0, rbytes);
state.myQueue.Enqueue(bytesReceived);
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback_onQuery), state)
}