Develop Reference

Basic HttpServer | How to read data from NetworkStream properly

I am trying to make a basic webserver for practice, however i think that i'm not reading from the NetworkStream properly... On the picture bellow you can see that request 2 and 3 do not have 2 newlines \r\n\r\n separating them , which means that i'm reading less data.
Also the requests hang for no obvious reason. I think that I am properly closing the stream and the client, by using networkStream.Close(); and tcpClient.Close();
using System;
using System.Net;
using System.Net.Sockets;
using System.Text;
namespace HttpServer
{
class Program
{
static void Main(string[] args)
{
var webserver = new Program();
webserver.Listen(12345);
}
public static void ProcessTcpClientAsync(TcpClient tcpClient)
{
// Get a stream object for reading and writing
NetworkStream networkStream = tcpClient.GetStream();
byte[] buffer = readByteBufferFromNetworkStream(networkStream);
string request = Encoding.UTF8.GetString(buffer);
Console.WriteLine(request);
const string HTTPNewLine = "\r\n";
string html = "<h1>Hello from TestServer</h1>";
string response = "HTTP/1.1 200 OK" + HTTPNewLine;
response += "Server: TestServer 2020" + HTTPNewLine;
response += "Content-Type: text/html; charset=utf-8" + HTTPNewLine;
response += "Content-Length: " + html.Length + HTTPNewLine;
response += HTTPNewLine;
response += html;
response += HTTPNewLine;
byte[] responseBytes = Encoding.UTF8.GetBytes(response);
networkStream.Write(responseBytes);
networkStream.Close();
tcpClient.Close();
}
public void Listen(Int32 port)
{
//Int32 port = 12345;
IPAddress localaddr = IPAddress.Parse("127.0.0.1");
TcpListener tcpListener = new TcpListener(localaddr, port);
tcpListener.Start();
// Enter the listening loop (service/daemon)
while (true)
{
//# Sync Version for debugging
TcpClient tcpClient = tcpListener.AcceptTcpClient();
ProcessTcpClientAsync(tcpClient);
//tcpListener.AcceptTcpClientAsync().ContinueWith((res) => ProcessTcpClientAsync(res.GetAwaiter().GetResult()));
}
}
public static byte[] readByteBufferFromNetworkStream(NetworkStream networkStream)
{
byte[] buffer = new byte[256];
int bytesRead = 0;
while (true)
{
int currentBytesRead = networkStream.Read(buffer, bytesRead, buffer.Length - bytesRead);
bytesRead += currentBytesRead;
// If we've read less data than the size of our buffer -> it means there is no more information, so break.
if (currentBytesRead < buffer.Length) { break; }
// If we've reached the end of our buffer -> check to see if there's any more information, or break.
if (bytesRead == buffer.Length)
{
int nextByte = networkStream.ReadByte();
// End of stream? If so, we're done.
if (nextByte == -1)
{
break;
}
// Otherwise resize the buffer, put in the byte we've just read and continue.
else
{
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[bytesRead] = (byte)nextByte;
buffer = newBuffer;
bytesRead++;
}
}
}
// Buffer is now too big. Shrink it.
byte[] resultBuffer = new byte[bytesRead];
Array.Copy(buffer, resultBuffer, bytesRead);
return resultBuffer;
}
}
}

It's part of the standard for this type of response to seperate the header from the body content.
See https://www.rfc-editor.org/rfc/rfc7230#section-3
All HTTP/1.1 messages consist of a start-line followed by a sequence
of octets in a format similar to the Internet Message Format
[RFC5322]: zero or more header fields (collectively referred to as
the "headers" or the "header section"), an empty line indicating the
end of the header section, and an optional message body.
HTTP-message = start-line
*( header-field CRLF )
CRLF
[ message-body ]
(CRLF above will be treated like \r\n)

Async socket client using TcpClient class C#

I have implemented a socket client using TcpClient class. So I can send and receive data, everything works good. But I ask some of the guru's out there :) Is there something wrong with my implementation? maybe there is a better way of doing things. In particular, how do I handle disconnects? Is there some indicator (or maybe I can write one myself) that tells me that socket has disconnected?
I also have looked into async await features of Socket class, but can't wrap my head around "SocketAsyncEventArgs", why is it there in the first place.
Why cant i just: await Client.SendAsync("data"); ?
public class Client
{
private TcpClient tcpClient;
public void Initialize(string ip, int port)
{
try
{
tcpClient = new TcpClient(ip, port);
if (tcpClient.Connected)
Console.WriteLine("Connected to: {0}:{1}", ip, port);
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Initialize(ip, port);
}
}
public void BeginRead()
{
var buffer = new byte[4096];
var ns = tcpClient.GetStream();
ns.BeginRead(buffer, 0, buffer.Length, EndRead, buffer);
}
public void EndRead(IAsyncResult result)
{
var buffer = (byte[])result.AsyncState;
var ns = tcpClient.GetStream();
var bytesAvailable = ns.EndRead(result);
Console.WriteLine(Encoding.ASCII.GetString(buffer, 0, bytesAvailable));
BeginRead();
}
public void BeginSend(string xml)
{
var bytes = Encoding.ASCII.GetBytes(xml);
var ns = tcpClient.GetStream();
ns.BeginWrite(bytes, 0, bytes.Length, EndSend, bytes);
}
public void EndSend(IAsyncResult result)
{
var bytes = (byte[])result.AsyncState;
Console.WriteLine("Sent {0} bytes to server.", bytes.Length);
Console.WriteLine("Sent: {0}", Encoding.ASCII.GetString(bytes));
}
}
And the usage:
static void Main(string[] args)
{
var client = new Client();
client.Initialize("127.0.0.1", 8778);
client.BeginRead();
client.BeginSend("<Names><Name>John</Name></Names>");
Console.ReadLine();
}

Okay it took me 10 seconds to find biggest issue you could've done :
public void BeginRead()
{
var buffer = new byte[4096];
var ns = tcpClient.GetStream();
ns.BeginRead(buffer, 0, buffer.Length, EndRead, buffer);
}
But don't worry that's why we are on SO.
First let me explain why it's such a big issue.
Assume that you're sending message which is 4097 bytes long. Your buffer can only accept 4096 bytes meaning that you cannot pack whole message into this buffer.
Assume you're sending message which is 12 bytes long. You're still allocating 4096 bytes in your memory just to store 12 bytes.
How to deal with this?
Every time you work with networking you should consider making some kind of protocol ( some people call it message framing, but it's just a protocol ) that will help you to identify incomming package.
Example of a protocol could be :
[1B = type of message][4B = length][XB = message]
- where X == BitConvert.ToInt32(length);
Receiver:
byte messageType = (byte)netStream.ReadByte();
byte[] lengthBuffer = new byte[sizeof(int)];
int recv = netStream.Read(lengthBuffer, 0, lengthBuffer.Length);
if(recv == sizeof(int))
{
int messageLen = BitConverter.ToInt32(lengthBuffer, 0);
byte[] messageBuffer = new byte[messageLen];
recv = netStream.Read(messageBuffer, 0, messageBuffer.Length);
if(recv == messageLen)
{
// messageBuffer contains your whole message ...
}
}
Sender:
byte messageType = (1 << 3); // assume that 0000 1000 would be XML
byte[] message = Encoding.ASCII.GetBytes(xml);
byte[] length = BitConverter.GetBytes(message.Length);
byte[] buffer = new byte[sizeof(int) + message.Length + 1];
buffer[0] = messageType;
for(int i = 0; i < sizeof(int); i++)
{
buffer[i + 1] = length[i];
}
for(int i = 0; i < message.Length; i++)
{
buffer[i + 1 + sizeof(int)] = message[i];
}
netStream.Write(buffer);
Rest of your code looks okay. But in my opinion using asynchronous operations in your case is just useless. You can do the same with synchronous calls.

It's hard to answer because theres no exact question here but more some kind of code review. But still some hints:
Your connect mechanism seems wrong. I don't think TcpClient.Connected will block until the connection was established. So it will often just fail as the connection is in progress and then you start all over again. You should switch to using a blocking or async Connect method.
SocketAsyncEventArgs is a mechanism for high performance async data transmission. There's rarely a need for it. You should just ignore it
If you want to send data asynchronously you should use the Async methods which return a Task, because these can be easily combined with async/await.
The APM model (BeginXYZ/EndXYZ) is kind of deprecated, you shouldn't use it anymore in new code. One issue with it is that sometimes the End method is called synchronously inside the Begin method which can lead to surprising behavior. If this is not the case the completion callback will be performed from a random thread on the ThreadPool. This is also often not what you want. The TPL methods avoid this.
For your simple use case the blocking methods are also totally fine and do not come with the complexity of the various async methods.
The reading side of the code with TPL methods (untested):
public async Task Initialize(string ip, int port)
{
tcpClient = new TcpClient;
await tcpClient.ConnectAsync(ip, port);
Console.WriteLine("Connected to: {0}:{1}", ip, port);
}
public async Task Read()
{
var buffer = new byte[4096];
var ns = tcpClient.GetStream();
while (true)
{
var bytesRead = await ns.ReadAsync(buffer, 0, buffer.Length);
if (bytesRead == 0) return; // Stream was closed
Console.WriteLine(Encoding.ASCII.GetString(buffer, 0, bytesRead));
}
}
In the initialization part you would do:
await client.Initialize(ip, port);
// Start reading task
Task.Run(() => client.Read());
For using synchronous methods delete all Async occurences and replace Task with Thread.

C# good practice waiting for TCP response

While making a c# application for remote controlling cisco routers using TCP, I got the problem of waiting for a response from the router.
For the application I have to connect to a Cisco router using a TCP connection. After the connection has been made a networkstream will push my command to the Cisco router. To let the router process the command, I am using Thread.Sleep. This is not the best solution.
Here is the complete code to get a idea what my program is doing.
string IPAddress = "192.168.1.1";
string message = "show running-config"; // command to run
int bytes;
string response = "";
byte[] responseInBytes = new byte[4096];
var client = new TcpClient();
client.ConnectAsync(IPAddress, 23).Wait(TimeSpan.FromSeconds(2));
if (client.Connected == true)
{
client.ReceiveTimeout = 3;
client.SendTimeout = 3;
byte[] messageInBytes = Encoding.ASCII.GetBytes(message);
NetworkStream stream = client.GetStream();
Console.WriteLine();
stream.Write(messageInBytes, 0, messageInBytes.Count()); //send data to router
Thread.Sleep(50); // temporary way to let the router fill his tcp response
bytes = stream.Read(responseInBytes, 0, responseInBytes.Length);
response = Encoding.ASCII.GetString(responseInBytes, 0, bytes);
return response; //whole command output
}
return null;
What is a good and reliable way to get the full response.
Thanks for any help or command.
More info:
The networksteam is always filled with something, most of the time it is filled with the cisco IOS login page. The biggest problem is to determine when the router is done filling up the response.
The response I most of the time get:
"??\u0001??\u0003??\u0018??\u001f\r\n\r\nUser Access Verification\r\n\r\nUsername: "
The return data will be diffent every time because it will be a result of a cisco command. This can vary from a short string to a very long string.
mrmathijs95 -

When reading from NetworkStream with Stream.Read it not 100% sure that you will read all expected data. Stream.Read can return when only few packet arrived and not waiting for others.
To be sure that you get all data use BinaryReader for reading.
BinaryReader.Read method will block current thread until all expected data arrived
private string GetResponse(string message)
{
const int RESPONSE_LENGTH = 4096;
byte[] messageInBytes = Encoding.ASCII.GetBytes(message);
bool leaveStreamOpen = true;
using(var writer = new BinaryWriter(client.GetStream()))
{
writer.Write(messageInBytes);
}
using(var reader = New BinaryReader(client.GetStream()))
{
byte[] bytes = reader.Read(RESPONSE_LENGTH );
return Encoding.ASCII.GetString(bytes);
}
}

Don't use Thread.Sleep. I would async/await the entire thing, given that you don't always know what the data is based on your recent edit. This is how I would do it (untested):
public class Program
{
// call Foo write with "show running-config"
}
public class Foo
{
private TcpClient _client;
private ConcurrentQueue<string> _responses;
private Task _continualRead;
private CancellationTokenSource _readCancellation;
public Foo()
{
this._responses = new ConcurrentQueue<string>();
this._readCancellation = new CancellationTokenSource();
this._continualRead = Task.Factory.StartNew(this.ContinualReadOperation, this._readCancellation.Token, this._readCancellation.Token);
}
public async Task<bool> Connect(string ip)
{
this._client = new TcpClient
{
ReceiveTimeout = 3, // probably shouldn't be 3ms.
SendTimeout = 3 // ^
};
int timeout = 1000;
return await this.AwaitTimeoutTask(this._client.ConnectAsync(ip, 23), timeout);
}
public async void StreamWrite(string message)
{
var messageBytes = Encoding.ASCII.GetBytes(message);
var stream = this._client.GetStream();
if (await this.AwaitTimeoutTask(stream.WriteAsync(messageBytes, 0, messageBytes.Length), 1000))
{
//write success
}
else
{
//write failure.
}
}
public async void ContinualReadOperation(object state)
{
var token = (CancellationToken)state;
var stream = this._client.GetStream();
var byteBuffer = new byte[4096];
while (!token.IsCancellationRequested)
{
int bytesLastRead = 0;
if (stream.DataAvailable)
{
bytesLastRead = await stream.ReadAsync(byteBuffer, 0, byteBuffer.Length, token);
}
if (bytesLastRead > 0)
{
var response = Encoding.ASCII.GetString(byteBuffer, 0, bytesLastRead);
this._responses.Enqueue(response);
}
}
}
private async Task<bool> AwaitTimeoutTask(Task task, int timeout)
{
return await Task.WhenAny(task, Task.Delay(timeout)) == task;
}
public void GetResponses()
{
//Do a TryDequeue etc... on this._responses.
}
}
I didn't expose the read cancellation publicly, but you could add this method to cancel the read operation:
public void Cancel()
{
this._readCancellation.Cancel();
}
And then dispose of your client and all that fun stuff.
Lastly, because you said there's always data available on the stream, where you're doing the read you may have to do some logic on the number of bytes last read to offset yourself within the stream if the data doesn't clear. You'll know if the responses you're getting is always the same.

This is the working code for me.
It uses the solution of Fabio,
combined with a while loop to check every X miliseconds if the response has changed.
client.ReceiveTimeout = 3;
client.SendTimeout = 3;
byte[] messageInBytes = Encoding.ASCII.GetBytes(message);
NetworkStream stream = client.GetStream();
Console.WriteLine();
using (var writer = new BinaryWriter(client.GetStream(),Encoding.ASCII,true))
{
writer.Write(messageInBytes);
}
using (var reader = new BinaryReader(client.GetStream(),Encoding.ASCII, true))
{
while (itIsTheEnd == false)
{
bytes = reader.Read(responseInBytes, 0, responseInBytes.Count());
if (lastBytesArray == responseInBytes)
{
itIsTheEnd = true;
}
lastBytesArray = responseInBytes;
Thread.Sleep(15);
}
}
response = Encoding.ASCII.GetString(responseInBytes);
Thanks for everyone who suggested a solution.
And thanks to Fabio for the given solution.

Very basic web server in C#

I'm wanting to run a little socket server in C# to be accessed by a browser. I have a socket listener up and running on a specific port, and am trying to access it via the browser:
class WebSocketServer
{
private static string output = string.Empty;
public void CreateListener()
{
TcpListener tcpListener = null;
var ipAddress = Dns.GetHostEntry("localhost").AddressList[0];
try
{
tcpListener = new TcpListener(ipAddress, 1313);
tcpListener.Start();
output = "Waiting for a connection";
}
catch (Exception e)
{
throw;
}
var socketHelper = new SocketHelper();
while (true)
{
Thread.Sleep(10);
TcpClient tcpClient = tcpListener.AcceptTcpClient();
byte[] bytes = new byte[256];
var stream = tcpClient.GetStream();
stream.Read(bytes, 0, bytes.Length);
socketHelper.ProcessMessage(tcpClient, stream, bytes);
}
}
}
class SocketHelper
{
private static int counter = 0;
public void ProcessMessage(TcpClient tcpClient, NetworkStream stream, byte[] bytesReceived)
{
// handle message received and send response to client
var msg = Encoding.ASCII.GetString(bytesReceived);
string response = string.Empty;
if (msg.Substring(0, 10) == "GET / HTTP")
{
response = "";// html page
}
byte[] bytesToSend = Encoding.ASCII.GetBytes(response);
stream.Write(bytesToSend, 0, bytesToSend.Length);
}
The browser appears to connect to it, but it never seems to display the html - what's wrong? I'm eventually wanting to be able to serve up JSON data via a REST interface. In addition, is there a much easier solution to (I assume) this common problem.

How can I send data with a TcpListener, and wait for the response?

I have the following code:
using (TcpClient client = new TcpClient())
{
client.Connect(host, port);
using (SslStream stream = new SslStream(client.GetStream(), true))
{
stream.AuthenticateAsClient(host);
stream.Write(System.Text.Encoding.ASCII.GetBytes(dataToSend));
int byteRead = 0;
byte[] buffer = new byte[1000];
do
{
byteRead = stream.Read(buffer, 0, 1000);
reponse += System.Text.Encoding.ASCII.GetString(buffer, 0, byteRead);
}
while (byteRead > 0);
}
}
I send a string to a server, and then wait for the response.
Is this the proper way to do it?
If the server takes some time to process what I sent, will it still work or will stream.Read return 0 and exit the loop? Or if some packets from the response are lost and need to be resent, will it still work?

The overall structure of your code looks right.
byteRead = stream.Read(buffer, 0, 1000); will block until all of the response data is retrieved from the server. If the remote server shuts down the connection (timeout, etc), 0 will be returned.
See the remarks found here.
The framework will properly deal with packets lost during network operations - don't worry about them.

public string Method()
{
m_Client = new TcpClient();
m_Client.Connect(m_Server, m_Port);
m_Stream = m_Client.GetStream();
m_Writer = new StreamWriter(m_Stream);
m_Reader = new StreamReader(m_Stream);
m_Writer.WriteLine(request);
m_Writer.Flush();
return m_Reader.ReadToEnd();
}