I am trying to make HTTP request/response using sockets in C#. GET request appears below.
StringBuilder sb = new StringBuilder();
sb.AppendFormat("GET http://{0}/ HTTP/1.1\r\n", hostname);
sb.AppendFormat("Host: {0}\r\n", hostname);
sb.Append("Connection: keep-alive\r\n");
sb.Append(#"Accept:text/html,*/*");
sb.Append("\r\n\r\n");
where hostname is something like 'mit.edu' or 'facebook.com' or 'google.com' or anything else. For some strange reason I have just a status-line (with 200 status code) and headers as http response. But there is no message body in the response: attached srceenshot of my console app
Here is a method that perform all manipulations with socket and make http request:
public static void DoHttpGetRequest(String hostname, Int16 port = 80) {
IPHostEntry ipHostEntry = Dns.GetHostEntry(hostname);
IPAddress ipAdress = ipHostEntry.AddressList[0];
IPEndPoint ipEndPoint = new IPEndPoint(ipAdress, port);
Socket socket = new Socket(ipAdress.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
socket.Connect(ipEndPoint);
String request = CreateRequest(hostname);
Byte[] byteRequest = Encoding.UTF8.GetBytes(request);
Byte[] byteResponse = new Byte[1000];
int bytesSent = socket.Send(byteRequest);
int bytesReceive = socket.Receive(byteResponse);
Console.WriteLine(request);
Console.WriteLine();
Console.WriteLine(Encoding.UTF8.GetString(byteResponse, 0, bytesReceive));
socket.Shutdown(SocketShutdown.Both);
socket.Close();
}
My first thought was that the socket hadn't received the whole response from the server. In this case I do still not know how to solve the problem.
So what is going on? Where is the mistake?
Don't expect to get the full response inside a single receive. What you do is to receive data until you have the full response header (this could take several receive calls too), then parse the header to find out how long the response is and then read the necessary data of the response which also can need multiple receive calls. And since you are doing a HTTP/1.1 request you also have to deal with chunked responses.
I recommend to better use a HTTP library to handle all the problems. If you insist on doing it all by your own read the specification of HTTP and implemented accordingly.
It also helps to look around at stackoverflow for similar requests because this problem you have is very typical for someone trying to implement HTTP first time, without understanding enough on how sockets, TCP and HTTP work.
I think you should keep the socket open for a longer time. Insert a
Thread.Wait(5000);
may help. Then you can do a second socket.Receive as Steffen proposed. Maybe you can try the following (wait until the server closed the connection):
while (socket.Connected) do
{
int bytesReceive = socket.Receive(byteResponse);
}
But I didn't test this. It's easier to use the System.Net.Http.HttpClient
Everything I need to do is to receive data with multiple receive calls until all response information received.
do
{
bytesReceive = socket.Receive(byteResponse, byteResponse.Length, 0);
response += Encoding.UTF8.GetString(byteResponse, 0, bytesReceive);
}
while (bytesReceive > 0);
Thanx everybody for your help!
Related
I'm working on a TCP connection where my client connects to a server's IP on 2 different ports. So I have 2 instances of TcpClient objects, one connecting to the IP on port 9000 and the other on port 9001.
The aim of 2 connections is that the server uses the active connection on port 9000 to give certain responses to the client frequently, and the client uses these responses to form and send a request on port 9001.
Now, the first time I connect on 9000, I get a response, I then form a request and fire off via 9001. Now I have a feeling I'm doing something wrong with the way I'm managing asynchronous requests to both ports, but I can't figure an alternate way of doing this:
IPAddress IPAddress = IPAddress.Parse("192.168.1.10");
public static async Task ConnectToPort9000()
{
TcpClient TcpClient1 = new TcpClient();
try
{
await TcpClient1.ConnectAsync(IPAddress, 9000);
if (TcpClient1.Connected)
{
byte[] Buffer = new byte[1024];
while (await TcpClient1.GetStream().ReadAsync(Buffer, 0, Buffer.Length) > 0)
{
//Server returns a message on this port
string Port9000Response = Encoding.UTF8.GetString(Buffer, 0, Buffer.Length);
//Setting ConfigureAwait(false) so that any further responses returned
//on this port can be dealt with
await Task.Run(async () =>
{
await SendRequestToPort9001BasedOnResponseAsync(Port9000Response);
}).ConfigureAwait(false);
}
}
}
catch (Exception)
{
throw;
}
}
private async Task SendRequestToPort9001BasedOnResponseAsync(string Port9000Response)
{
//Open connection on port 9001 and send request
TcpClient TcpClient2 = new TcpClient();
await TcpClient2.ConnectAsync(IPAddress, 9001);
if (TcpClient2.Connected)
{
//Handle each string response differently
if (Port9000Response == "X")
{
//Form a new request message to send on port 9001
string _NewRequestMesssage = "Y";
byte[] RequestData = Encoding.UTF8.GetBytes(_NewRequestMesssage);
new SocketAsyncEventArgs().SetBuffer(RequestData, 0, RequestData.Length);
await TcpClient2.GetStream().WriteAsync(RequestData, 0, RequestData.Length);
await TcpClient2.GetStream().FlushAsync();
//Handle any responses on this port
//At this point, based on the request message sent on this port 9001
//server sends another response on **port 9000** that needs separately dealing with
//so the while condition in the previous method should receive a response and continue handling that response again
}
else if (Port9000Response == "A")
{
//Do something else
}
}
}
The issue I am having at the moment is, after I send the request on port 9001, when processing any response messages on port 9001, the server has already sent me a response on port 9000, but my while loop on the first method isn't getting triggered, and it seems like that's because it's still executing the second method to process request/response on port 9001. I tried using ConfigureAwait(false) to basically fire and forget, but it doesn't seem to be working. Am I handling asynchronous processes the wrong way? Or should I look at alternatives such as action/delegates?
The aim of 2 connections is that the server uses the active connection on port 9000 to give certain responses to the client frequently, and the client uses these responses to form and send a request on port 9001.
Please don't do this. Socket programming is hard enough without making it extremely more complicated with multiple connections. Error handling becomes harder, detection of half-open connections becomes harder (or impossible), and communication deadlocks are harder to avoid.
Each socket connection is already bidirectional; it already has two independent streams. The only thing you need to do is always be reading, and send as necessary. The read and write streams are independent, so keep your reads and writes independent.
I followed the example in MSDN about C# socket programming.
private static void StartClient() {
// Connect to a remote device.
try {
// Establish the remote endpoint for the socket.
// The name of the
// remote device is "host.contoso.com".
IPHostEntry ipHostInfo = Dns.Resolve("host.contoso.com");
IPAddress ipAddress = ipHostInfo.AddressList[0];
IPEndPoint remoteEP = new IPEndPoint(ipAddress, port);
// Create a TCP/IP socket.
Socket client = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);
// Connect to the remote endpoint.
client.BeginConnect( remoteEP,
new AsyncCallback(ConnectCallback), client);
connectDone.WaitOne();
// Send test data to the remote device.
Send(client,"This is a test<EOF>");
sendDone.WaitOne();
// Receive the response from the remote device.
Receive(client);
receiveDone.WaitOne();
// Write the response to the console.
Console.WriteLine("Response received : {0}", response);
// Release the socket.
client.Shutdown(SocketShutdown.Both);
client.Close();
} catch (Exception e) {
Console.WriteLine(e.ToString());
}
}
I've created a simple test application that works.
The app sends and receives the response almost instantaneous.
So now the challenge is to implement the solution where the socket connects to an external device that will process the information from the socket and then send back a response.
At the line "Send test data to the remote device" --> the device receives this information at processes.
The debug keeps on going. The process is still going on in the remote device because I can see the data churning away.
But once it gets to receiveDone.WaitOne(); the debug is gone and the application hangs.
The problem here is the remote server is still processing. When it is done it is supposed to send back a response. Nothing happens.
Since I'm not that experienced with socket coding, I was wondering if anyone has run into this before. And if yes, how would I go about solving this issues so I can get a response back and the application doesn't hang?
Update:
I feel like smacking myself in the head.
The issue isn't with the WaitOne().
The debug is gone from WaitOne() because it's waiting for a response.
When the response comes, it goes to Receive() which then calls ReceiveCallback().
private void Receive(Socket client)
{
try
{
// Create the state object.
StateObject state = new StateObject();
state.workSocket = client;
// Begin receiving the data from the remote device.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
//THIS IS WHERE THE CODE DOESN'T CONTINUE.
}
catch (Exception e)
{
string error = e.InnerException.ToString();
}
}
When ReceiveCallback() is complete and it finishes getting all the data - it stops. It doesn't go back to my code where I initiate the sending and receiving of data.
It's supposed to go:
1. Get input request string
2. Pass it to the socket code
3. Socket code sends input request string
4. Socket code receives response string
5. Take the response string and do something with it
So it stops at #4 and ends without completing all the steps.
Am i missing code for it to continue the flow? Or have I set the process up incorrectly?
Thanks again for the help.
#John - thanks for updating my title. I'll try to keep the title in mind for the future.
========================================
UPDATE 2:
If I change the code and do not use the Receive() as it is in the MSDN sample, MSDN sample link and do something like:
var response = client.Receive(buffer);
var message = Encoding.ASCII.GetString(buffer);
This would work and everything continues. I need to make sure that the response isn't cut off though. But then that would defeat the purpose of it by async client correct?
Hmm....
thanks again for everyone's input.
OK. I got it after seeing how I went wrong.
I didn't check for the end response. I assumed that the end of the buffer transmission would set off the method that would return the response.
Thanks to everyone for your help. I couldn't have come to the solution without your help.
cheers!
I created a client in Java and I want simply to send some data to my server that is in C#.
The problem is that if I write in the client for example hello, I get only the first letter. In the byte array there is only one element.
I guess there is some problem with the server side because in my server in JAVA works everything fine so the client in JAVA works fine.
Does anybody see any problem?
Thank you in advance.
You're thinking about TCP the wrong way, you don't simply "Receive" once and get the result of one "Send".
TCP is a "streaming" protocol and does not automatically separate into "packets". You may even get data of 2 sends in one receive.
A common pattern is to prefix one message with its length, so you can call receive until you get the amount of bytes requested. To make Receive return immediately if no data is in the buffer, set your socket to non-blocking.
Here's another good article on the topic.
Now, your provided code should work either way because there is next to no latency on local networks. Have you checked if your Java part buffers steam / can you manually flush them?
As Damon Gant said, TCP is a streaming protocol. I suggest you create your own protocol. I wouldn't send strings. If you're doing anything non-trivial this is really the best way to go.
Typically I include a magic number, checksum, packet body length in bytes, and protocol version in my protocol headers. The magic number makes it easier to delineate packets in a stream (very useful for debugging your custom protocol stream.) Having a checksum helps ensure you're parsing things correctly. A checksum doesn't help much with integrity over TCP as the TCP protocol already has a checksum. The packet body length helps you detect when you have all the bytes for your packet. The protocol version can help you know how to interpret the packet body's bytes.
Upon receiving data, place all bytes into a separate buffer and scan for your protocol header. If you can parse your header, check to see that packet's bytes are all present. If so, parse the packet. Repeat this process till you find an incomplete packet, or the buffer is empty.
For each packet you want to send, I'd create a class. When you want to send a packet, create and serialize the proper class, and prepend your protocol header for that class's bytes.
You could use Java's serializer, but if you've many client's connecting to a single server, you probably don't want to use Java for the server. This makes things difficult because now you need to implement a java serializer in another language. Because of this its typically better to either convert your packets into bytes by hand (tedious but simple,) OR you could write your own serializer using reflection. I'd suggest the latter for bigger projects.
problem is prabably in java side because your listener works fine.
I copy pasted your listener code in a test application.
Than I created another test applicationand send hello word and I listened it completely.
public static void sender()
{
TcpClient client = new TcpClient();
IPEndPoint serverEndPoint = new IPEndPoint(IPAddress.Parse("192.168.2.236"), 30000);
client.Connect(serverEndPoint);
NetworkStream clientStream = client.GetStream();
ASCIIEncoding encoder = new ASCIIEncoding();
byte[] buffer = encoder.GetBytes("Hello Server!");
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
}
Connection accepted from 192.168.2.236:22811
Recieved...
Hello Server!
Btw, this might be better listener.
public void listener()
{
TcpListener tcpListener = new TcpListener(IPAddress.Any, 30000);
tcpListener.Start();
TcpClient tcpClient = tcpListener.AcceptTcpClient();
NetworkStream clientStream = tcpClient.GetStream();
byte[] message = new byte[4096];
int bytesRead;
while (true)
{
bytesRead = 0;
try
{
//blocks until a client sends a message
bytesRead = clientStream.Read(message, 0, 4096);
}
catch
{
//a socket error has occured
break;
}
if (bytesRead == 0)
{
//the client has disconnected from the server
break;
}
//message has successfully been received
ASCIIEncoding encoder = new ASCIIEncoding();
Console.Write(encoder.GetString(message, 0, bytesRead));
}
tcpClient.Close();
}
I have a simple UDP listener that I am trying to collect datagrams from. My datagrams can be in one of two data formats. With the first data format, I am receiving data in my program as expected. With the second, there is absolutely no indication that data is ever received from my program, even though I can verify that the UDP data is passing onto the the network interface via Wireshark. I thought that maybe these were malformed UDP packets that Windows was rejecting but Wireshark does label them as UDP. My code is below:
static void Main(string[] args)
{
Thread thdUdpServer = new Thread(new ThreadStart(serverThread));
thdUdpServer.Start();
}
static void serverThread()
{
Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
socket.Bind(new IPEndPoint(new IPAddress(0), 2000));
while (true)
{
byte[] responseData = new byte[128];
socket.Receive(responseData);
string returnData = Encoding.ASCII.GetString(responseData);
Console.WriteLine(DateTime.Now + " " + returnData);
}
The missing packets are all 29 byte datagrams that look something like this (translated to ASCII).
#01RdFFFF...?...... ........F
Why would Wireshark indicate their presence but .NET not seem to see them?
If the bytes contain non-printable ASCII characters, you may not see them on the Console.
There's something missing in your code. It should be throwing a socket exception when calling ReceiveFrom (at least according to MSDN, haven't tried your code.)
You should bind your socket to the address:port you want to listen on (or use 0.0.0.0 as the address to listen on any local address):
socket.Bind(new IPEndPoint(new IPAddress(0), 2000);
The EndPoint in ReceiveFrom is not the listening port for the server. It's the address you want to receive packets from. You can use an Endpoint of 0.0.0.0:0 to receive from any host.
After returning from the method the Endpoint will be filled with the address of the host that sent the packet (client).
You can use Receive instead of ReceiveFrom if you don't care about the client end point.
Likely your client is not sending packets from 192.168.1.100:2000, and that is why you are not receiving them; thought why you're not getting an exception when calling ReceiveFrom is beyond me.
Also:
There is no need to call Convert.ToInt32 in:
new IPEndPoint(IPAddress.Parse("192.168.1.100"), Convert.ToInt32(2000));
2000 is already an int.
preface:
I've been stumped on this for awhile now and am not having much luck finding what I need.
I have a C# (.NET 3.5) Service. One thread acts as an asynchronous listener for incoming TCP connections. When data comes in I spawn off a new worker thread to handle the data, and sends an acknowledgement back.
On a second thread in the same service we send commands out, until today it would gather information from the data base, build a new socket, connect then ship the command and I'm using the Socket.Receive to invoke blocking and wait for a response (or until a timeout occurrs).
Everything has been working great until a new client has a need to send data to us so fast (5-10 second intervals) that we can no longer open a new socket to get a command through. So I started looking into when a command needs to be sent that the "listener" thread has a client connected. If that client is connected currently use that socket instead of creating a new one.
Issue:
I'm to the point where I can send my command back on the same socket the listener receives, but when the client sends data back as the response it takes twice for the Socket.Receive method to actually fire thinking it received data. The first time it gets into my listener class, the 2nd time, in my command class where I actually want it to be.
Question:
Is there some option or something I need to do before calling my Socket.Receive method to ensure the data gets to the correct place?
In my listener class I have a list of objects "CSocketPacket"
public class CSocketPacket
{
public CSocketPacket(System.Net.Sockets.Socket socket)
{
thisSocket = socket;
this.IpAddress =
((System.Net.IPEndPoint)socket.RemoteEndPoint).Address.ToString();
}
public System.Net.Sockets.Socket thisSocket;
public byte[] dataBuffer = new byte[BUFFER_SIZE];
public string IpAddress; //Use this to search for the socket
}
Then when I send a command I'm creating a new tcp socket object:
client = new Socket(
AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
IPEndPoint ep = new IPEndPoint(
IPAddress.Parse(Strings.Trim(ipAddress)), port);
IPEndPoint LocalIp = new IPEndPoint(IPAddress.Parse(
System.Configuration.ConfigurationManager.AppSettings["SourceIP"]), port);
then I'm looking into my listener class list to see if that socket is connected:
if (listener.SocketExists(ipAddress))
{
// set the client socket in this class to the
// instance of the socket from the listener class
SocketIndex = listener.FindSocketInList(ipAddress);
if (SocketIndex != -1)
{
// might need to figure out how to avoid copying the socket
// to a new variable ???
client = listener.ConnectedSockets[SocketIndex].thisSocket;
SocketBeingReUsed = true;
}
}
else
{
// try to connect to the client
client.Connect(ep);
}
finally I go through my steps of sending and receiving
if (client.Connected)
{
if (client.Poll(1000, SelectMode.SelectWrite))
{
int sentAmount = Send(ref client);
client.ReceiveTimeout = 90000; //90 seconds
returnData = ReceiveData(ref client, sentAmount);
}
}
everything works up to the point in my ReceiveData(ref client, sentAmount) method where I call the Socket.Receive(data, total, Socket.ReceiveBufferSize, SocketFlags.None); method.
I've been using a tool called Hercules to test sending/receiving packets across two machines on my home network.
Does anyone have any ideas of what I can do to solve this? I do apologize for such a lengthy question but I want to try to give as much info and not paste my entire project. I'm up for any suggestions.
Disclaimer: I wrote this code approx 3 years ago, so I'm pry doing things I shouldn't be I'm sure :P
Thanks to all who read this.
Sincerely,
Chris
OK, so now I'm following along! Given what you've said in the comments above, then the way to solve the problem is to have a single class/thread that reads from the socket (which is the correct way to read from sockets anyway) and then it will coordinate which class gets the data. I think it might work a little like the Command Design Pattern.