I am trying to send HTTP request and recieve responce from the server over C# sockets, and i'm new with this language.
I've wrote following code (IP resolved correctly):
IPEndPoint RHost = new IPEndPoint(IP, Port);
Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
socket.Connect(RHost);
String HTTPRequestHeaders_String = "GET ?q=fdgdfg HTTP/1.0
Host: google.com
Keep-Alive: 300
Connection: Keep-Alive
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.205 Safari/534.16
Referer: http://google.com/";
MessageBox.Show(HTTPRequestHeaders_String, "Request");
byte[] HTTPRequestHeaders = System.Text.Encoding.ASCII.GetBytes(HTTPRequestHeaders_String);
socket.Send(HTTPRequestHeaders, SocketFlags.None);
String Response = "";
byte[] buffer = new byte[(int) socket.ReceiveBufferSize];
int bytes;
do
{
// On this lane program stops to react
bytes = socket.Receive(buffer);
// This line cannot be reached, tested with breakpoint
Response += Encoding.ASCII.GetString(buffer, 0, bytes);
}
while (bytes >= 0);
MessageBox.Show(Response, "Response");
What am i doing wrong? I need just to load full HTML of page, or at least few characters from response (i cant do even this).
I would suggest looking into the protocol itself if you want to do this raw, http://www.w3.org/Protocols/HTTP/1.0/spec.html#Request
And try sending the CRLF to terminate the request ;)
var webClient = new WebClient();
webClient.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
Stream responseStream = webClient.OpenRead("http://www.google.com");
if (responseStream != null)
{
var responseReader = new StreamReader(responseStream);
string response = responseReader.ReadToEnd();
MessageBox.Show(response);
}
I found the Mentalis Proxy to be extremely helpful in understanding the Http Request/Response cycle at a socket level: http://www.mentalis.org/soft/projects/proxy/
There is the TcpClient class which on one hand allows you to have a full control over the request (you create request body as string) and on the other hand is much simpler to use than a low-level socket.
http://msdn.microsoft.com/en-us/library/system.net.sockets.tcpclient.aspx
Related
I'm trying to automate a WebSocket service that denies connection unless you send a user agent with the CONNECT request.
I tried sending the upgrade request with HttpWebRequest and setting User-Agent using the property.
Using Fiddler to debug the request this was sent out:
CONNECT *.*.com:443 HTTP/1.1
Host: *.*.com:443
Connection: keep-alive
How do I add the User-Agent string to the CONNECT request and then upgrade to using WebSocket protocol?
My code so far:
public void Login ( Action onEnd = null ) {
var req = CreateUpgradeRequest();
var res = GetResponse(req);
}
private HttpWebRequest CreateUpgradeRequest ( ) {
HttpWebRequest request = WebRequest.Create("https://lobby35.runescape.com/") as HttpWebRequest;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36";
request.Connection = "Upgrade";
SetWebSocketHeader(request, "Key", "5LENZfSifyj/Rw1ghTvpgw==");
SetWebSocketHeader(request, "Version", "13");
SetWebSocketHeader(request, "Extensions", "permessage-deflate; client_max_window_bits");
SetWebSocketHeader(request, "Protocol", "jagex");
return request;
}
You cannot use WebRequest to create a websocket connection. You will need ClientWebSocket and use `ClientWebSocket.Options.SetRequestHeader.
Note, you may have issues adding that header: Setting "User-Agent" HTTP header in ClientWebSocket
Update: Since you cannot add that header with ClientWebSocket try with Websocket4Net.
I want to have my application listen for http requests on port 8040. I am using the code below which is a sample that I copied from the web. The problem with the way I have it structured is once a request comes in and a response is generated the listener is closed and the task is returned. How can I restructure the code to keep the listener alive for subsequent requests?
Task.Factory.StartNew<bool>(() =>
{
System.Net.HttpListener listener = new System.Net.HttpListener();
listener.Prefixes.Add("http://*:8040/");
listener.Start();
Console.WriteLine("Listening...");
// Note: The GetContext method blocks while waiting for a request.
HttpListenerContext context = listener.GetContext();
HttpListenerRequest request = context.Request;
// Obtain a response object.
HttpListenerResponse response = context.Response;
// Construct a response.
string responseString = "<HTML><BODY> Hello world!</BODY></HTML>";
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
// Get a response stream and write the response to it.
response.ContentLength64 = buffer.Length;
System.IO.Stream output = response.OutputStream;
output.Write(buffer, 0, buffer.Length);
// You must close the output stream.
output.Close();
listener.Stop();
return true;
});
I use the Async method. I use the BeginGetContext for set a callback, so when it receives a request it runs the method RequestCallback.
In the requestCallback I get some data about the request and then I write some data to the response.
Finally I run BeginGetContext again, for process new requests.
Results
In Firefox I got:
Hello Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
In Chrome I got:
Hello Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
Code
public MainWindow()
{
InitializeComponent();
System.Net.HttpListener listener = new System.Net.HttpListener();
listener.Prefixes.Add("http://localhost:8040/");
listener.Start();
listener.BeginGetContext(RequestCallback, listener);
}
private void RequestCallback(IAsyncResult ar)
{
HttpListener listener = (HttpListener) ar.AsyncState;
var context = listener.EndGetContext(ar);
var userAgent = context.Request.UserAgent;
var responseMsg = "Hello " + userAgent;
var responseMsgBytes = Encoding.UTF8.GetBytes(responseMsg);
context.Response.ContentLength64 = responseMsgBytes.Length; //Response msg size
context.Response.OutputStream.Write(responseMsgBytes,0,responseMsgBytes.Length);
context.Response.OutputStream.Close();
listener.BeginGetContext(RequestCallback, listener); //Enable new requests
}
Also take a look to this post
I hope this can help you.
I want to read Https page using TcpClient. I use below code
var client = new TcpClient(url, 443);//"127.0.0.1", 8888);// Fiddler port
client.SendTimeout = 30000;
Stream responseStream = client.GetStream();
// send CONNECT request to server
byte[] tunnelRequest = Encoding.ASCII.GetBytes("CONNECT www.google.com:443 HTTP/1.1\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:44.0) Gecko/20100101 Firefox/35.0\r\nProxy-Connection: keep-alive\r\nConnection: keep-alive\r\nHost: www.google.com:443\r\n\r\n");
responseStream.Write(tunnelRequest, 0, tunnelRequest.Length);
responseStream.Flush();
// read CONNECT response
string connectResponse = ReadResponse(responseStream);
Console.WriteLine("server connect response : " + connectResponse);
that send CONNECT request to host (google.com)
CONNECT www.google.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: www.google.com:443
respone must be somthing like this
HTTP/1.1 200 Connection Established
StartTime: 22:42:38.774
Connection: close
but responseStream return nothing. when I use Fiddler as a proxy
var client = new TcpClient("127.0.0.1", 8888);
it works fine and return 200 response. There is something wrong that Fiddler fixed it?
I use windows 8.1 and test with .Net 2 and 4.5.1.
Base on rfc CONNECT method used when proxy exists.
Since TLS, in particular, requires end-to-end connectivity to provide
authentication and prevent man-in-the-middle attacks, this memo
specifies the CONNECT method to establish a tunnel across proxies.
In case of direct communication :
var client = new TcpClient("127.0.0.1", 8888);//url, 443);//
client.SendTimeout = 30000;
Stream responseStream = client.GetStream();
// Wrap in SSL stream
SslStream sslStream = new SslStream(responseStream);
sslStream.AuthenticateAsClient(url);
byte[] byts = Encoding.ASCII.GetBytes("GET / HTTP/1.1\r\nHost: www.google.com\r\nConnection: keep-alive\r\n\r\n");
sslStream.Write(byts, 0, byts.Length);
var str = ReadResponse(sslStream);
Just a piece of code
WebClient wc = new WebClient();
String str = wc.DownloadString(new Uri("http://content.warframe.com/dynamic/rss.php"));
And I got exception:
An unhandled exception of type 'System.Net.WebException' occurred in
System.dll
Additional information: The underlying connection was closed: The
connection was closed unexpectedly.
I've head that this is a bug in .NET (I am using 3.5), but I tried other methods to obtain the content of this link (its rss, xml). No lucky shot yet
var webrequest = (WebRequest)HttpWebRequest.Create(#"http://content.warframe.com/dynamic/rss.php");
var resp = webrequest.GetResponse();
//HttpWebResponse webresponse = (HttpWebResponse)webrequest.GetResponse(); // Wont work also
This code above won't work either, both casts the same exception
Fiddler logs:
SESSION STATE: Aborted.
Response Entity Size: 512 bytes.
== FLAGS ==================
BitFlags: [ResponseGeneratedByFiddler] 0x100
X-ABORTED-WHEN: Done
X-CLIENTIP: 127.0.0.1
X-CLIENTPORT: 2747
X-EGRESSPORT: 2748
X-FAILSESSION-WHEN: ReadingResponse
X-HOSTIP: 205.185.216.10
X-PROCESSINFO: willitwork.vshost:3300
== TIMING INFO ============
ClientConnected: 10:29:11.706
ClientBeginRequest: 10:29:11.713
GotRequestHeaders: 10:29:11.713
ClientDoneRequest: 10:29:11.713
Determine Gateway: 0ms
DNS Lookup: 164ms
TCP/IP Connect: 74ms
HTTPS Handshake: 0ms
ServerConnected: 10:29:11.953
FiddlerBeginRequest: 10:29:11.953
ServerGotRequest: 10:29:11.953
ServerBeginResponse: 10:29:12.372
GotResponseHeaders: 00:00:00.000
ServerDoneResponse: 10:29:12.372
ClientBeginResponse: 10:29:12.385
ClientDoneResponse: 10:29:12.385
Overall Elapsed: 0:00:00.672
The response was buffered before delivery to the client.
== WININET CACHE INFO ============
This URL is not present in the WinINET cache. [Code: 2]
* Note: Data above shows WinINET's current cache state, not the state at the time of the request.
* Note: Data above shows WinINET's Medium Integrity (non-Protected Mode) cache only.
Also - 504, this does not makes sense since I can get data from link via chrome / firefox / ie...
I just did it to work in other language, but I am forced to do it with C# (I' ve made 2 much code to rewrite it)
I've added some settings like fiddler said
myHttpWebRequest1.ProtocolVersion = HttpVersion.Version11;
myHttpWebRequest1.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36";
myHttpWebRequest1.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
At least now I get 504 error instead of "unknown", but I can still view the content via webbrowser, so the 504 error is fake
Edit: There is no response error when I added
myHttpWebRequest1.Headers["Accept-Encoding"] = "gzip";
but now the output is messed and unreadable
I have same error.
You can add User Agent to your httpRequest
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
Ok, i got this all fixes & working!
static void Main(string[] args)
{
Uri url = new Uri(#"http://content.warframe.com/dynamic/rss.php");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// MAGIC LINE GOES HERE \/
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
// Assign the response object of HttpWebRequest to a HttpWebResponse variable.
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream streamResponse = response.GetResponseStream())
{
using (StreamReader streamRead = new StreamReader(streamResponse))
{
Char[] readBuff = new Char[2000];
int count = streamRead.Read(readBuff, 0, 2000);
while (count > 0)
{
String outputData = new String(readBuff, 0, count);
Console.Write(outputData);
count = streamRead.Read(readBuff, 0, 2000);
}
}
}
}
Console.ReadKey();
}
Besides of not-using WebClient.DownloadString method i had to add decompresion line
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
Thanks for tips (especially fiddler one, Decode button saved my time to find what's wrong)
Check this answer:
..The underlying connection was closed: An unexpected error occurred on a receive
So this may work for you:
var webRequest = (HttpWebRequest)HttpWebRequest.Create(#"http://content.warframe.com/dynamic/rss.php");
webRequest.KeepAlive = false;
var resp = webRequest.GetResponse();
EDIT:
You are right, check rather this:
http://msdn.microsoft.com/cs-cz/library/system.net.httpwebrequest.keepalive%28v=vs.110%29.aspx
Here is working code that will print out the recieved response content:
static void Main(string[] args)
{
// Create a new HttpWebRequest object.Make sure that
// a default proxy is set if you are behind a firewall.
HttpWebRequest myHttpWebRequest1 =
(HttpWebRequest)WebRequest.Create(#"http://content.warframe.com/dynamic/rss.php");
myHttpWebRequest1.KeepAlive=false;
// Assign the response object of HttpWebRequest to a HttpWebResponse variable.
HttpWebResponse myHttpWebResponse1 =
(HttpWebResponse)myHttpWebRequest1.GetResponse();
Console.WriteLine("\nThe HTTP request Headers for the first request are: \n{0}", myHttpWebRequest1.Headers);
Stream streamResponse = myHttpWebResponse1.GetResponseStream();
StreamReader streamRead = new StreamReader(streamResponse);
Char[] readBuff = new Char[256];
int count = streamRead.Read(readBuff, 0, 256);
Console.WriteLine("The contents of the Html page are.......\n");
while (count > 0)
{
String outputData = new String(readBuff, 0, count);
Console.Write(outputData);
count = streamRead.Read(readBuff, 0, 256);
}
Console.WriteLine();
// Close the Stream object.
streamResponse.Close();
streamRead.Close();
Console.ReadKey();
}
Recently I've been experiencing a nightmare with .NET (C#) and SOAP Transmissions.
I've got to consume a webservice (which was supposed to be an easy task) but it become terrible and nothing seem to works.
HttpWebRequest webRequest = (HttpWebRequest)System.Net.WebRequest.Create("http://api.myapi.com/apis/services/theapi");
webRequest.AllowAutoRedirect = true;
webRequest.Timeout = 1000 * 30;
webRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
webRequest.PreAuthenticate = true;
webRequest.Method = "POST";
webRequest.Headers.Add("SOAPAction: \"\"");
webRequest.Accept = "text/xml";
WebResponse webResponse = null;
try
{
webResponse = webRequest.GetResponse();
Stream Stream = webRequest.GetRequestStream();
string SoapEnvelope = "<soap:Envelope>...SOAP CODE ...</soap:Envelope>";
StreamReader streamReader = new StreamReader(webResponse.GetResponseStream(), System.Text.Encoding.UTF8);
XmlDocument SoapEnvelopeXML = new XmlDocument();
SoapEnvelopeXML.LoadXml(SoapEnvelope);
SoapEnvelopeXML.Save(Stream);
string result = streamReader.ReadToEnd();
return result;
}
When I try to sniff the packages by using Wireshark, that's what I get:
---- CLIENT INPUT ------
POST /apis/services/theapi HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
SOAPAction: ""
Accept: text/xml
Host: api.myapi.com
Connection: Keep-Alive
---- SERVER ANSWER ------
HTTP/1.1 500 Internal Server Error
Date: Sat, 14 May 2011 15:35:32 GMT
X-Powered-By: Servlet 2.4; JBoss-4.0.5.GA (build: CVSTag=Branch_4_0 date=200610162339)/Tomcat-5.5
Content-Type: text/xml;charset=ISO-8859-1
Content-Length: 225
Connection: close
X-Pad: avoid browser bug
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><soap:Fault><faultcode>soap:Client</faultcode><faultstring>Error reading XMLStreamReader.</faultstring></soap:Fault></soap:Body></soap:Envelope>
As expected, since I haven't posted a Soap Request (no XML in the request), I receive a SOAP Fault and ERROR 500.
Any ideas?
I've got to somehow do this manually. I've tried to use even TCPClient (to deal with it in a lower level), but all my attempts were frustrated.
You should use the VS Add Service Reference wizard to load the service into your project. The Add Service Reference generates classes to work with the api automatically at a much higher level, from the url of the service API endpoint. It will look something like this:
MyApiClient client = new MyApiClient();
MyApiResult result;
try {
client.Open();
result = client.CallMethod(param1, param2, ...);
client.Close();
} catch (Exception ex) {
// do something with FaultException or API error
}
// do something with the result returned, if needed
If you've done it correct, you shouldn't have to deal with HttpWebRequest, explicit URL's, or hand-typing out any SOAP XML at all!!