I want to have my application listen for http requests on port 8040. I am using the code below which is a sample that I copied from the web. The problem with the way I have it structured is once a request comes in and a response is generated the listener is closed and the task is returned. How can I restructure the code to keep the listener alive for subsequent requests?
Task.Factory.StartNew<bool>(() =>
{
System.Net.HttpListener listener = new System.Net.HttpListener();
listener.Prefixes.Add("http://*:8040/");
listener.Start();
Console.WriteLine("Listening...");
// Note: The GetContext method blocks while waiting for a request.
HttpListenerContext context = listener.GetContext();
HttpListenerRequest request = context.Request;
// Obtain a response object.
HttpListenerResponse response = context.Response;
// Construct a response.
string responseString = "<HTML><BODY> Hello world!</BODY></HTML>";
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
// Get a response stream and write the response to it.
response.ContentLength64 = buffer.Length;
System.IO.Stream output = response.OutputStream;
output.Write(buffer, 0, buffer.Length);
// You must close the output stream.
output.Close();
listener.Stop();
return true;
});
I use the Async method. I use the BeginGetContext for set a callback, so when it receives a request it runs the method RequestCallback.
In the requestCallback I get some data about the request and then I write some data to the response.
Finally I run BeginGetContext again, for process new requests.
Results
In Firefox I got:
Hello Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
In Chrome I got:
Hello Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
Code
public MainWindow()
{
InitializeComponent();
System.Net.HttpListener listener = new System.Net.HttpListener();
listener.Prefixes.Add("http://localhost:8040/");
listener.Start();
listener.BeginGetContext(RequestCallback, listener);
}
private void RequestCallback(IAsyncResult ar)
{
HttpListener listener = (HttpListener) ar.AsyncState;
var context = listener.EndGetContext(ar);
var userAgent = context.Request.UserAgent;
var responseMsg = "Hello " + userAgent;
var responseMsgBytes = Encoding.UTF8.GetBytes(responseMsg);
context.Response.ContentLength64 = responseMsgBytes.Length; //Response msg size
context.Response.OutputStream.Write(responseMsgBytes,0,responseMsgBytes.Length);
context.Response.OutputStream.Close();
listener.BeginGetContext(RequestCallback, listener); //Enable new requests
}
Also take a look to this post
I hope this can help you.
Related
I want to use the US as a country to access this
url = http://www.tillys.com/product/Say-What/Short-Dresses/SAY-WHAT--Ribbed-Tank-Midi-Dress/Heather-Grey/285111595,
I've tried with cookies and all but the url still it redirects to the site's home page.
I want to see if there is any way i can access this page. Below is the code with which i am trying.
Below is the function with which i am trying to do this:
public static string getUrlContent (string url)
{
var myHttpWebRequest = (HttpWebRequest)WebRequest.Create(url);
myHttpWebRequest.Method = "GET";
myHttpWebRequest.AllowAutoRedirect = true;
myHttpWebRequest.ContentLength = 0;
myHttpWebRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
myHttpWebRequest.Headers.Add("Cookie", "=en%5FUS;");
myHttpWebRequest.UserAgent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/49.0.2623.108 Chrome/49.0.2623.108 Safari/537.36";
//myHttpWebRequest.Headers.Add("Accept-Encoding", "gzip, deflate, sdch");
myHttpWebRequest.Headers.Add("Accept-Language", "en-US,en;q=0.8");
myHttpWebRequest.Headers.Add("Cookie", "wlcme=true");
//myHttpWebRequest.CookieContainer = new CookieContainer();
//myHttpWebRequest.Headers.Add("X-Macys-ClientId", "NavApp");
var response = (HttpWebResponse)myHttpWebRequest.GetResponse();
var rmyResponseHeaders = response.Headers;
Console.WriteLine ("Content length is {0}", response.ContentLength);
Console.WriteLine ("Content type is {0}", response.ContentType);
// Get the stream associated with the response.
Stream receiveStream = response.GetResponseStream ();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader (receiveStream, Encoding.UTF8);
//Console.WriteLine ("Response stream received.");
Console.WriteLine (readStream.ReadToEnd ());
var josnStr = readStream.ReadToEnd ();
Console.WriteLine (josnStr);
return josnStr;
//Encoding enc1 = Encoding.GetEncoding(1252);
}
If the site www.tillys.com is using Geo-fencing it will show you different content based on a lookup of the requesting IP address. In this case there's nothing C# or other languages can do.
You'll need to either proxy your request through a VPN (see How to send WebRequest via proxy?) or deploy your code to a data center in the US. For example, if you use Azure you can deploy to several different data centers through out the world including several data centers in the US. Once your code is running in the US it should be able to access the US version of the page.
I want to read Https page using TcpClient. I use below code
var client = new TcpClient(url, 443);//"127.0.0.1", 8888);// Fiddler port
client.SendTimeout = 30000;
Stream responseStream = client.GetStream();
// send CONNECT request to server
byte[] tunnelRequest = Encoding.ASCII.GetBytes("CONNECT www.google.com:443 HTTP/1.1\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:44.0) Gecko/20100101 Firefox/35.0\r\nProxy-Connection: keep-alive\r\nConnection: keep-alive\r\nHost: www.google.com:443\r\n\r\n");
responseStream.Write(tunnelRequest, 0, tunnelRequest.Length);
responseStream.Flush();
// read CONNECT response
string connectResponse = ReadResponse(responseStream);
Console.WriteLine("server connect response : " + connectResponse);
that send CONNECT request to host (google.com)
CONNECT www.google.com:443 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0
Proxy-Connection: keep-alive
Connection: keep-alive
Host: www.google.com:443
respone must be somthing like this
HTTP/1.1 200 Connection Established
StartTime: 22:42:38.774
Connection: close
but responseStream return nothing. when I use Fiddler as a proxy
var client = new TcpClient("127.0.0.1", 8888);
it works fine and return 200 response. There is something wrong that Fiddler fixed it?
I use windows 8.1 and test with .Net 2 and 4.5.1.
Base on rfc CONNECT method used when proxy exists.
Since TLS, in particular, requires end-to-end connectivity to provide
authentication and prevent man-in-the-middle attacks, this memo
specifies the CONNECT method to establish a tunnel across proxies.
In case of direct communication :
var client = new TcpClient("127.0.0.1", 8888);//url, 443);//
client.SendTimeout = 30000;
Stream responseStream = client.GetStream();
// Wrap in SSL stream
SslStream sslStream = new SslStream(responseStream);
sslStream.AuthenticateAsClient(url);
byte[] byts = Encoding.ASCII.GetBytes("GET / HTTP/1.1\r\nHost: www.google.com\r\nConnection: keep-alive\r\n\r\n");
sslStream.Write(byts, 0, byts.Length);
var str = ReadResponse(sslStream);
Just a piece of code
WebClient wc = new WebClient();
String str = wc.DownloadString(new Uri("http://content.warframe.com/dynamic/rss.php"));
And I got exception:
An unhandled exception of type 'System.Net.WebException' occurred in
System.dll
Additional information: The underlying connection was closed: The
connection was closed unexpectedly.
I've head that this is a bug in .NET (I am using 3.5), but I tried other methods to obtain the content of this link (its rss, xml). No lucky shot yet
var webrequest = (WebRequest)HttpWebRequest.Create(#"http://content.warframe.com/dynamic/rss.php");
var resp = webrequest.GetResponse();
//HttpWebResponse webresponse = (HttpWebResponse)webrequest.GetResponse(); // Wont work also
This code above won't work either, both casts the same exception
Fiddler logs:
SESSION STATE: Aborted.
Response Entity Size: 512 bytes.
== FLAGS ==================
BitFlags: [ResponseGeneratedByFiddler] 0x100
X-ABORTED-WHEN: Done
X-CLIENTIP: 127.0.0.1
X-CLIENTPORT: 2747
X-EGRESSPORT: 2748
X-FAILSESSION-WHEN: ReadingResponse
X-HOSTIP: 205.185.216.10
X-PROCESSINFO: willitwork.vshost:3300
== TIMING INFO ============
ClientConnected: 10:29:11.706
ClientBeginRequest: 10:29:11.713
GotRequestHeaders: 10:29:11.713
ClientDoneRequest: 10:29:11.713
Determine Gateway: 0ms
DNS Lookup: 164ms
TCP/IP Connect: 74ms
HTTPS Handshake: 0ms
ServerConnected: 10:29:11.953
FiddlerBeginRequest: 10:29:11.953
ServerGotRequest: 10:29:11.953
ServerBeginResponse: 10:29:12.372
GotResponseHeaders: 00:00:00.000
ServerDoneResponse: 10:29:12.372
ClientBeginResponse: 10:29:12.385
ClientDoneResponse: 10:29:12.385
Overall Elapsed: 0:00:00.672
The response was buffered before delivery to the client.
== WININET CACHE INFO ============
This URL is not present in the WinINET cache. [Code: 2]
* Note: Data above shows WinINET's current cache state, not the state at the time of the request.
* Note: Data above shows WinINET's Medium Integrity (non-Protected Mode) cache only.
Also - 504, this does not makes sense since I can get data from link via chrome / firefox / ie...
I just did it to work in other language, but I am forced to do it with C# (I' ve made 2 much code to rewrite it)
I've added some settings like fiddler said
myHttpWebRequest1.ProtocolVersion = HttpVersion.Version11;
myHttpWebRequest1.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36";
myHttpWebRequest1.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
At least now I get 504 error instead of "unknown", but I can still view the content via webbrowser, so the 504 error is fake
Edit: There is no response error when I added
myHttpWebRequest1.Headers["Accept-Encoding"] = "gzip";
but now the output is messed and unreadable
I have same error.
You can add User Agent to your httpRequest
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
Ok, i got this all fixes & working!
static void Main(string[] args)
{
Uri url = new Uri(#"http://content.warframe.com/dynamic/rss.php");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
// MAGIC LINE GOES HERE \/
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
// Assign the response object of HttpWebRequest to a HttpWebResponse variable.
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream streamResponse = response.GetResponseStream())
{
using (StreamReader streamRead = new StreamReader(streamResponse))
{
Char[] readBuff = new Char[2000];
int count = streamRead.Read(readBuff, 0, 2000);
while (count > 0)
{
String outputData = new String(readBuff, 0, count);
Console.Write(outputData);
count = streamRead.Read(readBuff, 0, 2000);
}
}
}
}
Console.ReadKey();
}
Besides of not-using WebClient.DownloadString method i had to add decompresion line
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
Thanks for tips (especially fiddler one, Decode button saved my time to find what's wrong)
Check this answer:
..The underlying connection was closed: An unexpected error occurred on a receive
So this may work for you:
var webRequest = (HttpWebRequest)HttpWebRequest.Create(#"http://content.warframe.com/dynamic/rss.php");
webRequest.KeepAlive = false;
var resp = webRequest.GetResponse();
EDIT:
You are right, check rather this:
http://msdn.microsoft.com/cs-cz/library/system.net.httpwebrequest.keepalive%28v=vs.110%29.aspx
Here is working code that will print out the recieved response content:
static void Main(string[] args)
{
// Create a new HttpWebRequest object.Make sure that
// a default proxy is set if you are behind a firewall.
HttpWebRequest myHttpWebRequest1 =
(HttpWebRequest)WebRequest.Create(#"http://content.warframe.com/dynamic/rss.php");
myHttpWebRequest1.KeepAlive=false;
// Assign the response object of HttpWebRequest to a HttpWebResponse variable.
HttpWebResponse myHttpWebResponse1 =
(HttpWebResponse)myHttpWebRequest1.GetResponse();
Console.WriteLine("\nThe HTTP request Headers for the first request are: \n{0}", myHttpWebRequest1.Headers);
Stream streamResponse = myHttpWebResponse1.GetResponseStream();
StreamReader streamRead = new StreamReader(streamResponse);
Char[] readBuff = new Char[256];
int count = streamRead.Read(readBuff, 0, 256);
Console.WriteLine("The contents of the Html page are.......\n");
while (count > 0)
{
String outputData = new String(readBuff, 0, count);
Console.Write(outputData);
count = streamRead.Read(readBuff, 0, 256);
}
Console.WriteLine();
// Close the Stream object.
streamResponse.Close();
streamRead.Close();
Console.ReadKey();
}
Following on from the post How to create a simple proxy in C#? I have been playing around with implementing a basic proxy.
Where I am getting stuck and confused is trying to issue a WebRequest with the information provided in the original request.
Using the following code.
WebRequest webRequest = WebRequest.Create("http://www.google.com");
(webRequest as HttpWebRequest).UserAgent = "MOZILLA/5.0 (WINDOWS NT 6.1; WOW64) APPLEWEBKIT/537.1 (KHTML, LIKE GECKO) CHROME/21.0.1180.75 SAFARI/537.1";
webRequest.Method = "GET";
WebResponse webResponse = webRequest.GetResponse();
Stream responseStream = webResponse.GetResponseStream();
byte[] responseBytes = responseStream.ReadFully();
I can successfully issue a request and return the page content.
However when I put it inside a Proxy request (IE: TcpListener) like such.
TcpListener _listener = new TcpListener(IPAddress.Any, 1234);
this._listener.Start();
byte[] bytes = new byte[1024];
while (true)
{
TcpClient client = this._listener.AcceptTcpClient();
NetworkStream networkStream = client.GetStream();
int i = networkStream.Read(bytes, 0, bytes.Length);
while (i != 0)
{
string data = System.Text.Encoding.ASCII.GetString(bytes, 0, i);
RequestHeader header = new RequestHeader(data.ToUpper());
WebRequest webRequest = WebRequest.Create(header.URL);
(webRequest as HttpWebRequest).UserAgent = header.UserAgent;
webRequest.Method = "GET";
WebResponse webResponse = webRequest.GetResponse(); //It gets here and never returns
Stream responseStream = webResponse.GetResponseStream();
byte[] responseBytes = responseStream.ReadFully();
networkStream.Write(responseBytes, 0, responseBytes.Length);
i = networkStream.Read(bytes, 0, bytes.Length);
}
client.Close();
}
It blocks at the line WebResponse webResponse = webRequest.GetResponse(); and never returns.
This has definitely got nothing to do with the data provided by the RequestHeader class I created as I've also tried hardcoding the values.
I'm assuming I'm missing something fundamental about the way sockets work in such a scenario and the approach required. Hopefully someone can clarify for me.
Yeah, you are assuming you have read all the header.
Instead of this, some kind of state machine should be implemented to parse the incoming HTTP-request. The state machine must collect the information about the request and, of course, detect the end of request, then you process the request (proxy stuff) and send the response. Just Google C# http state machine for examples.
So it turned out to be a proxy issue.
Basically for testing I needed to set the machine proxy to 127.0.0.1:1234 or similar.
This in turn was being used in the default settings when initializing a WebRequest.
So all I needed to do in the end was the following to bypass the proxy.
(webRequest as HttpWebRequest).UserAgent = header.UserAgent;
webRequest.Method = "GET";
webRequest.Proxy = null; //Adding this line cleared the proxy.
I am trying to send HTTP request and recieve responce from the server over C# sockets, and i'm new with this language.
I've wrote following code (IP resolved correctly):
IPEndPoint RHost = new IPEndPoint(IP, Port);
Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
socket.Connect(RHost);
String HTTPRequestHeaders_String = "GET ?q=fdgdfg HTTP/1.0
Host: google.com
Keep-Alive: 300
Connection: Keep-Alive
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.205 Safari/534.16
Referer: http://google.com/";
MessageBox.Show(HTTPRequestHeaders_String, "Request");
byte[] HTTPRequestHeaders = System.Text.Encoding.ASCII.GetBytes(HTTPRequestHeaders_String);
socket.Send(HTTPRequestHeaders, SocketFlags.None);
String Response = "";
byte[] buffer = new byte[(int) socket.ReceiveBufferSize];
int bytes;
do
{
// On this lane program stops to react
bytes = socket.Receive(buffer);
// This line cannot be reached, tested with breakpoint
Response += Encoding.ASCII.GetString(buffer, 0, bytes);
}
while (bytes >= 0);
MessageBox.Show(Response, "Response");
What am i doing wrong? I need just to load full HTML of page, or at least few characters from response (i cant do even this).
I would suggest looking into the protocol itself if you want to do this raw, http://www.w3.org/Protocols/HTTP/1.0/spec.html#Request
And try sending the CRLF to terminate the request ;)
var webClient = new WebClient();
webClient.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
Stream responseStream = webClient.OpenRead("http://www.google.com");
if (responseStream != null)
{
var responseReader = new StreamReader(responseStream);
string response = responseReader.ReadToEnd();
MessageBox.Show(response);
}
I found the Mentalis Proxy to be extremely helpful in understanding the Http Request/Response cycle at a socket level: http://www.mentalis.org/soft/projects/proxy/
There is the TcpClient class which on one hand allows you to have a full control over the request (you create request body as string) and on the other hand is much simpler to use than a low-level socket.
http://msdn.microsoft.com/en-us/library/system.net.sockets.tcpclient.aspx