Develop Reference

Posting to a form, can't get the address right

I have intercepted an HTTP POST as follows
Header
Key Value
Request POST /east-berkshire/local/quick_search HTTP/1.1
Accept text/html, application/xhtml+xml, */*
Referer https://www.netmums.com/east-berkshire/local/index/childcare/nannies-au-pairs
Accept-Language en-GB
User-Agent Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Content-Type application/x-www-form-urlencoded
Accept-Encoding gzip, deflate
Host www.netmums.com
Content-Length 107
DNT 1
Connection Keep-Alive
Cache-Control no-cache
Cookie AMCV_44326DF2572396FB7F000101%40AdobeOrg=817868104%7CMCMID%7C34574735755395522184062187835447062918%7CMCAAMLH-1486721296%7C6%7CMCAAMB-1486721296%7CNRX38WO0n5BH8Th-nqAG_A%7CMCOPTOUT-1486123696s%7CNONE; _ga=GA1.2.258060262.1486116497; _gat=1; _lp4_u=dZXxbBpqGf; __qca=P0-238174588-1486116496764; _tynt_crtg=; aam_uuid=34158303305859258534090346121149142657; __gads=ID=b3ba42a045f2be6a:T=1486116505:S=ALNI_MZHsVecqphdMO7SI-l4IEGrCyFpsg; AMCVS_44326DF2572396FB7F000101%40AdobeOrg=1; ABTastySession=LiwioHashMRASN%3Anull%5E%7C%5ELiwioUTMC%3A1; ABTasty=ABTastyUTMB%3A1%5E%7C%5ELiwioTracking%3A17020310101198682%5E%7C%5EsegmentationTracking%3A17020310101198682%5E%7C%5ELiwioUTMA%3A0.1.1486116611618.0.1486116611618.2; firstvisit=1; Cake=3qdc1afjmdvq0fg9kdunu2okn4; NetmumsLocation=east-berkshire; OX_plg=swf|sl|shk|pm
Body
_method=POST&data%5BListing%5D%5Blisting_category_id%5D=2&data%5BListing%5D%5Blisting_subcategory_id%5D=211
I have written the following C# code to try simulate this
var request = WebRequest.Create("https://www.netmums.com/east-berkshire/local/quick_search") as HttpWebRequest;
if (request == null) throw new HttpRequestException("Could not create web request");
request.Method = "post";
request.ContentType = "application/x-www-form-urlencoded";
var bs = Encoding.ASCII.GetBytes("[Listing][listing_category_id]=2&[Listing][listing_subcategory_id]=211");
using (var reqStream = request.GetRequestStream())
reqStream.Write(bs, 0, bs.Length);
string result;
using (var response = request.GetResponse())
{
var stream = response.GetResponseStream();
if (stream == null) throw new HttpRequestException("No data returned");
var sr = new StreamReader(stream);
result = sr.ReadToEnd();
sr.Close();
}
However when I execute it, on the GetResponse() call I get the error
The remote server returned an error: (404) Not Found.
What am I doing wrong?

send CONNECT with User-Agent C#

I'm trying to automate a WebSocket service that denies connection unless you send a user agent with the CONNECT request.
I tried sending the upgrade request with HttpWebRequest and setting User-Agent using the property.
Using Fiddler to debug the request this was sent out:
CONNECT *.*.com:443 HTTP/1.1
Host: *.*.com:443
Connection: keep-alive
How do I add the User-Agent string to the CONNECT request and then upgrade to using WebSocket protocol?
My code so far:
public void Login ( Action onEnd = null ) {
var req = CreateUpgradeRequest();
var res = GetResponse(req);
}
private HttpWebRequest CreateUpgradeRequest ( ) {
HttpWebRequest request = WebRequest.Create("https://lobby35.runescape.com/") as HttpWebRequest;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36";
request.Connection = "Upgrade";
SetWebSocketHeader(request, "Key", "5LENZfSifyj/Rw1ghTvpgw==");
SetWebSocketHeader(request, "Version", "13");
SetWebSocketHeader(request, "Extensions", "permessage-deflate; client_max_window_bits");
SetWebSocketHeader(request, "Protocol", "jagex");
return request;
}

You cannot use WebRequest to create a websocket connection. You will need ClientWebSocket and use `ClientWebSocket.Options.SetRequestHeader.
Note, you may have issues adding that header: Setting "User-Agent" HTTP header in ClientWebSocket
Update: Since you cannot add that header with ClientWebSocket try with Websocket4Net.

How to reuse a HttpListener?

I want to have my application listen for http requests on port 8040. I am using the code below which is a sample that I copied from the web. The problem with the way I have it structured is once a request comes in and a response is generated the listener is closed and the task is returned. How can I restructure the code to keep the listener alive for subsequent requests?
Task.Factory.StartNew<bool>(() =>
{
System.Net.HttpListener listener = new System.Net.HttpListener();
listener.Prefixes.Add("http://*:8040/");
listener.Start();
Console.WriteLine("Listening...");
// Note: The GetContext method blocks while waiting for a request.
HttpListenerContext context = listener.GetContext();
HttpListenerRequest request = context.Request;
// Obtain a response object.
HttpListenerResponse response = context.Response;
// Construct a response.
string responseString = "<HTML><BODY> Hello world!</BODY></HTML>";
byte[] buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
// Get a response stream and write the response to it.
response.ContentLength64 = buffer.Length;
System.IO.Stream output = response.OutputStream;
output.Write(buffer, 0, buffer.Length);
// You must close the output stream.
output.Close();
listener.Stop();
return true;
});

I use the Async method. I use the BeginGetContext for set a callback, so when it receives a request it runs the method RequestCallback.
In the requestCallback I get some data about the request and then I write some data to the response.
Finally I run BeginGetContext again, for process new requests.
Results
In Firefox I got:
Hello Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
In Chrome I got:
Hello Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
Code
public MainWindow()
{
InitializeComponent();
System.Net.HttpListener listener = new System.Net.HttpListener();
listener.Prefixes.Add("http://localhost:8040/");
listener.Start();
listener.BeginGetContext(RequestCallback, listener);
}
private void RequestCallback(IAsyncResult ar)
{
HttpListener listener = (HttpListener) ar.AsyncState;
var context = listener.EndGetContext(ar);
var userAgent = context.Request.UserAgent;
var responseMsg = "Hello " + userAgent;
var responseMsgBytes = Encoding.UTF8.GetBytes(responseMsg);
context.Response.ContentLength64 = responseMsgBytes.Length; //Response msg size
context.Response.OutputStream.Write(responseMsgBytes,0,responseMsgBytes.Length);
context.Response.OutputStream.Close();
listener.BeginGetContext(RequestCallback, listener); //Enable new requests
}
Also take a look to this post
I hope this can help you.

HTTP over C# sockets

I am trying to send HTTP request and recieve responce from the server over C# sockets, and i'm new with this language.
I've wrote following code (IP resolved correctly):
IPEndPoint RHost = new IPEndPoint(IP, Port);
Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
socket.Connect(RHost);
String HTTPRequestHeaders_String = "GET ?q=fdgdfg HTTP/1.0
Host: google.com
Keep-Alive: 300
Connection: Keep-Alive
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.205 Safari/534.16
Referer: http://google.com/";
MessageBox.Show(HTTPRequestHeaders_String, "Request");
byte[] HTTPRequestHeaders = System.Text.Encoding.ASCII.GetBytes(HTTPRequestHeaders_String);
socket.Send(HTTPRequestHeaders, SocketFlags.None);
String Response = "";
byte[] buffer = new byte[(int) socket.ReceiveBufferSize];
int bytes;
do
{
// On this lane program stops to react
bytes = socket.Receive(buffer);
// This line cannot be reached, tested with breakpoint
Response += Encoding.ASCII.GetString(buffer, 0, bytes);
}
while (bytes >= 0);
MessageBox.Show(Response, "Response");
What am i doing wrong? I need just to load full HTML of page, or at least few characters from response (i cant do even this).

I would suggest looking into the protocol itself if you want to do this raw, http://www.w3.org/Protocols/HTTP/1.0/spec.html#Request
And try sending the CRLF to terminate the request ;)

var webClient = new WebClient();
webClient.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
Stream responseStream = webClient.OpenRead("http://www.google.com");
if (responseStream != null)
{
var responseReader = new StreamReader(responseStream);
string response = responseReader.ReadToEnd();
MessageBox.Show(response);
}

I found the Mentalis Proxy to be extremely helpful in understanding the Http Request/Response cycle at a socket level: http://www.mentalis.org/soft/projects/proxy/

There is the TcpClient class which on one hand allows you to have a full control over the request (you create request body as string) and on the other hand is much simpler to use than a low-level socket.
http://msdn.microsoft.com/en-us/library/system.net.sockets.tcpclient.aspx

.NET (C#) and RAW Soap Transactions

Recently I've been experiencing a nightmare with .NET (C#) and SOAP Transmissions.
I've got to consume a webservice (which was supposed to be an easy task) but it become terrible and nothing seem to works.
HttpWebRequest webRequest = (HttpWebRequest)System.Net.WebRequest.Create("http://api.myapi.com/apis/services/theapi");
webRequest.AllowAutoRedirect = true;
webRequest.Timeout = 1000 * 30;
webRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
webRequest.PreAuthenticate = true;
webRequest.Method = "POST";
webRequest.Headers.Add("SOAPAction: \"\"");
webRequest.Accept = "text/xml";
WebResponse webResponse = null;
try
{
webResponse = webRequest.GetResponse();
Stream Stream = webRequest.GetRequestStream();
string SoapEnvelope = "<soap:Envelope>...SOAP CODE ...</soap:Envelope>";
StreamReader streamReader = new StreamReader(webResponse.GetResponseStream(), System.Text.Encoding.UTF8);
XmlDocument SoapEnvelopeXML = new XmlDocument();
SoapEnvelopeXML.LoadXml(SoapEnvelope);
SoapEnvelopeXML.Save(Stream);
string result = streamReader.ReadToEnd();
return result;
}
When I try to sniff the packages by using Wireshark, that's what I get:
---- CLIENT INPUT ------
POST /apis/services/theapi HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
SOAPAction: ""
Accept: text/xml
Host: api.myapi.com
Connection: Keep-Alive
---- SERVER ANSWER ------
HTTP/1.1 500 Internal Server Error
Date: Sat, 14 May 2011 15:35:32 GMT
X-Powered-By: Servlet 2.4; JBoss-4.0.5.GA (build: CVSTag=Branch_4_0 date=200610162339)/Tomcat-5.5
Content-Type: text/xml;charset=ISO-8859-1
Content-Length: 225
Connection: close
X-Pad: avoid browser bug
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><soap:Fault><faultcode>soap:Client</faultcode><faultstring>Error reading XMLStreamReader.</faultstring></soap:Fault></soap:Body></soap:Envelope>
As expected, since I haven't posted a Soap Request (no XML in the request), I receive a SOAP Fault and ERROR 500.
Any ideas?
I've got to somehow do this manually. I've tried to use even TCPClient (to deal with it in a lower level), but all my attempts were frustrated.

You should use the VS Add Service Reference wizard to load the service into your project. The Add Service Reference generates classes to work with the api automatically at a much higher level, from the url of the service API endpoint. It will look something like this:
MyApiClient client = new MyApiClient();
MyApiResult result;
try {
client.Open();
result = client.CallMethod(param1, param2, ...);
client.Close();
} catch (Exception ex) {
// do something with FaultException or API error
}
// do something with the result returned, if needed
If you've done it correct, you shouldn't have to deal with HttpWebRequest, explicit URL's, or hand-typing out any SOAP XML at all!!