Develop Reference

Handling HttpResponseMessage with persistent connection c #

I am using Httpclient C # .net
to connect to a url.
I need to make a GET request. This connection is persistent.
The problem is that I need to perform actions according to the notifications that the server sends me.
When executing the HttpResponseMessage, it hangs (being persistent it works and receiving notifications constantly, but it never gives me back control to be able to work according to the notifications of the server)
when canceling the execution I can see all notifications by console.
Is there any way to control HttpResponseMessage to be able to work with every response the server sends me?
Should I use another type of technology for this?
This is what I get back by console de "await response.Content.ReadAsStringAsync ();" when I cancel the operation.
<? xml version = "1.0" encoding = "UTF-8" standalone = "yes"?>
<message xmlns = "http://asdasd.com/asd/08/DS/Sync" xmlns: ns2 = "http: // asdadasdasdasd">
<event> KEEPALIVE </ event>
</ message>
<? xml version = "1.0" encoding = "UTF-8" standalone = "yes"?>
<message xmlns = "http://asdasd.com/asd/08/DS/Sync" xmlns: ns2 = "http: // asdadasdasdasd">
<event> KEEPALIVE </ event>
</ message>
<? xml version = "1.0" encoding = "UTF-8" standalone = "yes"?>
<message xmlns = "http://asdasd.com/asd/08/DS/Sync" xmlns: ns2 = "http: // asdadasdasdasd">
<event> KEEPALIVE </ event>
</ message>
every ('message') arrives approximately every 2 minutes I wish that each time I get one I can perform an action
I leave the code that ultilizo. Thank you
public async static Task<int> GetRequest(string url)
{
HttpClient client = new HttpClient();
HttpResponseMessage response = new HttpResponseMessage();
response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
Console.WriteLine("STATUS OF CONNECTION");
Console.WriteLine(response.StatusCode);
response.EnsureSuccessStatusCode();
var body = await response.Content.ReadAsStringAsync().ConfigureAwait(continueOnCapturedContext: false); ;
Console.WriteLine("ANSWER");
Console.WriteLine(body);
Console.WriteLine(" \n");
Console.WriteLine("\n ATTENTION!! Disconnected from the Persistence line \n");
Console.WriteLine("Connections done");
int result1 = 1;
return result1;
}

You are looking to parse partial HTTP responses, as and when each part is received. The HTTP protocol does not know/understand that you are sending/receiving multiple discrete messages separated by a newline in a single HTTP response.
As far as I know, HttpClient won't be much help to you because it is designed to receive one whole HTTP response. Rick Strahl has a blog on exactly this subject Using .NET HttpClient to capture partial Responses.
You should be able to use HttpWebRequest to manually read bytes from the network stream into a buffer. After appending to the buffer, you will want to check if it contains a complete message (<message> ... </ message>). If so, convert the message to a string and announce it as required: eg raise an event, start a task, call a method, add to a queue. Then clear the buffer and repeat.
TcpClient is probably not the best approach in this scenario, because you may then also need to implement TLS (if your endpoint is HTTPS).

According MSDN, you don't have to create a new HttpClient per each request
HttpClient is intended to be instantiated once and re-used throughout
the life of an application. Instantiating an HttpClient class for
every request will exhaust the number of sockets available under heavy
loads.
Next, you can specify the default timeout for httpClient or use it per request.
Then, you are using ResponseHeadersRead option, it means
The operation should complete as soon as a response is available and
headers are read. The content is not read yet.
Try to switch to ResponseContentRead instead

Socket exception in sending HTTP request without Connection header using sockets

When I use the following code to send HTTP GET request using sockets in C#
IPEndPoint RHost = new IPEndPoint(IPAddress.Parse("xxx.xxx.xxx.xxx"), 80);
Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
socket.Connect(RHost);
String req = "GET / HTTP/1.1\r\nHost: awebsite.com\r\n\r\n";
socket.Send(Encoding.ASCII.GetBytes(req), SocketFlags.None);
int bytes = 0;
byte[] buffer = new byte[256];
var result = new StringBuilder();
do
{
bytes = socket.Receive(buffer, 0, buffer.Length, SocketFlags.None);
result.Append(Encoding.ASCII.GetString(buffer, 0, bytes));
}
while (bytes > 0);
I get
System.Net.Sockets.SocketException: 'An existing connection was
forcibly closed by the remote host'
and when I add Connection: Close header to the request, it is working without any problem.
But using Repeater tool in the Burp Suite I am able to send and receive a response from the server without setting Connection: Close header.
Notes:
I have to use socket.
I am facing this problem just with few websites, not every website

You are sending a HTTP/1.1 request. Without an explicit Connection: close there is an implicit Connection: keep-alive with HTTP/1.1 (different to HTTP/1.0). This means that the server might wait for new requests within the same TCP connection after the response is done and close the connection sometimes later if no new requests arrive. This later close might be done with a RST which results in the error you see.
But, your code expects the server to behave differently: it expects that the server closes the connection once the request is done and not to wait for more requests and not to close the idle connection with RST.
To fix this you have either to adapt your request so that the server behavior matches your expectations or to adjust the expectations.
The first can be done by either explicitly adding Connection: close header or by simply using HTTP/1.0 instead of HTTP/1.1. The latter one is recommended because this also results in simpler responses (no chunked response).
If you instead want to adjust your expectations you have to properly parse the servers response: first read the header, then check for Transfer-Encoding: chunked or Content-length and then read the response body based on what these headers say. For details see the HTTP standard.

HTTP range header ignored by server for webpages

I'm trying to download first bytes of a webpage.
I add Range to the HTTP request header. it's my code in C#:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://example.com");
request.AddRange(0,1000);
//request.Proxy = null;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
Stream st = response.GetResponseStream();
StreamReader sr = new StreamReader(st);
string str = sr.ReadToEnd();
sr.Close();
st.Close();
}
for some webpages it works fine, but some servers will ignore the HTTP range header, so the server send all the page in the response.
I changed my code to this:
string str = sr.Read(buffer,0,999);
but it doesn't work! because the problem is not in this line. actually the response will send to my program when I call request.GetResponse()! in this line all the bytes received by the program and write the all bytes to the RAM. I want to stop receiving data from the server when I received first 1000 bytes.
but there is no control on HttpWebRequest class and .GetResponse() method to stop receiving data after 1000 bytes received.
How can I do that?
I think there would be another HTTP Request custom class that allow us to stop receiving data when we want.
please tell me any Idea about this problem. I'm thinking to override the HttpWebRequest or write a MFC Library (C++ language) and import it into my project, but I don't know how to do this.
EDIT: I know it's optional for server to allow or ignore the Range header! but my question is how can I stop receiving data from the server! for example the server is sending 10,000 bytes to my computer, I want to stop receiving bytes after I see the 1000th byte! (I don't want the server to send just first 1000 bytes, I want to close connection and stop receiving the bytes after first 1000 bytes! even if the server send all 10,000 bytes)

Decoding a response c#

I'm developing some API for testing, and I have a problem when I make a webrequest and especially when i retrieve the webresponse.
I use this code:
string request = HttpPost("http://iunlocker.net/check_imei.php", "ime_i=013270000134001");
public static string HttpPost(string URI, string Parameters)
{
try
{
System.Net.WebRequest req = System.Net.WebRequest.Create(URI);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(Parameters);
req.ContentLength = bytes.Length;
System.IO.Stream os = req.GetRequestStream();
os.Write(bytes, 0, bytes.Length);
os.Close();
System.Net.WebResponse resp= req.GetResponse();
if (resp == null) return null;
System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream());
return sr.ReadToEnd().Trim();
}
catch (Exception ex) { }
return null;
}
The website in the call is an example, because with this and with other websites I can't retrieve the result correctly. I receive an exception "Error 403"
Can anyone maybe help me by telling what I may be doing wrong?
I thought the problem was on encoding/decoding -- in fact using Fiddler it asks me if I want to decode before see the text -- but with another website, used for examples, I receive the same message from Fiddler but I can retrieve the response without a problem.
Thanks in advance.

HTTP 403 error means "access forbidden". The destination website is refusing to fulfill your request, for reasons of its own.
Given this particular website http://iunlocker.net/, I'm going to hazard a guess that it may be checking the HTTP_REFERER. In other words it's refusing to fulfill your request because it knows it didn't come from a browser that was viewing the form.
[EDIT] After viewing the response from
curl --form ime_i=013270000134001 -i http://iunlocker.net/check_imei.php
I can see that the immediate response is setting a cookie and a redirect.
HTTP/1.1 307 Temporary Redirect
Server: nginx
Date: Wed, 03 Jul 2013 04:00:27 GMT
Content-Type: text/html
Content-Length: 180
Connection: keep-alive
Set-Cookie: PMBC=35e9e4cd3a7f9d50e7f3bb39d43750d1; path=/
Location: http://iunlocker.net/check_imei.php?pmtry=1
<html>
<head><title>307 Temporary Redirect</title></head>
<body bgcolor="white">
<center><h1>307 Temporary Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
This site does not want you scraping it; if you wish to defeat this you will have to make use of its cookies.

http://en.wikipedia.org/wiki/HTTP_403 - The web server is denying you access to that URL.
Perhaps the IP address you are using, is not allowed to access that resource. Check web server.

HttpWebRequest, BeginGetResponse not called until endOfStream

I am working on a client that uses a webservice to get some events pushed its way - the webservice is designed so, that upon the client POST'ing a subscribe command, it will send back some events of interest and keep doing so as long as the client stay connected.
When POSTing the command, the service responds (immediately) with an initial answer with these headers
Keep-Alive: timeout=5, max=98
Connection: Keep-Alive
Transfer-Encoding: chunked
and then keeps the connection open until it times out (after 30s, if the client does not send some keep-alive data)
Since it is a mix of POST + having to read the response + keeping the connection open until endOFStream, it appears I have to use HttpWebRequest with BeginGetRequestStream (to POST) and BeginGetResponse to read and act on the response.
My problem is that the BeginGetResponse callback is not called until the input stream is actually closed by the server/service (after 30s), despite AllowReadStreamBuffering being set to false.
The doc have this to say on AllowReadStreamBuffering:
The AllowReadStreamBuffering property affects when the callback from BeginGetResponse method is called. When the AllowReadStreamBuffering property is true, the callback is raised once the entire stream has been downloaded into memory. When the AllowReadStreamBuffering property is false, the callback is raised as soon as the stream is available for reading which may be before all data has arrived.
I've seen a few suggestions that no matter what AllowReadStreamBuffering is set to, HttpWebRequest will not call BeginGetResponse until it's buffer is filled up - but I have not been able to find anything on that in the docs.
Does any one have an idea on how to control this buffering behaviour or maybe suggestion to another approach I should try when dealing with this kind of webservice?
The relevant snippets of the code I currently use, look like this:
public void open()
{
string url = "http://funplaceontheinternet/webservice";
HttpWebRequest request = WebRequest.CreateHttp(url);
request.Method = "POST";
request.Credentials = new NetworkCredential("username", "password");
request.CookieContainer = new CookieContainer();
request.AllowReadStreamBuffering = false;
request.BeginGetRequestStream(new AsyncCallback(GetRequestStreamCallback), request);
}
void GetRequestStreamCallback(IAsyncResult result)
{
Debug.WriteLine("open.GetRequestStreamCallback");
HttpWebRequest webRequest = (HttpWebRequest)result.AsyncState;
// End the stream request operation
Stream postStream = webRequest.EndGetRequestStream(result);
// Create the post data
byte[] byteArray = Encoding.UTF8.GetBytes(_xmlEncodedSubscribeCommand);
// Add the post data to the web request
postStream.Write(byteArray, 0, byteArray.Length);
postStream.Close();
// Start the web request
webRequest.BeginGetResponse(new AsyncCallback(BeginGetResponseCallback), webRequest);
}
void BeginGetResponseCallback(IAsyncResult result)
{
HttpWebRequest request = (HttpWebRequest)result.AsyncState;
HttpWebResponse response = null;
if (request != null)
response = (HttpWebResponse)request.EndGetResponse(result);
else
Debug.WriteLine("request==null :-(");
if (response != null)
{
using (var reader = new StreamReader(response.GetResponseStream()))
{
while (!reader.EndOfStream)
{
string line = reader.ReadLine();
Debug.WriteLine("BeginGetResponseCallback - received: " + line);
}
Debug.WriteLine("BeginGetResponseCallback - reader.EndOfStream");
}
}
else
Debug.WriteLine("response==null :-(");
}

You've mentioned that the service is a web service, but not which platform.
If this is a "normal" web service, then I assume that XML is the transport format.
If so, I suspect the problem may be that this style of communication does not really lend itself to streaming. The web service infrastructure at the server end might not be creating the SOAP envelope and payload until all the data is available. If you wanted to stream like this, you might be better using some custom service at the server end, rather than a web service.
Do you know for sure that the server is really streaming the response? (e.g confirmed with something like wireshark?)
If you really want to use a web service, then I would suggest you complete the request when the first event(s) are available, and don't wait for the timeout. This will still achieve the latency reduction that I assume you are trying to get.