Omit images from webpage requested through HttpWebRequest - c#

I fetch webpages in order to feed data to my application. However, the pages contain a lot of images which I don't require at all. I only need the text data.
My problem is that the web requests take an unacceptable amount of time. I think the images also are fetch during a web request. Is there any way to eliminate the images and download only the text data?
The following is the code that I am using currently.
var httpWebRequest = HttpWebRequest.Create(url) as HttpWebRequest;
httpWebRequest.Method = "GET";
httpWebRequest.ProtocolVersion = HttpVersion.Version11;
httpWebRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
httpWebRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
httpWebRequest.Proxy = null;
httpWebRequest.KeepAlive = true;
httpWebRequest.Accept = "text/html";
string responseString = null;
var httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse;
using (var responseStream = httpWebResponse.GetResponseStream())
{
using (var streamReader = new StreamReader(responseStream))
{
responseString = streamReader.ReadToEnd();
}
}
Also, any other optimization suggestions are most welcome.

That is incorrect.
HttpWebRequest does not know anything about HTML or images; it just sends raw HTTP requests.
You can use Fiddler to see exactly what's going on.

Related

Get Response from HttpWebRequest on Windows Phone 8

I'm trying to do a WebRequest to a site from a Windows Phone Application.
But is vry important for me to also get the response from the server.
Here is my code:
Uri requestUri = new Uri(string.Format("http://localhost:8099/hello/{0}", metodo));
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(requestUri);
httpWebRequest.ContentType = "application/xml; charset=utf-8";
httpWebRequest.Method = "POST";
using (var stream = await Task.Factory.FromAsync<Stream>(httpWebRequest.BeginGetRequestStream,
httpWebRequest.EndGetRequestStream, null))
{
string xml = "<string xmlns=\"http://schemas.microsoft.com/2003/10/Serialization/\">Ahri</string>";
byte[] xmlAsBytes = Encoding.UTF8.GetBytes(xml);
await stream.WriteAsync(xmlAsBytes, 0, xmlAsBytes.Length);
}
Unfortunatelly, I have no idea of how I could get the response from the server.
Does anyone have an idea?
Thanks in advance.
Thanks to #max I found the solution and wanted to share it above.
Here is how my code looks like:
string xml = "<string xmlns=\"http://schemas.microsoft.com/2003/10/Serialization/\">Claor</string>";
Uri requestUri = new Uri(string.Format("http://localhost:8099/hello/{0}", metodo));
string responseFromServer = "no response";
HttpWebRequest httpWebRequest = HttpWebRequest.Create(requestUri) as HttpWebRequest;
httpWebRequest.ContentType = "application/xml; charset=utf-8";
httpWebRequest.Method = "POST";
using (Stream requestStream = await httpWebRequest.GetRequestStreamAsync())
{
byte[] xmlAsBytes = Encoding.UTF8.GetBytes(xml);
await requestStream.WriteAsync(xmlAsBytes, 0, xmlAsBytes.Length);
}
WebResponse webResponse = await httpWebRequest.GetResponseAsync();
using (var reader = new StreamReader(webResponse.GetResponseStream()))
{
responseFromServer = reader.ReadToEnd();
}
I hope it will help someone in the future.
This is very common question for people new in windows phone app
development. There are several sites which gives tutorials for the
same but I would want to give small answer here.
In windows phone 8 xaml/runtime you can do it by using HttpWebRequest or a WebClient.
Basically WebClient is a wraper around HttpWebRequest.
If you have a small request to make then user HttpWebRequest. It goes like this
HttpWebRequest request = HttpWebRequest.Create(requestURI) as HttpWebRequest;
WebResponse response = await request.GetResponseAsync();
using (var reader = new StreamReader(response.GetResponseStream()))
{
string responseContent = reader.ReadToEnd();
// Do anything with you content. Convert it to xml, json or anything.
}
Although this is a get request and i see that you want to do a post request, you have to modify a few steps to achieve that.
Visit this place for post request.
If you want windows phone tutorials, you can go here. He writes awesome tuts.

rest api is not giving desired results

I am not getting the results that documentation says. I login the Buddy; created application; copy this URL and assign to url string; when I execute the program I am not getting results that are expected (status + Accesstoken) as documentation says. Can anyone please tell me if I am missing something as newbie to http calls. Its running on http requester but not on Poster firefox add-on!
Documentation
http://dev.buddyplatform.com/Home/Docs/Getting%20Started%20-%20REST/HTTP?
Code
string parameters = "{appid:'xxxxxx', appkey: 'xxxxxxx', platform: 'REST Client'}";
private async void SimpleRequest()
{
HttpWebRequest request = null;
HttpWebResponse response = null;
try
{
request = (HttpWebRequest)WebRequest.Create(url);
request.Accept = "application/json";
request.ContentType = "application/json";
request.Method = "POST";
StreamWriter sw = new StreamWriter(await request.GetRequestStreamAsync());
sw.WriteLine(parameters);
sw.Close();
response = (HttpWebResponse) await request.GetResponseAsync();
}
catch (Exception)
{ }
}
Using the HTTP requester add-on on Firefox, I successfully retrieved an access token so their API work.
In C# they provide a line of code to submit your appid and appkey, that might be the problem :
Buddy.Init("yourAppId", "yourAppKey");
My guess is you have to use their .NET SDK!
You can certainly use the REST API from raw REST the way you're doing, though the .NET SDK will handle some of the more complex details of changing service root. I ran your code using my own Buddy credentials and I was able to get JSON containing an Access Token back. You may need to read the response stream back as JSON to retrieve the access token. I used the following code to dump the JSON to the console:
request = (HttpWebRequest)WebRequest.Create(url);
request.Accept = "application/json";
request.ContentType = "application/json";
request.Method = "POST";
StreamWriter sw = new StreamWriter(await request.GetRequestStreamAsync());
sw.WriteLine(parameters);
sw.Close();
response = (HttpWebResponse)await request.GetResponseAsync();
Console.WriteLine(await new StreamReader(response.GetResponseStream()).ReadToEndAsync());
Using Newtonsoft.Json I can parse out my accessToken like this:
Uri url = new Uri("https://api.buddyplatform.com/devices");
request = (HttpWebRequest)WebRequest.Create(url);
request.Accept = "application/json";
request.ContentType = "application/json";
request.Method = "POST";
StreamWriter sw = new StreamWriter(await request.GetRequestStreamAsync());
sw.WriteLine(parameters);
sw.Close();
response = (HttpWebResponse)await request.GetResponseAsync();
var parsed = JsonConvert.DeserializeObject<IDictionary<string,object>>( (await new StreamReader(response.GetResponseStream()).ReadToEndAsync()));
var accessToken = (parsed["result"] as JObject).GetValue("accessToken").ToString();
Console.WriteLine(accessToken);
The 3.0 SDK does all of this for you while exposing the rest of the service through a thin REST wrapper, the migration guide for the 3.0 SDK should help with this.

C# httpwebrequest blank chars in responsestream

I'm trying to read the reponse from a webserver using httpwebrequests in C#.
I use the following code:
UriBuilder urib = new UriBuilder();
urib.Host = "wikipedia.com";
HttpWebRequest req = WebRequest.CreateHttp(urib.Uri);
req.KeepAlive = false;
req.Host = "wikipedia.com/";
req.Method = "GET";
HttpWebResponse response = (HttpWebResponse) req.GetResponse();
byte[] buffer = new byte[response.ContentLength];
System.IO.Stream stream = response.GetResponseStream();
stream.Read(buffer, 0, buffer.Length);
Console.WriteLine(System.Text.Encoding.ASCII.GetString(buffer, 0, buffer.Length));
The code does indeed retrieve the correct amount of data (I compared the contentlength used to create the buffer, with the length of the console output, they're the same.
My problem is that the last 80% or so of the response is blank chars. They're all 0x00.
I tested this with several pages, including wikipedia.com and it just cuts off mid-file for some reason.
Have I misunderstood/misused the way to use webrequests or can anyone spot an error here?
Try to use this method:
public static String GetResponseString(Uri url, CookieContainer cc)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Method = WebRequestMethods.Http.Get;
request.CookieContainer = cc;
request.AutomaticDecompression = DecompressionMethods.GZip;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
String responseString = reader.ReadToEnd();
response.Close();
return responseString;
}
There are a couple of issues with your code:
Your trying to read the entire response in one go using Stream.Read - that's not what it was designed for. This should be used for more optimal reading e.g. 4KB chunks.
Your reading a HTML response as ASCII encoding - are you sure the page doesn't contain any Unicode characters? I would stick to UTF-8 encoding to be on the safe side (or alternatively read the Content-Type header in the response).
When reading characters from a byte stream (which is what your response is essentially) the recommended approach is to use StreamReader. More specifically, if you want to read the entire stream in one go then use StreamReader.ReadToEnd.
Your code could be shortened to:
HttpWebRequest req = WebRequest.CreateHttp(new Uri("http://wikipedia.org"));
req.Method = WebRequestMethods.Http.Get;
using (var response = (HttpWebResponse)req.GetResponse())
using (var reader = new StreamReader(response.GetResponseStream()))
{
Console.WriteLine(reader.ReadToEnd());
}

HttpWebRequest has empty response requesting a search from Bing

I have the following code that sends a HttpWebRequest to Bing. When I request the url below though it returns what appears to be an empty response when it should be returning a list of results.
var response = string.Empty;
var httpWebRequest = WebRequest.Create("http://www.bing.com/search?q=stackoverflow&count=100") as HttpWebRequest;
httpWebRequest.Method = WebRequestMethods.Http.Get;
httpWebRequest.Headers.Add("Accept-Language", "en-US");
httpWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Win32)";
httpWebRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
using (var httpWebResponse = httpWebRequest.GetResponse() as HttpWebResponse)
{
Stream stream = null;
using (stream = httpWebResponse.GetResponseStream())
{
if (httpWebResponse.ContentEncoding.ToLower().Contains("gzip"))
stream = new GZipStream(stream, CompressionMode.Decompress);
else if (httpWebResponse.ContentEncoding.ToLower().Contains("deflate"))
stream = new DeflateStream(stream, CompressionMode.Decompress);
var streamReader = new StreamReader(stream, Encoding.UTF8);
response = streamReader.ReadToEnd();
}
}
Its pretty standard code for requesting and receiving a web page. Any ideas why the response is empty? Thanks in advance.
EDIT I left off a query string parameter in the url. I also had &count=100 which I have now corrected. It seems to work for values of 50 and below but returns nothing when larger. This works ok when in the browser, but not for this web request.
It makes me think the issue is that the response is large and HttpWebResponse is not handling that for me the way I have it set up. Just a guess though.
This works just fine on my machine. Perhaps you are IP banned from Bing?
Your code works fine on my machine.
I suggest you get yourself a copy of Fiddler and examine the actual HTTP sesssion occuring. May be a proxy or firewall thing.

HTTP POST in .NET doesn't work

I've got a problem with creating an HTTP post request in .NET. When I do this request in ruby it does work.
When doing the request in .NET I get following error:
<h1>FOXISAPI call failed</h1><p><b>Progid is:</b> carejobs.carejobs
<p><b>Method is:</b> importvacature/
<p><b>Parameters are:</b>
<p><b> parameters are:</b> vacature.deelnemernr=478
</b><p><b>GetIDsOfNames failed with err code 80020006: Unknown name.
</b>
Does anyone knows how to fix this?
Ruby:
require 'net/http'
url = URI.parse('http://www.carejobs.be/scripts/foxisapi.dll/carejobs.carejobs.importvacature')
post_args = {
'vacature.deelnemernr' => '478',
}
resp, data = Net::HTTP.post_form(url, post_args)
print resp
print data
C#:
Uri address = new Uri(url);
// Create the web request
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
// Set type to POST
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
// Create the data we want to send
StringBuilder data = new StringBuilder();
data.Append("vacature.deelnemernr=" + HttpUtility.UrlEncode("478"));
// Create a byte array of the data we want to send
byte[] byteData = UTF8Encoding.UTF8.GetBytes(data.ToString());
// Set the content length in the request headers
request.ContentLength = byteData.Length;
// Write data
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(byteData, 0, byteData.Length);
}
// Get response
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
// Get the response stream
StreamReader reader = new StreamReader(response.GetResponseStream());
// Console application output
result = reader.ReadToEnd();
}
return result;
Don't you need the ? after the URL in order to do a post with parameters? I think that Ruby hides this behind the scenes.
I found the problem! The url variable in the C# code was "http://www.carejobs.be/scripts/foxisapi.dll/carejobs.carejobs.importvacature/"
It had to be "http://www.carejobs.be/scripts/foxisapi.dll/carejobs.carejobs.importvacature" without the backslash.

Categories

Resources