Develop Reference

Make httpwebrequest with "burn" CPU

I want to run about 10,000 concurrent requests, using .net HttpWebRequest, not all of them going to the same host and some of them go through a pool of proxies.
I'm currently using Threads, which works fine up to 1000 concurrent, but then my CPU load 100%. How to fix it?
public string HttpGet(string url)
{
try {
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Timeout = 20000;
request.CookieContainer = Cookie;
request.AutomaticDecompression = DecompressionMethods.GZip;
request.KeepAlive = true;
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string tmp = reader.ReadToEnd();
reader.Close();
dataStream.Close();
response.Close();
return tmp;
} catch (Exception ex) {
return "";
}
}
I send request each time, so make 1000 requests for 1 minute.

Use async tasks instead of threads. I/O operations don't need thread tho wait for them, but OS spends time on switching between them.
Use using blocks instead of Close call to properly dispose resources even if request fails.
Don't forget to set correct maximum limit for concurrent requests.
catch (Exception ex) is an antipattern.

How to know if a website/domain is available before loading a webview with that URL

hello I am trying to launch an intent with a webview from a user entered URL, I have been looking everywhere online and I can't find a concrete answer as to how to make sure the website will actually connect before allowing the user to proceed to the next activity. I have found many tools to make sure the URL follows the correct format but none that actually let me make sure it can actually connect.

You can use WebClient and check if any exception is thrown:
using (var client = new HeadOnlyClient())
{
try
{
client.DownloadString("http://google.com");
}
catch (Exception ex)
{
// URL is not accessible.
}
}
You can catch more specific exceptions to make it more elegant.
You can also use custom modification to WebClient to check HEAD only and decrease the amount of data downloaded:
class HeadOnlyClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest req = base.GetWebRequest(address);
req.Method = "HEAD";
return req;
}
}

I would suggest you to use HttpHead for simple request with AndroidHttpClient, but it is deprecated now. You can try to implement HEAD Request by sockets.

You can try to ping the address first.
See this SO question: How to Ping External IP from Java Android

Another option:
Connectivity Plugin for Xamarin and Windows
Task<bool> IsReachable(string host, int msTimeout = 5000);
But, any pre-check that succeeds isn't guaranteed as the very next request might fail so you should still handle that.

Here's what I ended up doing to Check if a Host name is reachable. I was connecting to a site with a self signed certificate so that's why I have the delegate in the ServiceCertificateValidationCallback.
private async Task<bool> CheckHostConnectionAsync (string serverName)
{
string Message = string.Empty;
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(serverName);
ServicePointManager.ServerCertificateValidationCallback += delegate
{
return true;
};
// Set the credentials to the current user account
request.Credentials = System.Net.CredentialCache.DefaultCredentials;
request.Method = "GET";
request.Timeout = 1000 * 40;
try
{
using (HttpWebResponse response = (HttpWebResponse) await request.GetResponseAsync ())
{
// Do nothing; we're only testing to see if we can get the response
}
}
catch (WebException ex)
{
Message += ((Message.Length > 0) ? "\n" : "") + ex.Message;
return false;
}
if (Message.Length == 0)
{
goToMainActivity (serverName);
}
return true;
}

Consuming Twitters 1.1 API using OAuth, getting no response and it happens instantly

I am trying to swap my website over to consuming the new Twitter 1.1 API with uses OAuth 1.0a. I am able to get the correct response using a REST client and I am now trying to duplicate that on my website using c#.
I have constructed my headers the appropriate way and I have verified that they are in the correct format for what Twitter is looking for.
The issue I am having is that I do not think I am actually sending the request. I say this because my application returns almost instantly. The request should take a second or so to send at least, and my response has totally empty, with no 401 or 400 status code.
Below I have the code that actually sends the request. I am actually sending the request and if so why am I not getting any status code or anything.
Thanks in advance for the help.
//string url = "https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=MYSCREENNAME&count=2";
string url = "https://api.twitter.com/1.1/statuses/user_timeline.json";
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.Method = "GET";
webRequest.Headers.Add("Authorization", authorizationHeaderParams);
try {
var response = webRequest.GetResponse() as HttpWebResponse;
if (response != null && response.StatusCode != HttpStatusCode.OK) {
lblresponse.InnerText = "The request did not complete and returned status code: {0} " + response.StatusCode;
}
if (response != null) {
var reader = new StreamReader(response.GetResponseStream());
reader.ReadToEnd();
lblresponse.InnerText += "success";
}
} catch {
lblresponse.InnerText += "fail";
}
So yeah this code goes straight to the catch block. My thoughts are I am not actually sending the request, since it takes no time to happen. I know there are some libraries designed to make this easier but I would much rather learn how to do it myself (with the help of you guys).
Thanks.

The request is going to throw an exception in the case of a 400 or 401. So catch System.Web.Exception in the catch block to see if there's a 400 or 401.
catch(System.Web.Exception ex) {
var errorReponse = (HttpWebResponse)ex.Response;
var statusCode = errorReponse.StatusCode;
lblresponse.InnerText += "fail";
}

System.Net.WebException

is seems to me that WebRequest.Create(url) fails a litle to soon..
Explanations:
on a url that it failed (or throwed a System.Net.WebException) ... copy-paste -ing that url into the browser seems to work..browser gives response!.. sometimes with a remarcable delay (~10 seconds).. but WebRequest in less than 3 seconds throws exception
Example of valid url on which it fails:
http://tracker.podtropolis.com:2710/announce?info_hash=%92%FD%2F%0B%40%F64%C5%86%19%D6%3E%B1%28%B2%81%A1J%D4%F6&peer_id=-AZ2060-%AD%18%05o%11%A6%26%B3%C3%D16%AC&port=6881&downloaded=0&uploaded=0&left=647749313&numwant=30&compact=1&event=started
if it has any meaning firefox reacts 10 times more slowly on this url that explorer, also sometimes firefox crashes on loading such a url
So question Why is WebRequest failing so soon?? I would like it to try a litle harder to get response from URL...
And this is the method that catches exeption (here i check if url is valid OR ~ "is tracker alive??")
public static bool isURLValid(string url)
{
try
{
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "HEAD";
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
//Returns TURE if the Status code == 200
return (response.StatusCode == HttpStatusCode.OK);
}
catch (Exception ex) //(WebException ex)
{
Logger.e(TAG, "isURLValid", ex);
return false; //Any exception will returns false.
}
}
if valid i get stream (i know .. I know ..double contact.. but still)
inputStream = WebRequest.Create(fullURL).GetResponse().GetResponseStream();
So.. thoughts?

How do you check if a website is online in C#?

I want my program in C# to check if a website is online prior to executing, how would I make my program ping the website and check for a response in C#?

A Ping only tells you the port is active, it does not tell you if it's really a web service there.
My suggestion is to perform a HTTP HEAD request against the URL
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("your url");
request.AllowAutoRedirect = false; // find out if this site is up and don't follow a redirector
request.Method = "HEAD";
try {
response = request.GetResponse();
// do something with response.Headers to find out information about the request
} catch (WebException wex)
{
//set flag if there was a timeout or some other issues
}
This will not actually fetch the HTML page, but it will help you find out the minimum of what you need to know. Sorry if the code doesn't compile, this is just off the top of my head.

You have use System.Net.NetworkInformation.Ping see below.
var ping = new System.Net.NetworkInformation.Ping();
var result = ping.Send("www.google.com");
if (result.Status != System.Net.NetworkInformation.IPStatus.Success)
return;

Small remark for Digicoder's code and complete example of Ping method:
private bool Ping(string url)
{
try
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
request.Timeout = 3000;
request.AllowAutoRedirect = false; // find out if this site is up and don't follow a redirector
request.Method = "HEAD";
using (var response = request.GetResponse())
{
return true;
}
}
catch
{
return false;
}
}

if (!NetworkInterface.GetIsNetworkAvailable())
{
// Network does not available.
return;
}
Uri uri = new Uri("http://stackoverflow.com/any-uri");
Ping ping = new Ping();
PingReply pingReply = ping.Send(uri.Host);
if (pingReply.Status != IPStatus.Success)
{
// Website does not available.
return;
}

The simplest way I can think of is something like:
WebClient webClient = new WebClient();
byte[] result = webClient.DownloadData("http://site.com/x.html");
DownloadData will throw an exception if the website is not online.
There is probably a similar way to just ping the site, but it's unlikely that the difference will be noticeable unless you are checking many times a second.