How can I check whether a page exists at a given URL?
I have this code:
private void check(string path)
{
try
{
Uri uri = new Uri(path);
WebRequest request = WebRequest.Create(uri);
request.Timeout = 3000;
WebResponse response;
response = request.GetResponse();
}
catch(Exception loi) { MessageBox.Show(loi.Message); }
}
But that gives an error message about the proxy. :(
First, you need to understand that your question is at least twofold,
you must first check if the server is responsive, using ping for example - that's the first check, while doing this, consider timeout, for which timeout you will consider a page as not existing?
second, try retrieving the page using many methods which are available on google, again, you need to consider the timeout, if the server taking long to replay, the page might still "be there" but the server is just under tons of pressure.
If the proxy needs to authenticate you with your Windows credentials (e.g. you are in a corporate network) use:
WebRequest request=WebRequest.Create(url);
request.UseDefaultCredentials=true;
request.Proxy.Credentials=request.Credentials;
try
{
Uri uri = new Uri(path);
HttpWebRequest request = HttpWebRequest.Create(uri);
request.Timeout = 3000;
HttpWebResponse response;
response = request.GetResponse();
if (response.StatusCode.Equals(200))
{
// great - something is there
}
}
catch (Exception loi)
{
MessageBox.Show(loi.Message);
}
You can check the content-type and length, see MSDN HTTPWebResponse.
At a guess, without knowing the specific error message or path, you could try casting the WebRequest to a HttpWebRequest and then setting the WebProxy.
See MSDN: HttpWebRequest - Proxy Property
Related
I'm getting an error 403 while I try to do anything to an image's Url (be it get the file size or download it) but I don't get any error while trying to show the image.
I hope I'm clear enough, but if need be this is an example of url posing problem:
Image URL / Site show the image
I'm using this code to get the file size which works great but not on this site for exemple :
public void getFileSize(string uri)
{
try
{
waitGetSize = 0;
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
req.Timeout = 5000;
req.Method = "HEAD";
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
imgSize = resp.ContentLength;
imgSizeKb = imgSize / 1024;
waitGetSize = 1;
}
catch (Exception ex)
{
MetroMessageBox.Show(this, ex.Message, "Exception :", MessageBoxButtons.OK, MessageBoxIcon.Error);
}
}
As pointed out by cFrozenDeath, I used a HEAD request, so I tried using a GET request to the exact same effect. Same result by simply not stating the request type I want.
So is there a way to get the file size or at least download the file knowing it's shown OK when opened in a browser?
You have to mimic a webbrowser when you want to scrape content from websites.
Sometimes this means you need to provide and/or keep the Cookies you get when you land initially on a website, sometimes you have to tell the webserver which page linked to the resource.
In this case you need to provide the Referer in the header:
public void getFileSize(string uri)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(uri);
// which page do we want that server to believe we call this from
req.Referer = "http://www.webtoons.com/";
req.Timeout = 5000;
req.Method = "GET"; // or do a HEAD
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
// rest omitted
}
That particular image has a length of 273073 bytes.
Do note that scraping content might be against the terms of service of the particular website. Make sure you don't end up doing illegal stuff.
hello I am trying to launch an intent with a webview from a user entered URL, I have been looking everywhere online and I can't find a concrete answer as to how to make sure the website will actually connect before allowing the user to proceed to the next activity. I have found many tools to make sure the URL follows the correct format but none that actually let me make sure it can actually connect.
You can use WebClient and check if any exception is thrown:
using (var client = new HeadOnlyClient())
{
try
{
client.DownloadString("http://google.com");
}
catch (Exception ex)
{
// URL is not accessible.
}
}
You can catch more specific exceptions to make it more elegant.
You can also use custom modification to WebClient to check HEAD only and decrease the amount of data downloaded:
class HeadOnlyClient : WebClient
{
protected override WebRequest GetWebRequest(Uri address)
{
WebRequest req = base.GetWebRequest(address);
req.Method = "HEAD";
return req;
}
}
I would suggest you to use HttpHead for simple request with AndroidHttpClient, but it is deprecated now. You can try to implement HEAD Request by sockets.
You can try to ping the address first.
See this SO question: How to Ping External IP from Java Android
Another option:
Connectivity Plugin for Xamarin and Windows
Task<bool> IsReachable(string host, int msTimeout = 5000);
But, any pre-check that succeeds isn't guaranteed as the very next request might fail so you should still handle that.
Here's what I ended up doing to Check if a Host name is reachable. I was connecting to a site with a self signed certificate so that's why I have the delegate in the ServiceCertificateValidationCallback.
private async Task<bool> CheckHostConnectionAsync (string serverName)
{
string Message = string.Empty;
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(serverName);
ServicePointManager.ServerCertificateValidationCallback += delegate
{
return true;
};
// Set the credentials to the current user account
request.Credentials = System.Net.CredentialCache.DefaultCredentials;
request.Method = "GET";
request.Timeout = 1000 * 40;
try
{
using (HttpWebResponse response = (HttpWebResponse) await request.GetResponseAsync ())
{
// Do nothing; we're only testing to see if we can get the response
}
}
catch (WebException ex)
{
Message += ((Message.Length > 0) ? "\n" : "") + ex.Message;
return false;
}
if (Message.Length == 0)
{
goToMainActivity (serverName);
}
return true;
}
I have tested a few plugins for Firefox and Chrome which can identify IP number of a given website ofcause. But some of them can also show what server-side technology the website runs on.
How do they do this? I know about client-user-agent, is there something similar in the HTTP protocol where the server sendes a "server-host-agent" kinda string?
And if so, how would the code for retreiving this look. I guess its something with WebClient?
Anyone?
Using the HttpWebRequest and setting the Method property to HEAD, you can do a HTTP HEAD request, which is very lightweight. It will return the HTTP Headers (which may or may not be correct). They HTTP Headers may also differ from server to server as there is no standard for what headers a server should expose.
The code:
HttpWebRequest myReq = (HttpWebRequest)WebRequest.Create("http://www.contoso.com/");
myReq.Method = "HEAD";
WebResponse myRes = myReq.GetResponse();
for(int i=0; i < myHttpWebResponse.Headers.Count; ++i) {
Console.WriteLine(
"\nHeader Name:{0}, Value :{1}",
myHttpWebResponse.Headers.Keys[i], myHttpWebResponse.Headers[i]
);
}
EDIT:
var request = (HttpWebRequest)WebRequest.Create("http://www.http500.com");
try
{
var response = request.GetResponse();
}
catch (WebException wex)
{
// Safe cast to HttpWebResponse using 'as', will return null if unsuccessful
var httpWebResponse = wex.Response as HttpWebResponse;
if(httpWebResponse != null)
{
var httpStatusCode = httpWebResponse.StatusCode;
// HttpStatusCode is an enum, cast it to int for its actual value
var httpStatusCodeInt = (int)httpWebResponse.StatusCode;
}
}
I'm trying to make a simple app that will "ping" a uri and tell me if its responding or not
I have the following code but it only seems to check domains at the root level
ie www.google.com and not www.google.com/voice
private bool WebsiteUp(string path)
{
bool status = false;
try
{
Uri uri = new Uri(path);
WebRequest request = WebRequest.Create(uri);
request.Timeout = 3000;
WebResponse response;
response = request.GetResponse();
if (response.Headers != null)
{
status = true;
}
}
catch (Exception loi)
{
return false;
}
return status;
}
Is there any existing code out there that would better fit this need?
Edit: Actually, I tell a lie - by default 404 should cause a web exception anyway, and I've just confirmed this in case I was misremembering. While the code given in the example is leaky, it should still work. Puzzling, but I'll leave this answer here for the better safety with the response object.
The problem with the code you have, is that while it is indeed checking the precise URI given, it considers 404, 500, 200 etc. as equally "successful". It also is a bit wasteful in using GET to do a job HEAD suffices for. It should really clean up that WebResponse too. And the term path is a silly parameter name for something that isn't just a path, while we're at it.
private bool WebsiteUp(string uri)
{
try
{
WebRequest request = WebRequest.Create(uri);
request.Timeout = 3000;
request.Method = "HEAD";
using(WebResponse response = request.GetResponse())
{
HttpWebResponse hRes = response as HttpWebResponse;
if(hRes == null)
throw new ArgumentException("Not an HTTP or HTTPS request"); // you may want to have this specifically handle e.g. FTP, but I'm just throwing an exception for now.
return hRes.StatusCode / 100 == 2;
}
}
catch (WebException)
{
return false;
}
}
Of course there are poor websites out there that return a 200 all the time and so on, but this is the best one can do. It assumes that in the case of a redirect you care about the ultimate target of the redirect (do you finally end up on a successful page or an error page), but if you care about the specific URI you could turn off automatic redirect following, and consider 3xx codes successful too.
There is a Ping class you can utilize for that, more details can be found here:
http://msdn.microsoft.com/en-us/library/system.net.networkinformation.ping.aspx
I did something similar when I wrote a torrent client to check valid tracker URLS, pretty sure I found the answer on SO but cant seem to find it anymore, heres the code sample I have from that post.
using(var client = new WebClient()) {
client.HeadOnly = true;
// exists
string Address1 = client.DownloadString("http://google.com");
// doesnt exist - 404 error
string Address2 = client.DownloadString("http://google.com/sdfsddsf");
}
I am making a Http Webrequest to an available site that I can visit fine, but the HTTP Web request keeps timing out. Is there any reason why this code might allow it to timeout when it shouldn't?
I've tried upping the timeout setting, but it still continues to timeout.
Uri CameraUrl = new Uri("http://" + cfg_cameraIps[i]);
HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(CameraUrl);
myRequest.Timeout = 5000;
myRequest.Method = "HEAD";
try
{
HttpWebResponse webresponse;
webresponse = (HttpWebResponse)myRequest.GetResponse();
if (webresponse.StatusCode.ToString() == "OK")
{
continue;
}
You're not closing your web response - if you find that the first couple of requests work, but ones after that don't, then that's the problem. It's trying to reuse the existing connection to the server, but it can't because you haven't closed the response.
Change your code to:
using (HttpWebResponse webresponse = (HttpWebResponse) myRequest.GetResponse())
{
if (webresponse.StatusCode == HttpStatusCode.OK)
{
continue;
}
...
}
and see if that helps.
If it's failing on the very first request to the server, then that's something different. In that case, use Wireshark to see what's going on at the network level.
Note that in the code above I've also removed the string conversion in favour of comparing the status codes directly.