Develop Reference

How to find the content of a URL in c#?

I have a URL. Now I want to find out the content of the URL. By content of the URL I mean whether the URL contains a html page, video or an image/photo. How can I do this in asp.net with c#.

The easiest way would be to do a HEAD request with HttpWebRequest:
var req = (HttpWebRequest)WebRequest.Create(url);
req.Method = "HEAD";
using (var response = (HttpWebResponse)req.GetResponse())
{
// Here, examine the response headers.
// In particular response.ContentType
}
In some cases, HEAD might give you a 405 error, meaning that the server doesn't support HEAD.
In that case, just do a GET request (change req.Method = "GET"). That will start to download the page, but you can still view the content type header.

Probably start off using a WebClient and visit/download the page. Then use an HTML parser, and whatever method you deem best, to determine what kind of content is on the page.

Except from following the link, fetching the result and figuring out from the file content what file it is (which is rather tricky), there is no fool proof way.
You can try to determine from the file extension or the returned content-type header (you can issue a HEAD request) what the type should be. This will tell you what the server claims the file type to be.

For easier testing, this is a console application, but it should work with ASP.NET all the same:
namespace ConsoleApplication1
{
using System;
using System.Net;
class Program
{
static void Main()
{
//var request = WebRequest.Create("https://www.google.com"); // page will result in html/text
var request = WebRequest.Create(#"https://www.google.de/logos/2013/douglas_adams_61st_birthday-1062005.2-res.png");
request.Method = "HEAD"; // only request header information, don't download the whole file
var response = request.GetResponse();
Console.WriteLine(response.ContentType);
Console.WriteLine("Done.");
Console.ReadLine();
}
}
}

in C#, how can I get the HTML content of a website before displaying it?

I have a web browser project in C#, I am thinking such system; when user writes the url then clicks "go" button, my browser get content of written web site ( it shouldn't visit that page, I mean it shouldn't display anything), then I want look for a specific "keyword" for ex; "violence", if there exists, I can navigate that browser to a local page that has a warning. Shortly, in C#, How can I get content of a web site before visiting?...
Sorry for my english,
Thanks in advance!

System.Net.WebClient:
string url = "http://www.google.com";
System.Net.WebClient wc = new System.Net.WebClient();
string html = wc.DownloadString(url);

You have to use WebRequest and WebResponse to load a site:
example:
string GetPageSource (string url)
{
HttpWebRequest webrequest = (HttpWebRequest)WebRequest.Create(url);
webrequest.Method = "GET";
HttpWebResponse webResponse = (HttpWebResponse)webrequest.GetResponse();
string responseHtml;
using (StreamReader responseStream = new StreamReader(webResponse.GetResponseStream()))
{
responseHtml = responseStream.ReadToEnd().Trim();
}
return responseHtml;
}
After that you can check the responseHtml for some Keywords... for example with RegEx.

You can make an HTTP request (via HttpClient to the site) and parse the results looking for the various keywords. Then you can make the decision whether or not to visibly 'navigate' the user there.
There's an HTTP client sample on Dev Center that may help.

How do you programmatically find a true URL in C# instead of a forwarding link?

Another way to ask this question is this: How do you programmatically expand a TinyURL link to its true location?
What I want to do is to find a way to programmatically take the link:
http://www.baidu.com/link?url=mW91GJqjJ4zBBpC8yDF8xDhiqDSn1JZjFWsHhEoSNd85PkV8Xil7qccoOX3rynaE
(the first link in a search for Jessica Alba using baidu.com) and have it return the actual link:
http://baike.baidu.com/view/270790.htm
That is just one example. What I want to do is not specific to Jessica but for using Baidu.com as part of my group of search engines in my meta search engine project.
Maybe there is a way of using the WebBrowser class but I did not see a member that was the URL.
Maybe there is a way of using WebRequest and WebResponse.

Make a request to tinied URL, and parse out the HTTP response Location header.

Here you go, it's nice and easy!
var WReq = WebRequest.Create("http://www.baidu.com/link?url=mW91GJqjJ4zBBpC8yDF8xDhiqDSn1JZjFWsHhEoSNd85PkV8Xil7qccoOX3rynaE");
WReq.Method = "HEAD"; // Only download the headers, not the page content
var ActualURL = WReq.GetResponse().ResponseUri.ToString();
MessageBox.Show(ActualURL);

string url = "http://www.baidu.com/link?url=mW91GJqjJ4zBBpC8yDF8xDhiqDSn1JZjFWsHhEoSNd85PkV8Xil7qccoOX3rynaE";
var req = (HttpWebRequest)HttpWebRequest.Create(url);
req.AllowAutoRedirect = false; //<--!!!!
var resp = req.GetResponse();
var realUrl = resp.Headers["Location"]; //http://baike.baidu.com/view/270790.htm
PS: The key point here is the req.AllowAutoRedirect = false

how to read the response from a web site?

I have a website url which gives corresponding city names by taking zip code as input parameter. Now I want to know how to read the response from the site.
This is the link I am using http://zipinfo.com/cgi-local/zipsrch.exe?zip=60680

You'll have to Use the HTTPWebRequest object to connect to the site and scrape the information from the response.
Look for html tags or class names that wrap the content you are trying to find, then use either regexes or string functions to get the required data.
Good example here:

try this (you'll need to include System.text and System.net)
WebClient client = new WebClient();
string url = "http://zipinfo.com/cgi-local/zipsrch.exe?zip=60680";
Byte[] requestedHTML;
requestedHTML = client.DownloadData(url);
UTF8Encoding objUTF8 = new UTF8Encoding();
string html = objUTF8.GetString(requestedHTML);
Response.Write(html);

The simplest way it to use the light-weight WebClient classes in System.Net namespace. The following example code will just download the entire response as a string:
using (WebClient wc = new WebClient())
{
string response = wc.DownloadString("http://zipinfo.com/cgi-local/zipsrch.exe?zip=60680");
}
However, if you require more control over the response and request process then you can use the more heavy-weight HttpWebRequest Class. For instance, you may want to deal with different status codes or headers. There's an example of using HttpWebRequest this in the article How to use HttpWebRequest and HttpWebResponse in .NET on CodeProject.

Used the WebClient Class (http://msdn.microsoft.com/en-us/library/system.net.webclient%28v=VS.100%29.aspx) to request the page and get the response as a string.
WebClient wc = new WebClient();
String s = wc.DownloadString(DestinationUrl);
You can search the response for specific HTML using String.IndexOf, SubString, etc, regular expressions, or try something like the HTML Agility Pack (http://htmlagilitypack.codeplex.com/) which was created specifically to help parse HTML.

first of all, you better find a good Web Service for this purpose.
and this is an HttpWebRequest example:
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create("http://zipinfo.com/cgi-local/zipsrch.exe?zip=60680");
httpRequest.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();
Stream dataStream = httpResponse.GetResponseStream();

You need to use HttpWebRequest for receiving content and some tools for parsing html and finding what you need. One of the most popular libs for working with html in c# is HtmlAgilityPack, you can see simple example here: http://www.fairnet.com/post/2010/08/28/Html-screen-scraping-with-HtmlAgilityPack-Library.aspx

you can use a WebClient object, and an easy way to scrape the data is with xpath.

How to read a file from a URI using StreamReader?

I have a file at a URI that I would like to read using StreamReader. Obviously, this causes a problem since File.OpenText does not support URI paths. The file is a txt file with a bunch of html in it. I have multiple web pages that use this same piece of html, so I have put it in a txt file, and am reading it into the page when the page loads (I can get it to work when I put the file on the file system, but need to put it in a document repository online so that a business user can get to it). I am trying to avoid using an iframe. Is there a way to use StreamReader with URI formats? If not, what other options are there using C# to read in the txt file of html? If this is not optimal, can someone suggest a better approach?

Is there a specific requirement to use StreamReader? Unless there is, you can use the WebClient class:
var webClient = new WebClient();
string readHtml = webClient.DownloadString("your_file_path_url");

You could try using the HttpWebRequestClass, or WebClient. Here's the slightly complicated web request example. It's advantage over WebClient is it gives you more control over how the request is made:
HttpWebRequest httpRequest = (HttpWebRequest) WebRequest.Create(lcUrl);
httpRequest.Timeout = 10000; // 10 secs
httpRequest.UserAgent = "Code Sample Web Client";
HttpWebResponse webResponse = (HttpWebResponse) httpRequest.GetResponse();
StreamReader responseStream = new StreamReader(webResponse.GetResponseStream());
string content = responseStream.ReadToEnd();

If you are behind a proxy don't forget to set your credentials:
WebRequest request=WebRequest.Create(url);
request.Timeout=30*60*1000;
request.UseDefaultCredentials=true;
request.Proxy.Credentials=request.Credentials;
WebResponse response=(WebResponse)request.GetResponse();
using (Stream s=response.GetResponseStream())
...