Develop Reference

Redirect url to obtain the direct url

I did an application which parses an html document and then obtains some urls, the problem is the urls only can be downloaded directly from the navigator.
In VB.NET or C#, how I could redirect this url to obtain a direct link for later paste the link to download it in a Download Manager?
dim url as string = "http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033"
I need to say that I'm not much experimented with http things, maybe I'm wrong and the url has anything to redirect or something similar fault, please just say me how can I redirect that kind of urls or If I'm wrong.
UPDATE:
Tried this, but I get the same url without any changes:
Dim url As String = _
"http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033"
Dim request As HttpWebRequest = DirectCast(HttpWebRequest.Create(url), HttpWebRequest)
request.AllowAutoRedirect = True
Dim response As HttpWebResponse
Dim resUri As String
response = request.GetResponse
resUri = response.ResponseUri.AbsoluteUri
MsgBox(resUri)
UPDATE 2:
In the answer from here HttpWebRequest Login data Then Redirect
He says
If the redirect is handled transparently, the _response.ResponseURI
will contain the address it redirected to. If not, you have to read
the redirect header and decide yourself whether or not to request the
new page.
so...if I need to do thatm, how I can do that?
UPDATE 3:
DownloadThemAll plugin for Firefox can obtain the direct urls... as you can see all the urls finishes with an .mp3 file extension, that's what I need

To my knowledge, the url
http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033
IS the direct url, a direct file url does not need to have the filetype in it.
you can download the file using
string url = "http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033"
WebClient wc = new WebClient();
wc.DownloadFile(url, fileName);
you can get the fileName (Madonna-Like A Virgin -www.mrtzcmp3.net.mp3) by using
HttpWebRequest myHttpWebRequest = (HttpWebRequest)HttpWebRequest.Create(url);
string header = myHttpWebResponse.Headers.ToString();
fileName = header.Remove(0, header.IndexOf("filename=")+10);
fileName = fileName.Remove(fileName.IndexOf('"'));
that is untested, but it should work.
edit: I think this does what you want, but I may have misunderstood your question

you can perform a web request using web client to get the content (url) from that url, then you just need to perform the redirect.

Use an HttpWebRequest and use the AllowAutoRedirect=true to get the direct link and download the file.

Can you try to paste the URL to an URl shortener like tinyUrl or BitLy? Maybe there is a shortener Service that provides an API?
The file then will be downloaded at: http://tinyurl.com/phzhxsr
You will never get a direct URL from the site owner because the URL is dynamicaly parsed and the file is send with the retrun datastream, not by downloading a specific URL.

Find extension of file from URL

I want to find what is the extension of file in current URL.
I have used
http://tol.imagingtech.com/eIL/cgi-bin/eIL.pl?ACTION=get_trans_doc&DOC=ALL&CAB_NAME=TOL_AP_2012&WHSE=18&PAYEE=23003655&INV=01235770
The extension of this file is (.pdf) how can i get this. Sometimes it will be (.doc,.txt,.jpeg) so i want the exact one.
Following is the code which i used to retrieve file extension
var extFile = Document.DocumentFilePath.Split('.');
return "Backup document." + extFile[extFile.Length-1].Trim().ToLower();
It works fine for normal local path but it fails to retrieve extension which DocumentFilePath is url.

I think that there is no way to get the file type without actually getting it.
You can get the information in the response header once your request is completed.
Content-Type: image/jpeg
You can do it in C# using WebClient
WebClient client = new WebClient();
var url = "http://tol.imagingtech.com/eIL/cgi-bin/eIL.pl?ACTION=get_trans_doc&DOC=ALL&CAB_NAME=TOL_AP_2012&WHSE=18&PAYEE=23003655&INV=01235770";
string data = client.DownloadString(url);
string contentType = client.Headers["Content-Type"];

Do a HEAD request to the URL and take a look at the Content-Disposition: attachment; filename=FILENAME header if that's being used.

To find the content type, take it from the Http response as follows:
byte[] myDataBuffer = webClient.DownloadData(fileAbsoluteUrl);
string contentType = webClient.ResponseHeaders["Content-Type"];

unshortening urls

I am trying to unshorten urls and have not been able to find code (vb.net/c#) to do this. These are the twitter shortened urls and I guess I could try and access one of the web services available and do a httpwebrequest but would prefer to find some programmatic way of doing this.

You can get it directly from response of the shortened url since it will return a status code MovedPermanently and the location for the real url.(This should work for most of the sites without the need for navigating to the real url)
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://t.co/xqbLEi6s");
req.AllowAutoRedirect = false;
var resp = req.GetResponse();
string realUrl = resp.Headers["Location"];
Other test data: http://goo.gl/zdf2n , http://tinyurl.com/8xc9vca , http://x.co/iEup, http://is.gd/vTOlz6 , http://bit.ly/FUA4YU

There is no magic way to unshorten a URL without asking the service which created the URL (and the way to ask will be different for each service), or more pragmatically, just opening the URL and watching where it redirects to.

Cannot handle redirect from HTTP/HTTPS protocols to other dissimilar ones

Basically, I'm trying to grab an EXE from CNet's Download.com
So i created web parser and so far all is going well.
Here is a sample link pulled directly from their site:
http://dw.com.com/redir?edId=3&siteId=4&oId=3001-20_4-10308491&ontId=20_4&spi=e6323e8d83a8b4374d43d519f1bd6757&lop=txt&tag=idl2&pid=10566981&mfgId=6250549&merId=6250549&pguid=PlvcGQoPjAEAAH5rQL0AAABv&destUrl=ftp%3A%2F%2F202.190.201.108%2Fpub%2Fryl2%2Fclient%2Finstaller-ryl2_v1673.exe
Here is the problem: When you attempt to download, it begins with HTTP, then redirects to an FTP site. I have tried .NET's WebClient and HttpWebRequest Objects, and it looks like Neither can support Redirects.
This Code Fails at GetResponse();
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://dw.com.com/redir");
WebResponse response = req.GetResponse();
Now, I also tried this:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://dw.com.com/redir");
req.AllowAutoRedirect = false;
WebResponse response = req.GetResponse();
string s = new StreamReader(response.GetResponseStream()).ReadToEnd();
And it does not throw the error anymore, however variable s turns out to be an empty string.
I'm at a loss! Can anyone help out?

You can get the value of the "Location" header from the response.headers, and then create a new FtpWebRequest to download that resource.

in your first code snippet you will be redirected to a link using a different protocol (i.e it's no longer Http as in HttpWebRequest) so it fails du to a malformed http response.
In the second part you're no longer redirected and hence you don't receive a FTP response (which is not malform when interpreted as HTTP response).
You need to acquire FTP link,as ferozo wrote you can do this by getting the value of the header "location", and use a FtpWebRequest to access the file

Is there a .NET ready made method to process response body of a HttpListener HttpListenerRequest body?

I'm using HttpListener to provide a web server to an application written in another technology on localhost. The application is using a simple form submission (application/x-www-form-urlencoded) to make its requests to my software. I want to know if there is already a parser written to convert the body of the html request document into a hash table or equivalent.
I find it hard to believe I need to write this myself, given how much .NET already seems to provide.
Thanks in advance,

You mean something like HttpUtility.ParseQueryString that gives you a NameValueCollection? Here's some sample code. You need more error checking and maybe use the request content type to figure out the encoding:
string input = null;
using (StreamReader reader = new StreamReader (listenerRequest.InputStream)) {
input = reader.ReadToEnd ();
}
NameValueCollection coll = HttpUtility.ParseQueryString (input);
If you're using HTTP GET instead of POST:
string input = listenerRequest.Url.QueryString;
NameValueCollection coll = HttpUtility.ParseQueryString (input);

The magic bits that fill out HttpRequest.Form are in System.Web.HttpRequest, but they're not public (Reflector the method "FillInFormCollection" on that class to see). You have to integrate your pipeline with HttpRuntime (basically write a simple ASP.NET host) to take full advantage.

If you want to avoid the dependency on System.Web that is required to use HttpUtility.ParseQueryString, you could use the Uri extension method ParseQueryString found in System.Net.Http.
Make sure to add a reference (if you haven't already) to System.Net.Http in your project.
Note that you have to convert the response body to a valid Uri so that ParseQueryString (in System.Net.Http)works.
string body = "value1=randomvalue1&value2=randomValue2";
// "http://localhost/query?" is added to the string "body" in order to create a valid Uri.
string urlBody = "http://localhost/query?" + body;
NameValueCollection coll = new Uri(urlBody).ParseQueryString();