Develop Reference

C# asp.net Using WebClient, is there a way to get a web page's rendered Html?

Is there a way to get the fully rendered html of a web page using WebClient instead of the page source? I'm trying to scrape some data from the page's html. My current code is like this:
WebClient client = new WebClient();
var result = client.DownloadString("https://somepageoutthere.com/");
//using CsQuery
CQ dom = result;
var someElementHtml = dom["body > main];

WebClient will only return the URL you requested. It will not run any javacript on the page (which runs on the client) so if javascript is changing the page DOM in any way, you will not get that through webclient.
You are better off using some other tools. Look for those that will render the HTML and javascript in the page.

I don't know what you mean by "fully rendered", but if you mean "with all data loaded by ajax calls", the answer is: no, you can't.
The data which is not present in the initial html page is loaded through javascript in the browser, and WebClient has no idea what javascript is, and cannot interpret it, only browsers do.
To get this kind of data, you need to identify these calls (if you don't know the url of the data webservice, you can use tools like Fiddler), simulate/replay them from your application, and then, if successful, get response data, and extract data from it (will be easy if data comes as json, and more tricky if it comes as html)

better use http://html-agility-pack.net
it has all the functionality to scrap web data and having good help on the site

Redirect url to obtain the direct url

I did an application which parses an html document and then obtains some urls, the problem is the urls only can be downloaded directly from the navigator.
In VB.NET or C#, how I could redirect this url to obtain a direct link for later paste the link to download it in a Download Manager?
dim url as string = "http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033"
I need to say that I'm not much experimented with http things, maybe I'm wrong and the url has anything to redirect or something similar fault, please just say me how can I redirect that kind of urls or If I'm wrong.
UPDATE:
Tried this, but I get the same url without any changes:
Dim url As String = _
"http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033"
Dim request As HttpWebRequest = DirectCast(HttpWebRequest.Create(url), HttpWebRequest)
request.AllowAutoRedirect = True
Dim response As HttpWebResponse
Dim resUri As String
response = request.GetResponse
resUri = response.ResponseUri.AbsoluteUri
MsgBox(resUri)
UPDATE 2:
In the answer from here HttpWebRequest Login data Then Redirect
He says
If the redirect is handled transparently, the _response.ResponseURI
will contain the address it redirected to. If not, you have to read
the redirect header and decide yourself whether or not to request the
new page.
so...if I need to do thatm, how I can do that?
UPDATE 3:
DownloadThemAll plugin for Firefox can obtain the direct urls... as you can see all the urls finishes with an .mp3 file extension, that's what I need

To my knowledge, the url
http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033
IS the direct url, a direct file url does not need to have the filetype in it.
you can download the file using
string url = "http://m.mrtzcmp3.net/get.php?singer=Madonna&song=Like%20A%20Virgin%20&size=5242104&ids=687474703a2h2h63733434303876342g766s2g6f652h75323237363831362h617564696h732h3132323564303466333839622g6f7033"
WebClient wc = new WebClient();
wc.DownloadFile(url, fileName);
you can get the fileName (Madonna-Like A Virgin -www.mrtzcmp3.net.mp3) by using
HttpWebRequest myHttpWebRequest = (HttpWebRequest)HttpWebRequest.Create(url);
string header = myHttpWebResponse.Headers.ToString();
fileName = header.Remove(0, header.IndexOf("filename=")+10);
fileName = fileName.Remove(fileName.IndexOf('"'));
that is untested, but it should work.
edit: I think this does what you want, but I may have misunderstood your question

you can perform a web request using web client to get the content (url) from that url, then you just need to perform the redirect.

Use an HttpWebRequest and use the AllowAutoRedirect=true to get the direct link and download the file.

Can you try to paste the URL to an URl shortener like tinyUrl or BitLy? Maybe there is a shortener Service that provides an API?
The file then will be downloaded at: http://tinyurl.com/phzhxsr
You will never get a direct URL from the site owner because the URL is dynamicaly parsed and the file is send with the retrun datastream, not by downloading a specific URL.

c# asp.net MVC Download a file via Ajax

I'm pretty sure this isn't possible but I thought I'd ask...
I have a FileResult which returns a file, when it works there's no problem. When there's an error in the FileResult for some reason, I'd like to display an exception error message on the users screen, preferably in a popup.
Can I do an Ajax post which returns a file when successful and displays a message if not?

I think it is not possible cause in order to handle ajax post, you will have to write a javascript handler on the client side and javascript cannot do file IO on client side.
However, what you can do is, make an ajax request to check if file exists and can be downloaded. If, not, respond to that request negatively which will popup a dialog on client side. If successful, make a file download request.

Not specifically related to MVC but...
it can be done using XMLHttpRequest in conjunction with the new HTML5 File System API that allows you to deal with binary data (fetched from http response in your case) and save it to the local file system.
See example here: http://www.html5rocks.com/en/tutorials/file/xhr2/#toc-example-savingimages

Controller (MyApiController) Code:
public ActionResult DownloadFile(String FileUniqueName)
{
var rootPath = Server.MapPath("~/UploadedFiles");
var fileFullPath = System.IO.Path.Combine(rootPath,FileUniqueName);
byte[] fileBytes = System.IO.File.ReadAllBytes(fileFullPath);
return File(fileBytes, System.Net.Mime.MediaTypeNames.Application.Octet, "MyDownloadFile");
}
Jquery Code:
$(document).on("click"," a ", function(){ //on click of anchor tag
var funame=$(this).attr('uname'); /*anchor tag attribute "uname" contain file unique name*/
var url = "http://localhost:14211/MyApi/DownloadFile?FileUniqueName= " + funame;
window.open(url);
});

Force the Download File dialog Not Working - ASP.NET C#

I am trying to implement force download file dialog in my ASP.NET C# application. The files I'd like to force download are media files not locally available available on the web server but are being served from a different location.
I am getting an error 'http://remote-site-to-webserver/somefile.asf' is not a valid virtual path.
I have searched the web for solutions but all examples point to relative path on the server using Server.MapPath
In the example below I created a webhandler.ashx page and send the download request to this page.
<%# WebHandler Language="C#" Class="DownloadHandler" %>
using System;
using System.Web;
public class DownloadHandler : IHttpHandler {
public void ProcessRequest(HttpContext context) {
var fileName = "http://remote-site-to-webserver/somefile.asf";
var r = context.Response;
r.AddHeader("Content-Disposition", "attachment; filename=" + fileName);
r.WriteFile(context.Server.MapPath(fileName));
}
public bool IsReusable { get { return false; } }
}

The Content-Disposition header looks wrong to me. I think it should be:
r.AddHeader("Content-Disposition",
"attachment; filename=DefaultNewFilename.ext");
the filename is the default name given to the downloaded file... Or in otherwords it's what is shown in the browsers save dialog.
You may also want:
r.AddHeader("Content-Type", "application/octetstream");
I'm not sure that's required.... But I've always included it for video files and so on.

In order for the download to start from a different server, you need to send a redirect answer to the client (Response.Redirect(mediaURL)).
As a consequence, you cannot force the download dialog from your web server because the browser will send a separate request to the other server. This must be solved on the server where the media is served from.
The only alternative is that you act as an intermediate, i.e. you download the media file to your server and send it as the response to the client. This shouldn't be too difficult if it's a small file that easily fits into memory. However, if it's a large file it might involve some tricky coding so you can receive and send it piecewise.

Server.MapPath()
is not used for remote http files. it is just a tool for converting virtual addresses to physical addresses, i.e. you can retrieve "C:\inetpub\wwwroot\MyWebSite\Files\blah.txt" by giving "~/Files/blah.txt" to Server.MapPath method.
if you are interested in downloading a file from another web server you will have to use HttpWebRequest class.
this is a sample code:
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create("http://remote-site-to-webserver/somefile.asf");
httpRequest.Credentials = CredentialCache.DefaultCredentials; //or a NetworkCredential if needed
HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse();
Stream dataStream = httpResponse.GetResponseStream();
now you can output the dataStream into your response.

How is an image served from a URL with an ASPX extension?

Can Somebody tell how to create such kind of urls
for example if you see the url
http://office.microsoft.com/global/images/default.aspx?assetid=ZA103873861033
you will redirect to an image ..
my question is , though this url is an image..its extension is aspx..how is it possible.
how to create such kind of url's
Thanks

This is a common method for displaying an image that's stored as a binary object in a database. One tutorial, among many, can be found here.
Essentially, what they're doing is using the aspx page to accept the URL parameter which tells them what image to fetch from the database. Then in the response they clear all output and headers, set the headers for the image, write the binary data to the response stream, and close the response stream.
So it's not really "redirecting" you to an image. The "page" being requested turns out to be an image resource in the response.

By setting the ContentType in the response from the server
HttpContext.Response.ContentType = "image/jpeg";

easiest way is to add generic handler *.ashx and in ashx file u'll have code behind which u can get querystring and manipulate response eg. Response.WriteFile(...)

File extensions literally have no meaning on the WWW. The thing that correctly describes the content at a particular URL is the content-type/MIME-type. This is delivered in an HTTP header when the URL is requested prior to delivery of the main HTTP payload. Other answers describe how you might correctly set this in ASP.NET.

Aside from all other answers they may be doing a Server.Transfer() (so that you don't see it client-side) to the image file. This still means the response headers are being set to the appropriate MIME type but it also means the image isn't necesarilly coming from a database. This technique can be used to hide the actual image URL in attempts to prevent hotlinking.