Develop Reference

How do I recover the full html of a page, including what is generated by javascript

How do I recover the full html of a page, including what is generated by javascript. The problem is that I want to access the contents of the select tag, but the page but it is coming empty, this probably being generated dynamically. Please I'm about to give up!
I just posted a piece of code because this very large, if I find it necessary to put the whole code.
res = (HttpWebResponse)req.GetResponse();
res.Cookies = req.CookieContainer.GetCookies(req.RequestUri);
cookieContainer.Add(res.Cookies);
sr = new StreamReader(res.GetResponseStream());
getHtml = sr.ReadToEnd();
viewstate = rxViewstate.Match(getHtml).Groups[1].Value;
EventValdidation = rxEventValidation.Match(getHtml).Groups[1].Value;
viewstate = HttpUtility.UrlEncode(viewstate);
EventValdidation = HttpUtility.UrlEncode(EventValdidation);
//Here I should take the contents of the select tag.
getHtml = rxDropDownMenu.Match(getHtml).Groups[2].Value;

You can't just do this with HttpWebRequest, all that does is download the raw HTML and non of the linked JavaScript files.
It also wouldn't run the JavaScript or give you any kind of DOM to inspect.
You'd really need to use WebBrowser or perhaps something like Awesomium.

Compile error CS1647 when opening large aspx file in browser

I am using XML/XSLT to create an aspx page which can grow quite large. Even though the file is successfully created, when the file size approaches 300K, the error: “CS1647: An expression is too long or complex to compile” is issued when opening the file in a browser (tried both IE and Chrome). No other useful information is shown on the failed .NET error page.
My development environment is VS 2012 Express for Web on a Win7 x64 laptop.
Since this problem does not occur during the execution of the program, I am at a loss as to how to approach solving this problem. Can anyone suggest a strategy or work around to this issue?
EDIT
The C# code used to create the aspx page is
// load the xml file
XmlDocument reportDetails = new XmlDocument();
reportDetails.Load(ReportDetailsPath);
//setup the xslt transform
XslCompiledTransform xslt = new XslCompiledTransform();
xslt.Load(XlsRptPath);
StringWriter sw = new StringWriter();
xslt.Transform(ReportDetails, null, sw);
//create the aspx file
using (StreamWriter outfile = new StreamWriter(aspxPath))
{
outfile.Write(sw.ToString());
}

Old question, and largely answered in the comments. But for the sake of completeness, let's answer it ;). Your issue is not with XSLT itself, but with the generated file which hits the 300k boundary of maximum expression size in C#. There are a couple of things you can do:
Make the generated code below 300k (but this might not be possible in your case)
Take any string constants in the generated code and place them in a
resource file. The format of such resource files is XML and you can
auto-generate it, just make sure the IDs in the resource match the
IDs you use in the generated ASPX
Place part of the generated code in a code-behind file. An ASPX is
parsed as one expression, but the code-behind is not. This is a
matter of structuring the generated code.
Split the ASPX in multiple pages, if your design allows this. You can
recombine them with iframes.
Split the ASPX file in multiple ASCX controls. This is perhaps the
most natural thing to do. Each ASCX control can be referenced/added
to the ASPX file. Each ASCX control should not exceed the 300k limit.
If there is a lot of generated CSS that blows up the size, place it
in a separate CSS file.
If there are a lot of long absolute paths in image references and the
like, you can collapse them and make the references relative, i.e. by
using <base>, which may save you some space.
If the error is caused by actual large (constant) expressions, consider the
hints in the answers in this post for a resolution.

I am sorry for not posting my solution sooner, but I was too stressed at the time to do so. Better late than never I guess.
Instead of trying to create a complete aspx web page for each associated xml file, I created a stub and applied the xslt transform at run-time from within an associated Site.Master. The stub's MasterPageFile property is set to this Site.Master. This approach does sacrifice some performance, but it works for any size web page. Here is an example of the outputted webpage.
Example aspx stub file:
<%# Page Title="Top Austin Beauty Salons List" MetaDescription="List of best Google-ranked Austin beauty salon" Language="C#" MasterPageFile="~/Site1.Master" %>
<asp:Content ID="Content2" ContentPlaceHolderID="ContentPlaceHolder1" runat="server">
</asp:Content>
Site.Master Page_Load:
protected void Page_Load(object sender, EventArgs e)
{
string vp = Page.AppRelativeVirtualPath;
if (vp.ToLower().EndsWith("default.aspx") || vp.ToLower().EndsWith("webform2.aspx")) return; // ignore some aspx files used for ohter reasons
string xmlPath = Page.MapPath(vp.Substring(0, vp.LastIndexOf(".")) + #".xml");
string xslPath = Page.MapPath("mainpage.xslt");
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(xmlPath);
XsltArgumentList argsList = new XsltArgumentList();
argsList.AddParam("xmlPath", "", xmlPath);
XslCompiledTransform xslt = new XslCompiledTransform();
xslt.Load(xslPath);
// Execute the transform and output the results to a string writer.
StringWriter sw = new StringWriter();
xslt.Transform(xmlDoc, argsList, sw);
content.InnerHtml = sw.ToString(); // add the generated html to the associated stub aspx content section
}

How to render XML file using CodeBehind C#

I wanted to know how can I get request to XML file (.xml) but to render the response using codebehind... (like doing <% ... %> )
Thanks

Not sure I understand the question, but you seem to ask about something like
<%
Response.Clear();
Response.ContentType="text/xml";
Response.Write("<?xml version=""1.0"" encoding=""UTF-8"">");
Response.Write("<MyFancyRecord>");
Response.Write("<Title>" + someRec.Title + "</Title>");
Response.Write("<Price>" + (someRec.Price * 1.20) + "</Price>");
// etc.
Response.Write("</MyFancyRecord>");
Response.End();
%>
As hinted in other response, while this approach is a direct answer to the OP's question, it is typically better to map non-html requests to handlers.

Requests to ASP.NET applications that return XML or other non-HTML responses are best implemented as handlers (*.ASHX). You can find an example of how to do this here:
http://www.aspcode.net/Creating-an-ASHX-handler-in-ASPNET.aspx

Read ASPX files from filesystem and render to HTML

Is it possible to read an aspx file and render as an html file, and write the resulting html file to disk?
The .aspx file is on the filesystem without the codebehind file. If it is possible, please provide some example code.

from remote url
byte[] buf = new byte[8192];
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(path);
webRequest.KeepAlive = false;
string content = string.Empty;
HttpWebResponse webResponse = (HttpWebResponse)webRequest.GetResponse();
if (!(webResponse.StatusCode == HttpStatusCode.OK))
if (_log.IsErrorEnabled) _log.Error(string.Format("Url {0} not found", path));
Stream resStream = webResponse.GetResponseStream();
int count = 0;
do
{
count = resStream.Read(buf, 0, buf.Length);
if (count != 0)
{
content += encoding.GetString(buf, 0, count);
}
}
while (count > 0);
from network or virtual path
string content = string.Empty;
path = HttpContext.Current.Server.MapPath(path);
if (!File.Exists(path))
if (_log.IsErrorEnabled) _log.Error(string.Format("file {0} not found", path));
StreamReader sr = new StreamReader(path, encoding);
content = sr.ReadToEnd();

You need to use the wwAspRuntimeHost class.
Rick Strahl had a post on this, and I actually used the same approach he recommendsd to host ASP.NET runtime engine in a non-IIS environment. Here's the link:
http://www.west-wind.com/presentations/aspnetruntime/aspnetruntime.asp
(update to the original post)
http://www.west-wind.com/Weblog/posts/1197.aspx

This is what ASP.NET does all the time. It looks for an ASPX page on the file system, compiles it, if required, and then processes the request.
Codebehind is optional. You can have a website with only ASPX in it, without any precompiled code.
Here's a ASPX page without codebehind
<%# Page language="c#" %>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" >
<HTML>
<HEAD>
<title>ClearCache</title>
</HEAD>
<body>
<form id="ClearCache" method="post" runat="server">
<%
IList keys = new ArrayList(Cache.Count);
foreach (DictionaryEntry de in Cache)
keys.Add(de.Key);
foreach (string key in keys)
{
this.Response.Write(key + "<br>");
Cache.Remove(key);
}
%>
</form>
</body>
</HTML>
Downloading the file as html:
var wc = new WebClient();
wc.DownloadFile(myUrl, filename);
If you don't have a ASP.NET web-server, you have to start a server. Cassini is great for this. Then your code should look like this:
var server = new Server(80,"/", pathToWebSite);
server.Start();
var wc = new WebClient();
wc.DownloadFile(server.RootUrl + "myPage.aspx", filename);
server.Stop();
If you run this more than once, the server should be cached.
Note that you could also use a RuntimeHost as mentioned by code4life. Cassini does something similar. I'd give goth a try and see, what better fits your purpose.

ASPX files are dynamic => generated HTML depends on state of the application.
If you are missing the codebehind file, you cannot properly translate the code.
Mono Project has a code evaluator. That said, it won't help you without application state.
The only thing you can do is parse the aspx file as xml (if it is valid) and filter out the dynamic content.

I don't think you can do what you need to, without the ASP.NET runtime. If you have the ASP.NET runtime, and still want to be able to generate a HTML file from the content of an ASPX file, you could write an IHttpModule which writes the response text to a file.

If I'm understanding your question correctly, you want an instance of the Page class created (i.e. the aspx page is compiled) and ulimately the resulting html? But you want that to happen outside the context of a web server request?
If you're looking for the html after an aspx page is actually processed, why not just grab the html returned after a page is actually rendered via IIS or whatever?
Perhaps if you shared your motivation(s) for attempting this you'll get some solid suggestions...

Load usercontrols using Ajax

Im trying to find a best practice to load usercontrols using Ajax.
My first approach where simply using an UpdatePanel and popuplating it with LoadControl() on ajax postbacks but this would rerender other loaded usercontrols in the same UpdatePanel. Also I cannot have a predefined set of UpdatePanels since the number of UserControls I need to load will vary.
Is there any best practice for this type of scenario?
If needed I could implement a framework or some type of custom controls if that would be a solution but I would love to do this with ASP.NET 3.5 and the AjaxControlToolkit if possible.

Probably dozens of high-brow reasons for not doing it this way, but simply initalizing a page, adding the usercontrol, then executing and dump the resulting HTML wherever it may behoove you, is (in my simpleminded view) so mind-numbingly fast & fun that I just have to mention it...
Skip the UpdatePanels, just use a Label, a plain old span, or how about an acronym...
Using JQuery on the client side:
$('#SomeContainer').Load("default.aspx?What=GimmeSomeSweetAjax");
ServerSide:
if(Request.QueryString["What"]==GimmeSomeSweetAjax)
{
Page page = new Page();
Control control = (Control)LoadControl("~/.../someUC.ascx");
StringWriter sw = new StringWriter();
page.Controls.Add(control);
Server.Execute(page, sw, false);
Response.Write(sw.ToString());
Response.Flush();
Response.Close();
}
Nothing else executes, and the page's lifecycle has a real Kevorkian moment ;-)

I'm not sure, but maybe this tutorial by Scott Guthrie could be useful.

Had to use this on order to get the properties work!
var page = new Page();
var sw = new StringWriter();
var control = (UserControl)page.LoadControl("~/.../someUC.ascx");
var type = control.GetType();
type.GetProperty("Prop1").SetValue(control, value, null);
page.Controls.Add(control);
context.Server.Execute(page, sw, false);
context.Response.ContentType = "text/html";
context.Response.Write(sw.ToString());
context.Response.Flush();
context.Response.Close();