Develop Reference

Get HTML Code from a website after it completed loading

I am trying to get the HTML Code from a specific website async with the following code:
var response = await httpClient.GetStringAsync("url");
But the problem is that the website usually takes another second to load the other parts of it. Which I need, so the question is if I can load the site first and read the content after a certain amount of time.
Sorry if this question already got answered, but I didn't really know what to search for.
Thanks,
Twenty
Edit #1
If you want to try it yourself the URL is http://iloveradio.de/iloveradio/, I need the Title and the Artist which do not immediately load.

You are on the wrong direction. The referenced site has playlist api which returns json. you can get information from :
http://iloveradio.de/typo3conf/ext/ep_channel/Scripts/playlist.php
Edit: Chome Inspector is used to find out Playlist link

You could use Puppeteer-Sharp:
await new BrowserFetcher().DownloadAsync(BrowserFetcher.DefaultRevision);
using (var browser = await Puppeteer.LaunchAsync(new LaunchOptions { Headless = false }))
using (var page = await browser.NewPageAsync())
{
await page.SetViewportAsync(new ViewPortOptions() { Width = 1280, Height = 600 });
await page.GoToAsync("http://iloveradio.de/iloveradio/");
await page.WaitForSelectorAsync("#artisttitle DIV");
var artist = await page.EvaluateExpressionAsync<string>("$('#artisttitle DIV')[0].innerText");
Console.WriteLine(artist);
Console.ReadLine();
}

If there are things that load after, it means that they are generated by javascript code after page load (an ajax request for example), so no matter how long you wait, it won't have the content you want (because they are not in the source code when it loads).
Easy way to do it:
Use a WebBrowser and when DocumentCompleated event triggers wait till the element you want appears.
The Right Way:
find the javascript yourself and trigger it yourself (easy to say, hard to do).

The thing to understand here is that when you read the response from the URL, all you will ever get is the raw response, in this case the HTML source code the server replied with.
Unlike what you might see in your browser's DOM Inspector developer tools, you will only get the original HTML source from the page (what you might see in the "Page Source" developer tool) which will not include any dynamically created content (JavaScript) or loaded content (like iframes).
So you aren't getting what you see here in the DOM Inspector:
You are getting what you see here in the Page Source (View > Developer > View Source in Chrome):
You can't wait for that other content to load because it will never load since that HTML content isn't being parsed or rendered like a browser would.
You have several options available though:
See if the website has an API you can use
Determine where that content you want is actually loaded from, and make another/different HTTP request to that content (the Network Panel is helpful here)
Use a headless browser to programmatically load the page and dynamically read the page contents (this will add a lot of overhead, and should probably be avoided if possible)

I have checked out the website, data is loaded by javascript. You only can get the html using httpClient.GetStringAsync("url");.
As far as I know, there is no luck to get the elements what is manipulate by browser.

unable to get complete url after # using query string in asp.net c# [duplicate]

I know on client side (javascript) you can use windows.location.hash but could not find anyway to access from the server side. I'm using asp.net.

We had a situation where we needed to persist the URL hash across ASP.Net post backs. As the browser does not send the hash to the server by default, the only way to do it is to use some Javascript:
When the form submits, grab the hash (window.location.hash) and store it in a server-side hidden input field Put this in a DIV with an id of "urlhash" so we can find it easily later.
On the server you can use this value if you need to do something with it. You can even change it if you need to.
On page load on the client, check the value of this this hidden field. You will want to find it by the DIV it is contained in as the auto-generated ID won't be known. Yes, you could do some trickery here with .ClientID but we found it simpler to just use the wrapper DIV as it allows all this Javascript to live in an external file and be used in a generic fashion.
If the hidden input field has a valid value, set that as the URL hash (window.location.hash again) and/or perform other actions.
We used jQuery to simplify the selecting of the field, etc ... all in all it ends up being a few jQuery calls, one to save the value, and another to restore it.
Before submit:
$("form").submit(function() {
$("input", "#urlhash").val(window.location.hash);
});
On page load:
var hashVal = $("input", "#urlhash").val();
if (IsHashValid(hashVal)) {
window.location.hash = hashVal;
}
IsHashValid() can check for "undefined" or other things you don't want to handle.
Also, make sure you use $(document).ready() appropriately, of course.

[RFC 2396][1] section 4.1:
When a URI reference is used to perform a retrieval action on the
identified resource, the optional fragment identifier, separated from
the URI by a crosshatch ("#") character, consists of additional
reference information to be interpreted by the user agent after the
retrieval action has been successfully completed. As such, it is not
part of a URI, but is often used in conjunction with a URI.
(emphasis added)
[1]: https://www.rfc-editor.org/rfc/rfc2396#section-4

That's because the browser doesn't transmit that part to the server, sorry.

Probably the only choice is to read it on the client side and transfer it manually to the server (GET/POST/AJAX).
Regards
Artur
You may see also how to play with back button and browser history
at Malcan

Just to rule out the possibility you aren't actually trying to see the fragment on a GET/POST and actually want to know how to access that part of a URI object you have within your server-side code, it is under Uri.Fragment (MSDN docs).

Possible solution for GET requests:
New Link format: http://example.com/yourDirectory?hash=video01
Call this function toward top of controller or http://example.com/yourDirectory/index.php:
function redirect()
{
if (!empty($_GET['hash'])) {
/** Sanitize & Validate $_GET['hash']
If valid return string
If invalid: return empty or false
******************************************************/
$validHash = sanitizeAndValidateHashFunction($_GET['hash']);
if (!empty($validHash)) {
$url = './#' . $validHash;
} else {
$url = '/your404page.php';
}
header("Location: $url");
}
}

Set C# Property Value by Javascript

I have a C# Property CategoryID, I want to set it's value in Javascript.
I am trying to set the value CategoryID like below:
var sPath = window.location.pathname;
var catId = null;
var sPage = sPath.substring(sPath.lastIndexOf('/') + 1);
if (sPage == 'xyz.aspx')
{
<%=CommonUtility.CategoryID=4%>;
}
else if(sPage == 'zxy.aspx')
{
<%=CommonUtility.CategoryID=5%>;
}
But by this method I always get the value of CategoryID= 5(which is in else block) .
Please suggest me how can get the Property value based on condition.

You can't set a C# property from a client-side (js). You may use ajax to do some work, but you simply can't manipulate server-side code.
edit:
if you still wonder how it's possible you get a value, see Mike's explanation of that fact. But the truth remains. You can't. It's impossible. If you want to know the longer explanation, see how asp.net actually works, it's lifecycle etc. Simple way of putting it would be like this:
A user sends a request to the server using his browser. The server receives it, creates a requested page and instantiates needed classes etc. Then it's gets parsed and sent to the client as html (and other resources of course, like images, css...). The instantiated page class CAN'T be accessed and modified afterwards by the client, because it's already flushed by the server. Every request creates a new instance. There's no way of interacting js with c# anyway. Can you imagine what it would be like, if you could use some js to modify C# on a remote server? It doesn't make sense at all.

You cannot set properties in your code-behind using client-side script this way. The only way to do something like that would be to use AJAX to send data to your server, although I'm pretty sure that's not appropriate for your case.
When you call <%=CommonUtility.CategoryID = 4%>, the server actually executes that statement when it is parsing the page before it sends it to the client. The reason that the property value is 5 is that both of those statements get executed, regardless of the logic in your Javascript if block. Your client side code will not actually be executed by the browser until the server has already parsed both of those tags, which at that point it would be too late to accomplish what you want anyways.
Is there any reason that you simply can't do all of this in the code-behind on page load? Is there some reason you feel like this has to be handled in JS?
Edit:
If you are unable to access the code-behind file (.aspx.vb or .aspx.cs) then simply use a server script block in the top of your .aspx page
<%
If (Request.Path.ToLower().Contains("xyz.aspx")) Then
CommonUtility.CategoryId = 4
ElseIf (Request.Path.ToLower().Contains("zxy.aspx")) Then
CommonUtility.CategoryId = 5
End If
%>

You can't set the C# variable from client script, because all the server code runs first, then the page is sent to the browser.
The client code will end up looking like this:
var sPath = window.location.pathname;
var catId = null;
var sPage = sPath.substring(sPath.lastIndexOf('/') + 1);
if (sPage == 'xyz.aspx')
{
4;
}
else if(sPage == 'zxy.aspx')
{
5;
}
}

Passing parameters using Server.Transfer but not using query string

I have a registration form and want to transfer the user to the success page, telling him that an email was sent to his email. I need to transfer his email from the register page to the success page.
I found about Server.Transfer, but I can't find out how to send parameters. I don't want to use query string as I don't like revealing information in the URL.
What is the proper way (if possible) to do this?

Server.Transfer("destination.aspx", true)
You might see that the above code contains the name of the page to which the control is transferred and a Boolean value ‘True’ to indicate to preserve the current form state in the destination page.

Set a property in your login page and store in it, the email.
Once this is done, do a Server.Transfer("~/SuccessPage.aspx", true);
On the other page, where you redirected, you should check something like that :
if(this.PreviousPage != null) {
((LoginPageType)this.PreviousPage).MyEmailProperty;
}

When you using server transfer you just move execution to different server handler , user will no see the new url or the parameters so it safe to make this transfer.

I would rather recommend that you do it differently.
When the user clicks the register button, you verify it all and then send the email from the still current page (so you need not transfer data to another page at all). If all went well, you just redirect:
Response.Redirect("/order/success.aspx");
If something was wrong (validation errors, sending email caused an exception) you are still on the right page for a retry. I would not use Server.Transfer at all in most cases.

You'll have to persist the value somewhere. The obvious options are in the Session object, or in a database.

For this kind of use case. You can use Context.Items to save the data with a key and read the value using the same key in the child page you are doing the Server.Transfer. Context.Items are sort of per request scoped cache for you.
Context.Items['DataKey'] = Data;
Server.Transfer("~/AnyRouteRelativePath", true);

Retrieving data from a POST method in ASP.NET

I am using ASP.NET.
There is a system that needs to POST data to my site and all they asked for is for me to provide them with a URL.
So I gave them my URL http://www.example.com/Test.aspx.
Now I do not know exactly how they POST it but now on my Test.aspx page I need to write code that will save that data to a database.
But how would this work and what must I do on my Test.aspx page?
I wrote some code in my Page Load Event that sends me an email on Page Load to see if they actually hit the page and it does not seem like they are even?

The data from the request (content, inputs, files, querystring values) is all on this object HttpContext.Current.Request
To read the posted content
StreamReader reader = new StreamReader(HttpContext.Current.Request.InputStream);
string requestFromPost = reader.ReadToEnd();
To navigate through the all inputs
foreach (string key in HttpContext.Current.Request.Form.AllKeys)
{
string value = HttpContext.Current.Request.Form[key];
}

You can get a form value posted to a page using code similiar to this (C#) -
string formValue;
if (!string.IsNullOrEmpty(Request.Form["txtFormValue"]))
{
formValue= Request.Form["txtFormValue"];
}
or this (VB)
Dim formValue As String
If Not String.IsNullOrEmpty(Request.Form("txtFormValue")) Then
formValue = Request.Form("txtFormValue")
End If
Once you have the values you need you can then construct a SQL statement and and write the data to a database.

You need to examine (put a breakpoint on / Quick Watch) the Request object in the Page_Load method of your Test.aspx.cs file.