Develop Reference

Parse page using AngleSharp

I want to parse website using c# with AngleSharp, it's easy to do with static pages, but there is a problem, I can't parse info avalible only for authorized users. What should I do to autorize programmatically into website and parse all info avalible for me?

Depending on the used authorization scheme this may either be super simple or ultra hard / impossible.
So let's first visit what can be done with AngleSharp:
Any kind of requests incl. their manipulation (on request, but also before response)
General cookie management (and their manipulation, of course)
Querying the DOM and perform "simple" actions (e.g., clicking a button, submitting a form)
Running trivial JavaScript files
Here trivial means: Scripts that do not need any capabilities beyond what AngleSharp offers, e.g., rendering tree information, advanced CSSOM access, ... - or scripts that require non-ES5 compliant parsers (e.g., make use of ES6 or some special non-standard capabilities).
Now since I do not know what is the authorization scheme or exact problem that you are hitting (some code / MWE would be helpful!) I'll just go for a simple click example.
var context = BrowsingContext.New(Configuration.Default.WithDefaultLoader().WithCookies());
var loginPage = await context.OpenAsync("http://yourpage.com");
var loginForm = loginPage.QuerySelector<IHtmlFormElement>("form");
var profilePage = await loginForm.SubmitAsync(new { userName = "myUser", password = "password" });
// get something on profilePage
Note that in this example the form field names for the login form are userName and password - they may be different for your login page. Also note that your page may contain multiple forms and the selector could be more sophisticated than a simple form.
HTH!

unable to get complete url after # using query string in asp.net c# [duplicate]

I know on client side (javascript) you can use windows.location.hash but could not find anyway to access from the server side. I'm using asp.net.

We had a situation where we needed to persist the URL hash across ASP.Net post backs. As the browser does not send the hash to the server by default, the only way to do it is to use some Javascript:
When the form submits, grab the hash (window.location.hash) and store it in a server-side hidden input field Put this in a DIV with an id of "urlhash" so we can find it easily later.
On the server you can use this value if you need to do something with it. You can even change it if you need to.
On page load on the client, check the value of this this hidden field. You will want to find it by the DIV it is contained in as the auto-generated ID won't be known. Yes, you could do some trickery here with .ClientID but we found it simpler to just use the wrapper DIV as it allows all this Javascript to live in an external file and be used in a generic fashion.
If the hidden input field has a valid value, set that as the URL hash (window.location.hash again) and/or perform other actions.
We used jQuery to simplify the selecting of the field, etc ... all in all it ends up being a few jQuery calls, one to save the value, and another to restore it.
Before submit:
$("form").submit(function() {
$("input", "#urlhash").val(window.location.hash);
});
On page load:
var hashVal = $("input", "#urlhash").val();
if (IsHashValid(hashVal)) {
window.location.hash = hashVal;
}
IsHashValid() can check for "undefined" or other things you don't want to handle.
Also, make sure you use $(document).ready() appropriately, of course.

[RFC 2396][1] section 4.1:
When a URI reference is used to perform a retrieval action on the
identified resource, the optional fragment identifier, separated from
the URI by a crosshatch ("#") character, consists of additional
reference information to be interpreted by the user agent after the
retrieval action has been successfully completed. As such, it is not
part of a URI, but is often used in conjunction with a URI.
(emphasis added)
[1]: https://www.rfc-editor.org/rfc/rfc2396#section-4

That's because the browser doesn't transmit that part to the server, sorry.

Probably the only choice is to read it on the client side and transfer it manually to the server (GET/POST/AJAX).
Regards
Artur
You may see also how to play with back button and browser history
at Malcan

Just to rule out the possibility you aren't actually trying to see the fragment on a GET/POST and actually want to know how to access that part of a URI object you have within your server-side code, it is under Uri.Fragment (MSDN docs).

Possible solution for GET requests:
New Link format: http://example.com/yourDirectory?hash=video01
Call this function toward top of controller or http://example.com/yourDirectory/index.php:
function redirect()
{
if (!empty($_GET['hash'])) {
/** Sanitize & Validate $_GET['hash']
If valid return string
If invalid: return empty or false
******************************************************/
$validHash = sanitizeAndValidateHashFunction($_GET['hash']);
if (!empty($validHash)) {
$url = './#' . $validHash;
} else {
$url = '/your404page.php';
}
header("Location: $url");
}
}

How to get data from a client on load?

I'm aware that data can be passed in through the URL, like "example.com/thing?id=1234", or it can be passed in through a form and a "submit" button, but neither of these methods will work for me.
I need to get a fairly large xml string/file. I need to parse it and get the data from it before I can even display my page.
How can I get this on page load? Does the client have to send a http request? Or submit the xml as a string to a hidden form?
Edit with background info:
I am creating a widget that will appear in my customer's application, embedded using C# WebBrowser control, but will be hosted on my server. The web app needs to pass some data (including a token for client validation) to my widget via xml, and this needs to be loaded in first thing when my widget starts up.

ASP.NET MVC 4 works great with jQuery and aJax posts. I have accomplished this goal many times by taking advantage of this.
jQuery:
$(document).ready(function() {
$.ajax({
type: "POST",
url: "/{controller}/{action}/",
data: { clientToken: '{token}', foo: 'bar',
success: function (data, text) {
//APPEND YOUR PAGE WITH YOUR PARSED XML DATA
//NOTE: 'data' WILL CONTAIN YOUR RETURNED RESULT
}
});
});
MVC Controller:
[HttpPost]
public JsonResult jqGetXML(string clientToken, string foo)
{
JsonResult jqResult = new JsonResult();
//GET YOUR XML DATA AND DO YOUR WORK
jqResult.Data = //WHATEVER YOU WANT TO RETURN;
return jqResult;
}
Note: This example returns Json data (easier to work with IMO), not XML. It also assumes that the XML data is not coming from the client but is stored server-side.
EDIT: Here is a link to jQuery's Ajax documentation,
http://api.jquery.com/jQuery.ajax/

Assuming you're using ASP.NET, since you say it's generated by another page, just stick the XML in the Session state.

Another approach, not sure if it helps in your situation.
If you share the second level domain name on your two sites (i.e. .....sitename.com ) then another potential way to share data is you could have them assert a cookie at this 2nd level with the token and xml data in it. You'll then be provided with this cookie.
I've only done this to share authentication details, you need to share machine keys at a minimum to support this (assuming .Net here...).

You won't be able to automatically upload a file from the client to the server - at least not via a browser using html/js/httprequests. The browser simply will not allow this.
Imagine the security implications if browsers allowed you to silently upload a file from the clients local machine without their knowledge.

Sample solution:
Background process imports xml file and parses it. The background process knows it is for customer YYY and updates their information so it know the xml file has been processed.
A visitor goes to the customer's web application where the widget is embedded. In the markup of the widget the customer token has been added. This could be in JavaScript, Flash, iFrame, etc.
When the widget loads, it makes a request to you app which then checks to see if the file was parsed for the provided customer (YYY) if it has, then show the page/widget.

If the XML is being served via HTTP you can use Liqn to parse the data.
Ex.
public partial class Sample : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
string url = "http://news.yahoo.com/rss/";
var el = XElement.Load(url).Elements("channel");
StringBuilder output = new StringBuilder();
foreach (var c in el.Elements())
{
switch (c.Name.LocalName.ToLower())
{
case "title":
output.Append(c.Value);
output.Append("<br />");
break;
}
}
this.Label1.Text = output.ToString();
}
}

It is not exactly clear what the application is and what kind of options you have, and what kind of control over web server you have.
If you are the owner of the web server/application your options are way wider. You can first send a file to web-server with HTTP POST or PUT, including a random token, and then use the same token for GET with token in the query string
or use other options, applicable to third party-owned websites
if you are trying to consume some auth api, learn more about it. since you are hosting web browser control, you have plenty of options to script it. including loading whatever form, setting textarea or hidden field text with your xml and then simulating a submit button click. you can then respond to any redirects and html responses.
you can also inject javascript inside the page that would send it to server with ajax request.
the choice heavily depends on the interaction model.
if you need better advice, it would be most helpful if you provided sample/simplified url/url pattern, form content, and sequence of events that is expected from you from code/api/sdk perspective. they are usually quite friendly.

There are limited number of ways to pass data between pages. Personally for this I would keep in session during the generating page and clear it when it is retrieved in the required page.
If it is generated server side then there is no reason to retrieve it from client side.
http://msdn.microsoft.com/en-us/library/6c3yckfw(v=vs.100).aspx

Create a webservice that your C# app can POST the XML to and get back HTML in response. Load this HTML string into the WebBrowser control rather than pointing the control to a URL.

Set C# Property Value by Javascript

I have a C# Property CategoryID, I want to set it's value in Javascript.
I am trying to set the value CategoryID like below:
var sPath = window.location.pathname;
var catId = null;
var sPage = sPath.substring(sPath.lastIndexOf('/') + 1);
if (sPage == 'xyz.aspx')
{
<%=CommonUtility.CategoryID=4%>;
}
else if(sPage == 'zxy.aspx')
{
<%=CommonUtility.CategoryID=5%>;
}
But by this method I always get the value of CategoryID= 5(which is in else block) .
Please suggest me how can get the Property value based on condition.

You can't set a C# property from a client-side (js). You may use ajax to do some work, but you simply can't manipulate server-side code.
edit:
if you still wonder how it's possible you get a value, see Mike's explanation of that fact. But the truth remains. You can't. It's impossible. If you want to know the longer explanation, see how asp.net actually works, it's lifecycle etc. Simple way of putting it would be like this:
A user sends a request to the server using his browser. The server receives it, creates a requested page and instantiates needed classes etc. Then it's gets parsed and sent to the client as html (and other resources of course, like images, css...). The instantiated page class CAN'T be accessed and modified afterwards by the client, because it's already flushed by the server. Every request creates a new instance. There's no way of interacting js with c# anyway. Can you imagine what it would be like, if you could use some js to modify C# on a remote server? It doesn't make sense at all.

You cannot set properties in your code-behind using client-side script this way. The only way to do something like that would be to use AJAX to send data to your server, although I'm pretty sure that's not appropriate for your case.
When you call <%=CommonUtility.CategoryID = 4%>, the server actually executes that statement when it is parsing the page before it sends it to the client. The reason that the property value is 5 is that both of those statements get executed, regardless of the logic in your Javascript if block. Your client side code will not actually be executed by the browser until the server has already parsed both of those tags, which at that point it would be too late to accomplish what you want anyways.
Is there any reason that you simply can't do all of this in the code-behind on page load? Is there some reason you feel like this has to be handled in JS?
Edit:
If you are unable to access the code-behind file (.aspx.vb or .aspx.cs) then simply use a server script block in the top of your .aspx page
<%
If (Request.Path.ToLower().Contains("xyz.aspx")) Then
CommonUtility.CategoryId = 4
ElseIf (Request.Path.ToLower().Contains("zxy.aspx")) Then
CommonUtility.CategoryId = 5
End If
%>

You can't set the C# variable from client script, because all the server code runs first, then the page is sent to the browser.
The client code will end up looking like this:
var sPath = window.location.pathname;
var catId = null;
var sPage = sPath.substring(sPath.lastIndexOf('/') + 1);
if (sPage == 'xyz.aspx')
{
4;
}
else if(sPage == 'zxy.aspx')
{
5;
}
}

How to do live update on section of page?

I need to refresh sections of my page to update when there is new data! what do i do? use jquery?
examples:

Yes, jQuery's great for this. Look into these methods:
http://api.jquery.com/category/ajax/

jQuery is usually not needed for basic AJAX. A simple example could be as follows:
liveSection = document.getElementById('latest-news');
request = new XMLHttpRequest;
request.open('GET', '/news-ajax', true);
request.send(null);
request.addEventListener('readystatechange', function() {
if (request.readyState == 4 && request.status == 200)
liveSection.innerHTML = request.responseText;
}, false);

If you're using Asp.NET, why not use an UpdatePanel? It's simple and reliable.
Edit
I just re-read your question and it looks (based on how you worded it) that you want to update a user's web page when the data changes on the server. I just want to make sure you understand that in a web app, the server can't trigger the browser to do anything. The server can only respond to browser requests, so you'll need to have the browser poll the server periodically.

I've created a simple example (using jQuery) to help you understand the breakdown of the things that will need to happen, which are:
1 - Periodically polling the server (via ajax) using Javascript's setTimeout to check that what is loaded into the browser is the latest content. We can achieve this by fetching the latest item ID or whatever and comparing it to a variable, which was initialised when the page first loaded.
2 - If the item ID does not match (a bit of an oversimplification) then we can assume that there has been an update, so we replace the content of some element with some content from some page.
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
<script>
function getLatestStuff() {
// fetch the output from a context which gives us the latest id
$.get("isthereanupdate.aspx", function(response) {
// we have the response, now compare to the stored value
if(resp != lastItemId) {
// it's different, so update the variable and grab the latest content
lastItemId = response;
$("#latestStuffDiv").load("updates.aspx");
}
});
}
$(document).ready(function() {
// the value which initializes this comes from the server
var lastItemId = 7;
setTimeout(getLatestStuff, 10000);
});
</script>

If you want to update when there is new data, you should look into comet or pubsubhubbub. jQuery can help you display the data in a pretty way, but you'll need to write stuff on the serverside to send the data.