I've tried my best to add comments through the code but Im kind of stuck at certain parts.
// create a new instance of the HtmlDocument Class called doc
1: HtmlDocument doc = new HtmlDocument();
// the Load method is called here to load the variable result which is html
// formatted into a string in a previous code snippet
2: doc.Load(new StringReader(result));
// a new variable called root with datatype HtmlNode is created here.
// Im not sure what doc.DocumentNode refers to?
3: HtmlNode root = doc.DocumentNode;
4:
// a list is getting constructed here. I haven't had much experience
// with constructing lists yet
5: List<string> anchorTags = new List<string>();
6:
// a foreach loop is used to loop through the html document to
// extract html with 'a' attributes I think..
7: foreach (HtmlNode link in root.SelectNodes("//a"))
8: {
// dont really know whats going on here
9: string att = link.OuterHtml;
// dont really know whats going on here too
10: anchorTags.Add(att)
11: }
I've lifted this code sample from here. Credit to Farooq Kaiser
In HTML Agility Pack terms, "//a" means "Find all tags named 'a' or 'A' anywhere in the document". See XPATH docs for a more general help on XPATH (independently from HTML agility pack). So if you document looks like this:
<div>
anchor 1
<table ...>
anchor 2
</table>
</div>
You will get the two anchor HTML elements. OuterHtml represents the node's HTML including the node itself, while InnerHtml represents only the node's HTML content. So, here the two OuterHtml are:
anchor 1
and
anchor 2
Note I have specified 'a' or 'A' because HAP implementation takes care or HTML case insensitivity. And "//A" dos not work by default. You need to specify tags using lowercase.
The key is the SelectNodes method. This part used XPath to grab a list of nodes out of the HTML that matches your query.
This is where I learned my XPath: http://www.w3schools.com/xpath/default.asp
Then it just walks through those nodes that matched and gets the OuterHTML - the full HTML including the tags, and adds them to the list of strings. A List is basically just an array, but more flexible. If you only wanted the contents, and not the enclosing tags, you'd use HtmlNode.InnerHTML, or HtmlNode.InnerText.
Related
I'm using the Html Agility Pack for this task, basically I've got a URL, and my program should read through the content of the html page on it, and if it finds a line of text (ie: "John had three apples"), it should change a label's text to "Found it".
I tried to do it with contains, but I guess it only checks for one word.
var nodeBFT = doc.DocumentNode.SelectNodes("//*[contains(text(), 'John had three apples')]");
if (nodeBFT != null && nodeBFT.Count != 0)
myLabel.Text = "Found it";
EDIT: Rest of my code, now with ako's attempt:
if (CheckIfValidUrl(v)) // foreach var v in a list..., checks if the URL works
{
HtmlWeb hw = new HtmlWeb();
HtmlDocument doc = hw.Load(v);
try
{
if (doc.DocumentNode.InnerHtml.ToString().Contains("string of words"))
{
mylabel.Text = v;
}
...
One possible option is using . instead of text(). Passing text() to contains() function the way you did will, as you suspected, effective only when the searched text is the first direct child of the current element :
doc.DocumentNode.SelectNodes("//*[contains(., 'John had three apples')]");
In the other side, contains(., '...') evaluates the entire text content of current element, concatenated. So, just a heads up, the above XPath will also consider the following element for example, as a match :
<span>John had <br/>three <strong>apples</strong></span>
If you need the XPath to only consider cases when the entire keyword contained in a single text node, and therefore considers the above case as a no-match, you can try this way :
doc.DocumentNode.SelectNodes("//*[text()[contains(., 'John had three apples')]]");
If none of the above works for you, please post minimal HTML snippet that contains the keyword but returned no match, so we can examine further what possibly causes that behavior and how to fix it.
use this:
if (doc.DocumentNode.InnerHtml.ToString().Contains("John had three apples"))
myLabel.Text="Found it";
I am having an issue with XPath syntax as I dont understand how to use it to extract certain HTML statements.
I am trying to load a videos information from a channel page; http://www.youtube.com/user/CinemaSins/videos
I know there is a line that holds all the details from views, title, ID, ect.
Here is what I am trying to get from within the html:
Thats line 2836;
<div class="yt-lockup clearfix yt-lockup-video yt-lockup-grid context-data-item" data-context-item-id="ntgNB3Mb08Y" data-context-item-views="243,456 views" data-context-item-time="9:01" data-context-item-type="video" data-context-item-user="CinemaSins" data-context-item-title="Everything Wrong With The Chronicles Of Riddick In 8 Minutes Or Less">
I'm not sure how, But I have HTML Ability Pack added as a resouce and have started attempts on getting it.
Can someone explain how to get all of those details and the XPath syntax involved?
What I have attemped:
foreach (HtmlNode node in doc.DocumentNode.SelectNodes("//div[#class='yt-lockup clearfix yt-lockup-video yt-lockup-grid context-data-item']//a"))
{
if (node.ChildNodes[0].InnerHtml != String.Empty)
{
title.Add(node.ChildNodes[0].InnerHtml);
}
}
^ The above code works in only getting the title of each video. But it also has a blank input aswell. Code executed and result is below.
Your xpath is selecting the <a> element inside the <div>. If you want the attributes of the <div> too, then you need to either
a) select both elements and process them separately.
b) run several xpath queries where you specify the exact attribute you want.
Lets go with (a) for this example.
var nodes = doc.DocumentNode.SelectNodes("//div[#class='yt-lockup clearfix yt-lockup-video yt-lockup-grid context-data-item']");
and get the attributes and title like so:
foreach(var node in nodes)
{
foreach(var attribute in node.Attributes)
{
// ... Get the values of the attributes here.
}
var linkNodes = node.SelectNodes("//a"));
// ... Get the InnerHtml as per your own example.
}
I hope this was clear enough. Good luck.
Seems the answer given to me did not help what so ever so after HEAPS of digging, I finally understand how XPath works and managed to do it myself as seen below;
foreach (HtmlNode node in doc.DocumentNode.SelectNodes("//div[#class='yt-lockup clearfix yt-lockup-video yt-lockup-grid context-data-item']"))
{
String val = node.Attributes["data-context-item-id"].Value;
videoid.Add(val);
}
I just had to grab the content within the class. Knowing this made it alot easier to use.
I would like to search an HTML file for a certain string and then extract the tags. Given:
<div_outer><div_inner>Happy birthday<div><div>
I would like to search the HTML for "Happy birthday" then have a function return some sort of tag structure: this is the innermost tag, this is the tag outside that one, etc. So, <div_inner></div> then <div_outer></div>.
Any ideas? I am thinking HTMLAgilityPack but I haven't been able to figure out how to do it.
Thanks as always, guys.
The HAP is a good place indeed for this.
You can use the OuterHtml and Parent properties of a Node to get the enclosing elements and markup.
You could use xpath for this. I use //*[text()='Happy birthday'][1]/ancestor-or-self::* expression which finds a first (for simplicity) node which text content is Happy birthday, and then returns all the ancestors (parent, grandparent, etc.) of this node and the node itself:
var doc = new HtmlDocument();
doc.LoadHtml("<div_outer><div_inner>Happy birthday<div><div>");
var ancestors = doc.DocumentNode
.SelectNodes("//*[text()='Happy birthday'][1]/ancestor-or-self::*")
.Reverse()
.ToList();
It seems that the order of the nodes returned is the order the nodes found in the document, so I used Enumerable.Reverse method to reverse it.
This will return 2 nodes: div_inner and div_outer.
Here's a link:
http://www.covers.com/pageLoader/pageLoader.aspx?page=/data/nba/results/2010-2011/boxscore819588.html
I'm using HTML Agility Pack and I would like to extract, say, the 188 from the 'Odds' column. My editor gives /html/body/form/div/div[2]/div/table/tr/td[2]/div/table/tr[3]/td[7] when asked for path. I tried that path with various of omissions of body or html, but neither of them return any results when passed to .DocumentNode.SelectNodes(). I also tried with the // at the beginning (which, I assume, is the root of the document tree). What gives?
EDIT:
Code:
WebClient client = new WebClient();
string html = client.DownloadString(url);
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(html);
foreach(HtmlNode node in doc.DocumentNode.SelectNodes("/some/xpath/expression"))
{
Console.WriteLine("[" + node.InnerText + "]");
}
When scraping sites, you can't rely safely on the exact XPATH given by tools as in general, they are too restrictive, and in fact catch nothing most of the time. The best way is to have a look at the HTML and determine something more resilient to changes.
Here is a piece of code that works with your example:
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.Load(your html);
foreach (HtmlNode node in doc.DocumentNode.SelectNodes("//a[text()='MIA']/ancestor::tr/td[7]"))
{
Console.WriteLine(node.InnerText.Trim());
}
It outputs 188.
The way it works is:
select an A element with inner text set to "MIA"
find the parent TR element of this A element
get to the seventh TD of this TR element
and then we use InnerText property of that TD element
Try this:
/html/body/form/div/div[2]/div/table/*/tr/td[2]/div/table/*/tr[3]/td[7]
The * catch the mandatory <tbody> element that is part of the DOM representation of tables even if it is not denoted in the HTML.
Other than that, it's more robust to select by ID, CSS class name or some other unique property instead of by hierarchy and document structure:
//table[#class='data']//tr[3]/td[7]
By default HtmlAgilityPack treats form tag differently (because form tags can overlap), so you need to remove form tag from xpath, for examle: /html/body//div/div[2]/div/table/tr/td[2]/div/table/tr[3]/td[7]
Other way is to force HtmlAgilityPack to treat form tag as others:
HtmlNode.ElementsFlags.Remove("form");
I need to get a list of tags that contain a specific attribute. I am using DITA xml and I need to find out all tags that has a href attribute.
The problem here is that the attribute may be inside any tag so XPath will not work in this case. For example, an image tag may contain a href, a topicref tag may contain a href, and so on.
So I need to get a XmlNodeList (as returned by the getElementByTagName method). Ideally I need a method getElementByAttributeName that should return XmlNodeList.
I might have misunderstood your problem here, but I think you could possibly use an XPath expression.
var nodes = doc.SelectNodes("//*[#href='pic1.jpg']");
The above should return all elements with href='pic1.jpg', where doc is the XmlDocument
If you're on C#, then the following approach might work for you:
XDocument document = XDocument.Load(xmlReader);
XAttribute xa = new XAttribute("href", "pic1.jpg");
var attrList = document.Descendants().Where (d => d.Attributes().Contains(xa));