Here's a link:
http://www.covers.com/pageLoader/pageLoader.aspx?page=/data/nba/results/2010-2011/boxscore819588.html
I'm using HTML Agility Pack and I would like to extract, say, the 188 from the 'Odds' column. My editor gives /html/body/form/div/div[2]/div/table/tr/td[2]/div/table/tr[3]/td[7] when asked for path. I tried that path with various of omissions of body or html, but neither of them return any results when passed to .DocumentNode.SelectNodes(). I also tried with the // at the beginning (which, I assume, is the root of the document tree). What gives?
EDIT:
Code:
WebClient client = new WebClient();
string html = client.DownloadString(url);
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(html);
foreach(HtmlNode node in doc.DocumentNode.SelectNodes("/some/xpath/expression"))
{
Console.WriteLine("[" + node.InnerText + "]");
}
When scraping sites, you can't rely safely on the exact XPATH given by tools as in general, they are too restrictive, and in fact catch nothing most of the time. The best way is to have a look at the HTML and determine something more resilient to changes.
Here is a piece of code that works with your example:
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.Load(your html);
foreach (HtmlNode node in doc.DocumentNode.SelectNodes("//a[text()='MIA']/ancestor::tr/td[7]"))
{
Console.WriteLine(node.InnerText.Trim());
}
It outputs 188.
The way it works is:
select an A element with inner text set to "MIA"
find the parent TR element of this A element
get to the seventh TD of this TR element
and then we use InnerText property of that TD element
Try this:
/html/body/form/div/div[2]/div/table/*/tr/td[2]/div/table/*/tr[3]/td[7]
The * catch the mandatory <tbody> element that is part of the DOM representation of tables even if it is not denoted in the HTML.
Other than that, it's more robust to select by ID, CSS class name or some other unique property instead of by hierarchy and document structure:
//table[#class='data']//tr[3]/td[7]
By default HtmlAgilityPack treats form tag differently (because form tags can overlap), so you need to remove form tag from xpath, for examle: /html/body//div/div[2]/div/table/tr/td[2]/div/table/tr[3]/td[7]
Other way is to force HtmlAgilityPack to treat form tag as others:
HtmlNode.ElementsFlags.Remove("form");
Related
I am currently trying to have a plain HTML file template in folder which I then retrieve from a C# project. I had small placeholder words that I would then replace with the help of a stringbuilder. This would make the HTML template dynamic so I can change certain parts depending on what I need.
But I was wondering if it was possible to find the HTML element by ID or something along those lines. Instead of replacing each place where the placeholder word is, I would instead try to manipulate the HTML element.
I had tried something named HTML Agility Pack but I couldn't seem to get that to work.
I have this file of simple HTML.
<h1 id="test> </h1>
Which I then parse into the HTML Agility Pack and try to find the id of the element and then I try to parse some text into it.
private string DefineHTML(string html, string id)
{
var doc = new HtmlDocument();
doc.LoadHtml(html);
doc.GetElementbyId(id).AppendChild(doc.CreateElement("<p>test</p>"));
return doc.Text;
}
But it just outputs the same HTML it got into it instead of adding the next child to the element.
I need it to input the element into the heading element. Like so
<h1 id="test">
<p>test</p>
</h1>
So I was wondering if there was a way to do this. Since I feel like replacing each placeholder word seems like more trouble than it is worth.
private string DefineHTML(string html, string id)
{
var doc = new HtmlDocument();
doc.LoadHtml(html);
var htmlNode = doc.DocumentNode.SelectSingleNode($"//p[#id='{id}']");
var child = htmlDoc.CreateElement("<p>test</p>");
htmlNode.AppendChild(child);
return doc.DocumentNode.InnerHtml;
}
I would like to search an HTML file for a certain string and then extract the tags. Given:
<div_outer><div_inner>Happy birthday<div><div>
I would like to search the HTML for "Happy birthday" then have a function return some sort of tag structure: this is the innermost tag, this is the tag outside that one, etc. So, <div_inner></div> then <div_outer></div>.
Any ideas? I am thinking HTMLAgilityPack but I haven't been able to figure out how to do it.
Thanks as always, guys.
The HAP is a good place indeed for this.
You can use the OuterHtml and Parent properties of a Node to get the enclosing elements and markup.
You could use xpath for this. I use //*[text()='Happy birthday'][1]/ancestor-or-self::* expression which finds a first (for simplicity) node which text content is Happy birthday, and then returns all the ancestors (parent, grandparent, etc.) of this node and the node itself:
var doc = new HtmlDocument();
doc.LoadHtml("<div_outer><div_inner>Happy birthday<div><div>");
var ancestors = doc.DocumentNode
.SelectNodes("//*[text()='Happy birthday'][1]/ancestor-or-self::*")
.Reverse()
.ToList();
It seems that the order of the nodes returned is the order the nodes found in the document, so I used Enumerable.Reverse method to reverse it.
This will return 2 nodes: div_inner and div_outer.
I am trying to grab elements from HTML source based on the class or id name, using C# windows forms application. I am putting the source into a string using WebClient and plugging it into the HTMLAgilityPack using HtmlDocument.
However, all the examples I find with the HTMLAgilityPack pack parse through and find items based on tags. I need to find a specific id, of say a link in the html, and retrieve the value inside of the tags. Is this possible and what would be the most efficient way to do this? Everything I am trying to parse out the ids is giving me exceptions. Thanks!
You should be able to do this with XPath:
HtmlDocument doc = new HtmlDocument();
doc.Load(#"file.htm");
HtmlNode node = doc.DocumentNode.SelectSingleNode("//*[#id=\"my_control_id\"]");
string value = (node == null) ? "Error, id not found" : node.InnerHtml;
Quick explanation of the xpath here:
// means search everywhere in the path, Use SelectNodes if it will be matching multiples
* means match any type of node
[] define "Predicates" which are basically checking properties relative to this node
[#id=\"my_control_id\"] means find nodes that have an attribute named "id" with the value "my_control_id"
Further reference
I've tried my best to add comments through the code but Im kind of stuck at certain parts.
// create a new instance of the HtmlDocument Class called doc
1: HtmlDocument doc = new HtmlDocument();
// the Load method is called here to load the variable result which is html
// formatted into a string in a previous code snippet
2: doc.Load(new StringReader(result));
// a new variable called root with datatype HtmlNode is created here.
// Im not sure what doc.DocumentNode refers to?
3: HtmlNode root = doc.DocumentNode;
4:
// a list is getting constructed here. I haven't had much experience
// with constructing lists yet
5: List<string> anchorTags = new List<string>();
6:
// a foreach loop is used to loop through the html document to
// extract html with 'a' attributes I think..
7: foreach (HtmlNode link in root.SelectNodes("//a"))
8: {
// dont really know whats going on here
9: string att = link.OuterHtml;
// dont really know whats going on here too
10: anchorTags.Add(att)
11: }
I've lifted this code sample from here. Credit to Farooq Kaiser
In HTML Agility Pack terms, "//a" means "Find all tags named 'a' or 'A' anywhere in the document". See XPATH docs for a more general help on XPATH (independently from HTML agility pack). So if you document looks like this:
<div>
anchor 1
<table ...>
anchor 2
</table>
</div>
You will get the two anchor HTML elements. OuterHtml represents the node's HTML including the node itself, while InnerHtml represents only the node's HTML content. So, here the two OuterHtml are:
anchor 1
and
anchor 2
Note I have specified 'a' or 'A' because HAP implementation takes care or HTML case insensitivity. And "//A" dos not work by default. You need to specify tags using lowercase.
The key is the SelectNodes method. This part used XPath to grab a list of nodes out of the HTML that matches your query.
This is where I learned my XPath: http://www.w3schools.com/xpath/default.asp
Then it just walks through those nodes that matched and gets the OuterHTML - the full HTML including the tags, and adds them to the list of strings. A List is basically just an array, but more flexible. If you only wanted the contents, and not the enclosing tags, you'd use HtmlNode.InnerHTML, or HtmlNode.InnerText.
I need to get a list of tags that contain a specific attribute. I am using DITA xml and I need to find out all tags that has a href attribute.
The problem here is that the attribute may be inside any tag so XPath will not work in this case. For example, an image tag may contain a href, a topicref tag may contain a href, and so on.
So I need to get a XmlNodeList (as returned by the getElementByTagName method). Ideally I need a method getElementByAttributeName that should return XmlNodeList.
I might have misunderstood your problem here, but I think you could possibly use an XPath expression.
var nodes = doc.SelectNodes("//*[#href='pic1.jpg']");
The above should return all elements with href='pic1.jpg', where doc is the XmlDocument
If you're on C#, then the following approach might work for you:
XDocument document = XDocument.Load(xmlReader);
XAttribute xa = new XAttribute("href", "pic1.jpg");
var attrList = document.Descendants().Where (d => d.Attributes().Contains(xa));