Can anyone help me by explaining how to extract urls/links from HTML File in C#
look at Html Agility Pack
HtmlDocument doc = new HtmlDocument();
doc.Load("file.htm");
foreach(HtmlNode link in doc.DocumentNode.SelectNodes("//a[#href]"))
{
HtmlAttribute att = link.Attributes["href"];
yourList.Add(att.Value)
}
doc.Save("file.htm");
Use HTMLAgility Pack...
private List<string> ParseLinks(string html)
{
var doc = new HtmlDocument();
doc.LoadHtml(html);
var nodes = doc.DocumentNode.SelectNodes("//a[#href]");
return nodes == null ? new List<string>() : nodes.ToList().ConvertAll(r => r.Attributes.ToList().ConvertAll(i => i.Value)).SelectMany(j => j).ToList();
}
It works for me.
You can use an HTQL COM object and query the page using query:
<a>:href
HTQLCOMLib.HtqlControl h = new HTQLCOMLib.HtqlControl();
string page = "<html><body><a href='test1.html'>test1</a><a href='test2.html'>test2</a> </body></html>";
h.setSourceData(page, page.Length);
h.setQuery("<a>: href ");
for (h.moveFirst(); 0 == h.isEOF(); h.moveNext() )
{
MessageBox.Show(h.getValueByIndex(1));
}
It will show messages of:
test1.html
test2.html
Related
I have to replace the tag with HAP - HTML Agility Pack, in order to get a link without removing the link text. For e.g. in this case:
<p>This is the link</p>
I want to replace the link and the desired result should be:
<p>This is <span>the link<span></p>
I made this function, getting a html string as input.
public string CleanLinks(string input) {
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(input);
var links = doc.DocumentNode.SelectNodes("//a");
if (links == null) return input;
foreach (HtmlNode tb in links)
{
HtmlNode lbl = doc.CreateElement("span");
lbl.InnerHtml = tb.InnerHtml;
tb.ParentNode.ReplaceChild(lbl, tb);
}
return doc.DocumentNode.OuterHtml;
}
i want to get "href" link in this html node.
I already trying, but the result still not work.
This is the code:
<a title="ASUS ROG" class="product-media__link js-tracker-product-link" href="https://www.bukalapak.com/p/komputer/laptop/8vl4vm-jual-asus-rog?search%5Bkeywords%5D=asus%20rog&from=omnisearch">
Here are some sample code to extract the href url on the page:
HtmlDocument htmlDoc = new HtmlDocument();
htmlDoc.Load("yourpage.html");
var link = htmlDoc.DocumentNode
.Descendants("a")
.First(x => x.Attributes["title"] != null
&& x.Attributes["title"].Value == "ASUS ROG");
string hrefValue = link.Attributes["href"].Value;
I made a simple program for fetching youtube users in comments.
This is the code
string html;
using (var client = new WebClient())
{
html = client.DownloadString("http://www.youtube.com/watch?v=ER5EnjskCvE");
}
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);
List<string> data = new List<string>();
HtmlNodeCollection nodeCollection = doc.DocumentNode.SelectNodes("//*[#id='comments-view']/ul[1]/li[1]/a/span/span/span/span/img");
foreach (HtmlNode node in nodeCollection)
{
data.Add(node.GetAttributeValue("alt",null));
}
But i have a problem that my nodeCollection is returning null.
For the XPath i used copy XPath option in chrome under F12
try this replace "*" , "div"
"/html/body//div[#id='comments-view']/ul[1]/li[1]/a/span/span/span/span/img"
Html code:
<div>
<div>Name</div>
Date
</div>
How to use HtmlAgilityPack to get the Name and Date values?
HtmlDocument doc = new HtmlDocument();
doc.Load("file.htm");
foreach(HtmlNode div in doc.DocumentElement.SelectNodes("//div"])
{
string parent = div.InnerText; //this will give you Name
foreach (HtmlNode child in div.ChildNodes)
{
string childDiv = child.InnerText; //this will give you the child
}
}
var doc = new HtmlDocument();
doc.LoadHtml("<div><div>Name</div>Date</div>");
var nodes = doc.DocumentNode
.DescendantNodes()
.Where(n => n.NodeType == HtmlNodeType.Text);
foreach (var node in nodes)
{
var value = node.InnerText;
}
Hard-coded:
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml("<div><div>Name</div>Date</div>");
Console.WriteLine(((HtmlTextNode)doc.DocumentNode.ChildNodes[0].ChildNodes[0]).Text);
Console.WriteLine(((HtmlTextNode)doc.DocumentNode.ChildNodes[1]).Text);
If what you're looking for, is all the text nodes (as in Kevin Babcock's answer!). You have to remember that text in Xml (and with HtmlAgility) are nodes.
Input
<html><body><p>foo <a href='http://www.example.com'>bar</a> baz</p></body></html>
Output
foo
bar
baz
I know of htmldoc.DocumentNode.InnerText, but it will give foobarbaz - I want to get each text, not all at a time.
XPATH is your friend :)
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(#"<html><body><p>foo <a href='http://www.example.com'>bar</a> baz</p></body></html>");
foreach(HtmlNode node in doc.DocumentNode.SelectNodes("//text()"))
{
Console.WriteLine("text=" + node.InnerText);
}
var root = doc.DocumentNode;
var sb = new StringBuilder();
foreach (var node in root.DescendantNodesAndSelf())
{
if (!node.HasChildNodes)
{
string text = node.InnerText;
if (!string.IsNullOrEmpty(text))
sb.AppendLine(text.Trim());
}
}
This does what you need, but I am not sure if this is the best way. Maybe you should iterate through something other than DescendantNodesAndSelf for optimal performance.
I was in the need of a solution that extracts all text but discards the content of script and style tags. I could not find it anywhere, but I came up with the following which suits my own needs:
StringBuilder sb = new StringBuilder();
IEnumerable<HtmlNode> nodes = doc.DocumentNode.Descendants().Where( n =>
n.NodeType == HtmlNodeType.Text &&
n.ParentNode.Name != "script" &&
n.ParentNode.Name != "style");
foreach (HtmlNode node in nodes) {
Console.WriteLine(node.InnerText);
var pageContent = "{html content goes here}";
var pageDoc = new HtmlDocument();
pageDoc.LoadHtml(pageContent);
var pageText = pageDoc.DocumentNode.InnerText;
The specified example for html content:
<html><body><p>foo <a href='http://www.example.com'>bar</a> baz</p></body></html>
will produce the following output:
foo bar baz
public string html2text(string html) {
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(#"<html><body>" + html + "</body></html>");
return doc.DocumentNode.SelectSingleNode("//body").InnerText;
}
This workaround is based on Html Agility Pack. You can also install it via NuGet (package name: HtmlAgilityPack).
https://github.com/jamietre/CsQuery
have you tried CsQuery? Though not being maintained actively - it's still my favorite for parsing HTML to Text. Here's a one liner of how simple it is to get the Text from HTML.
var text = CQ.CreateDocument(htmlText).Text();
Here's a complete console application:
using System;
using CsQuery;
public class Program
{
public static void Main()
{
var html = "<div><h1>Hello World <p> some text inside h1 tag under p tag </p> </h1></div>";
var text = CQ.CreateDocument(html).Text();
Console.WriteLine(text); // Output: Hello World some text inside h1 tag under p tag
}
}
I understand that OP has asked for HtmlAgilityPack only but CsQuery is another unpopular and one of the best solutions I've found and wanted to share if someone finds this helpful. Cheers!
I just changed and fixed some people's answers to work better:
var document = new HtmlDocument();
document.LoadHtml(result);
var sb = new StringBuilder();
foreach (var node in document.DocumentNode.DescendantsAndSelf())
{
if (!node.HasChildNodes && node.Name == "#text" && node.ParentNode.Name != "script" && node.ParentNode.Name != "style")
{
string text = node.InnerText?.Trim();
if (text.HasValue() && !text.StartsWith('<') && !text.EndsWith('>'))
sb.AppendLine(System.Web.HttpUtility.HtmlDecode(text.Trim()));
}
}