I have an HTML document with a number of links. Some of these links contain two identical rel attributes, and I would like to iterate over all of the a-tags, check if there are more than one rel attribute and remove one if there is, but I can't figure out how to do it.
Example:
<a rel="nofollow" rel="nofollow" href="blah"> link<a>
Should be:
<a rel="nofollow" href="blah"> link<a>
Thanks for any help.
If are you attributes are identical, this is a possible solution using nuget HtmlAgilityPack
using System.IO;
using System.Text;
using System.Linq;
using HtmlAgilityPack;
public class Program
{
public static void Main()
{
var html =
#"
<div>
<a rel='nofollow' rel='nofollow' href='lah'> link1</a>
<a rel='qwerty' rel='qwerty' href='lah'> link2</a>
<a rel='asdf' rel='asdf' href='lah'> link3</a>
</div>";
var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(html);
var htmlNodes = htmlDoc.DocumentNode.SelectNodes("//a");
foreach (var node in htmlNodes)
{
var attr = node.Attributes["rel"];
while(node.Attributes.Contains("rel")) node.Attributes.Remove("rel");
node.Attributes.Add("rel", attr.Value);
}
using var ms = new MemoryStream();
htmlDoc.Save(ms, Encoding.UTF8);
var result = Encoding.UTF8.GetString(ms.ToArray());
System.Console.WriteLine(result);
}
}
Related
Hello i am pretty new in c# sphere. I want to make a little program that will fetch data from the given page.
It is a fragment of website:
<h3 class="filmInfo__header cloneToCast cloneToOtherInfo" data-type="directing-header">reżyseria</h3>
<div class="filmInfo__info cloneToCast cloneToOtherInfo" data-type="directing-info"> <span itemprop="url" content="/person/Rupert+Sanders-1121101"></span> <span itemprop="name">Rupert Sanders</span> </div>
I want to get data from "Data-type="Directing-info" and get a result from title="Rupert Sanders"
Somebody can help me ?
My very simple code:
private void button1_Click(object sender, EventArgs e)
{
var url = "https://www.filmweb.pl/film/Kr%C3%B3lewna+%C5%9Anie%C5%BCka+i+%C5%81owca-2012-600541";
var httpClient = new HttpClient();
var html = httpClient.GetStringAsync(url);
textBox1.Text = (html.Result);
}
C# or .NET does not offer native HTML parsing functionality. However, there are a handful of libraries which provides HTML parsing functionality. For example, you can use Html Agility Pack.
First, you need to install it into your project. You can easily install it with NuGet Package Manager in Visual Studio if you use it.
After that, you can use it like this with your input HTML:
private void button1_Click(object sender, EventArgs e)
{
var url = "https://www.filmweb.pl/film/Kr%C3%B3lewna+%C5%9Anie%C5%BCka+i+%C5%81owca-2012-600541";
var httpClient = new HttpClient();
var html = httpClient.GetStringAsync(url);
// Create a HtmlDocument and load your HTML into it.
var htmlDocument = new HtmlDocument();
htmlDocument.LoadHtml(html.Result);
// Find your desired node inside it.
HtmlNode directingInfoNode = htmlDocument.DocumentNode.SelectSingleNode("//div[#data-type='directing-info']/a");
// Get the title attribute of that node.
HtmlAttribute titleAttribute = directingInfoNode.Attributes["title"];
textBox1.Text = (titleAttribute.Value);
}
Of course, you will need to put necessary using statement to the top of your file:
using HtmlAgilityPack;
I need take some data from url but inside of div have 3 a element. And i need slice it and take it to seperate list. In Url html data is like :
<div class="nesne row nobetciDiv ">
<div class="col-md-4 tablo yuksek">
<div class="hucre hucre-ortala">
ARAT ECZANESİ // I need take it to first list.
<br>
0(242) 237-67-22 // I need take it to second list.
</div>
</div>
<div class="col-md-8 tablo yuksek">
<div class="hucre hucre-ortala">
<a href="https://maps.google.com/maps?q=36.8905816274,30.6800764847" class="nadres" target="_blank">
<img src="/Resim/Upload/mapi.png" class="mapi">
K.KARABEKIR CD.EGITIM ARASTIRMA HASTANESI ACIL KARSISI </a> // I need take it to third list.
</div>
</div>
</div>
I tried do like :
List<string> pharmacyName = new List<string>();
List<string> pharmacyAdress = new List<string>();
List<string> pharmacyNumber = new List<string>();
Uri url = new Uri("https://www.antalyaeo.org.tr/tr/nobetci-eczaneler");
WebClient client = new WebClient();
client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
string html = client.DownloadString(url);
HtmlAgilityPack.HtmlDocument document = new HtmlAgilityPack.HtmlDocument();
document.LoadHtml(html);
var xpath = "//text()[not(normalize-space())]";
var emptyNodes = document.DocumentNode.SelectNodes(xpath);
foreach (HtmlNode emptyNode in emptyNodes)
{
emptyNode.ParentNode
.ReplaceChild(HtmlTextNode.CreateNode(""), emptyNode);
}
HtmlNodeCollection title = document.DocumentNode.SelectNodes("//div[contains(#class,'nobetciDiv')]");
foreach (var item in title)
{
HtmlNode x = item.SelectSingleNode("//div[contains(#class,'col-md-4')]");
HtmlNode a = x.SelectSingleNode("//a");
pharmacyName.Add(a.InnerText);
// Its giving me like : ARAT ECZANESİ0(242) 237-67-22 Cant seperate it or cant take third one.
}
Sorry for my bad english i tried describe with codes mostly. So thanks for all helps!
Firstly,<div class="col-md-4 tablo yuksek"></div> has two <a></a>,so you cannot use SelectSingleNode.Then the third <a></a> is in <div class="col-md-8 tablo yuksek"></div> rather than <div class="col-md-4 tablo yuksek"></div>.Try to change
foreach (var item in title)
{
HtmlNode x = item.SelectSingleNode("//div[contains(#class,'col-md-4')]");
HtmlNode a = x.SelectSingleNode("//a");
pharmacyName.Add(a.InnerText);
// Its giving me like : ARAT ECZANESİ0(242) 237-67-22 Cant seperate it or cant take third one.
}
to
foreach (var item in title)
{
HtmlNode x = item.SelectSingleNode("//div[contains(#class,'col-md-4')]").FirstChild;
pharmacyName.Add(x.SelectNodes("//a[#href]")[0].InnerText);
pharmacyAdress.Add(x.SelectNodes("a[#href]")[1].InnerText);
string s3= item.SelectSingleNode("//div[contains(#class,'col-md-8')]").SelectSingleNode("//a[contains(#class,'nadres')]").InnerText.Replace("\r\n","").Trim();
pharmacyNumber.Add(s3);
}
result:
I am trying to get element by using x-path tree element but showing null, and this type of x-path work for other site for me, only 2% site this types of X-Path not working, also i tried x-path from chrome also but when my x-path not work that time chrome x-path also not work.
public static void Main()
{
string url = "http://www.ndrf.gov.in/tender";
HtmlWeb web = new HtmlWeb();
var htmlDoc = web.Load(url);
var nodetest1 = htmlDoc.DocumentNode.SelectSingleNode("/html[1]/body[1]/section[2]/div[1]/div[1]/div[1]/div[1]/div[2]/table[1]"); // i want this type // not wroking
//var nodetest2 = htmlDoc.DocumentNode.SelectSingleNode("//*[#id=\"content\"]/div/div[1]/div[2]/table"); // from Google chrome // not wroking
//var nodetest3 = htmlDoc.DocumentNode.SelectSingleNode("//*[#id=\"content\"]"); // by ID but i don't want this type // wroking
Console.WriteLine(nodetest1.InnerText); //fail
//Console.WriteLine(nodetest2.InnerText); //fail
//Console.WriteLine(nodetest3.InnerText); //proper but I don't want this type
}
The answer that #QHarr suggested works perfectly, But the reason you get null with a correct x-path, is that there is a javascript file in the header of the site, that adds a wrapper div around the table, and since getting result in HtmlAgilityPack seems not loading or executing js, the x-path returns null.
what you observe, after that js runs is:
<div class="view-content">
<div class="guide-text">
...
</div>
<div class="scroll-table1">
<!-- Your table is here -->
</div>
</div>
but what actually you get whithout that js, is:
<div class="view-content">
<!-- Your table is here -->
</div>
thus your x-path should be:
var nodetest1 = htmlDoc.DocumentNode.SelectSingleNode("/html[1]/body[1]/section[2]/div[1]/div[1]/div[1]/div[1]/table[1]");
Your xpath when used in browser selects for entire table. You can shorten and use as follows (fiddle):
using System;
using HtmlAgilityPack;
public class Program
{
public static void Main()
{
string url = "http://www.ndrf.gov.in/tender";
HtmlWeb web = new HtmlWeb();
var htmlDoc = web.Load(url);
var nodetest1 = htmlDoc.DocumentNode.SelectSingleNode("//table");
Console.WriteLine(nodetest1.InnerText);
}
}
Use Fizzler.Systems.HtmlAgilityPack
details here : https://www.nuget.org/packages/Fizzler.Systems.HtmlAgilityPack/
This library adds extension methods called QuerySelector and QuerySelectorAll, that takes CSS Selector not XPath.
Ali Bordbar caught perfect, This Url adds a wrapper div when I navigating URL in WebBrowser control in this all JavaScript file are loaded,
but when i load URL using HtmlWeb there is none of the JavaScript file loaded.
The HtmlWeb retrieves the static HTML response that the server sends, and does not execute any javascript, whereas a WebBrowser would.
So WebBrowser control HTML DOM data XPath and HtmlWeb HTML DOM data XPath not match.
My below code work perfect for this switchvation
HtmlWeb web = new HtmlWeb();
web.AutoDetectEncoding = true;
HtmlAgilityPack.HtmlDocument theDoc1 = web.Load("http://www.ndrf.gov.in/tender");
var HtmlDoc = new HtmlAgilityPack.HtmlDocument();
var bodytag = theDoc1.DocumentNode.SelectSingleNode("//html");
HtmlDoc.LoadHtml(bodytag.OuterHtml);
var xpathHtmldata = HtmlDoc.DocumentNode.SelectSingleNode(savexpath); //savexpath is my first xpath make from HTML DOM data of WebBrowser control which is work for most url.
if (xpathHtmldata == null)
{
//take last tag name from first xpath
string mainele = savexpath.Substring(savexpath.LastIndexOf("/") + 1);
if (mainele.Contains("[")) { mainele = mainele.Remove(mainele.IndexOf("[")); }
//collect all tag name with name of which is sotre in mainele variable
var taglist = HtmlDoc.DocumentNode.SelectNodes("//" + mainele);
foreach (var ele in taglist) //check one by one element
{
string htmltext1 = ele.InnerText;
htmltext1 = Regex.Replace(htmltext1, #"\s", "");
htmltext1 = htmltext1.Replace("&", "&").Trim();
htmltext1 = htmltext1.Replace(" ", "").Trim();
string htmltext2 = saveInnerText; // my previus xpath text from HTML DOM data of WebBrowser control
htmltext2 = Regex.Replace(htmltext2, #"\s", "");
if (htmltext1 == htmltext2) // check equality to my previus xpath text..if it is equal thats my new xpath
{
savexpath = ele.XPath;
break;
}
}
}
Ok. So I found this code online everything is working on it but it shows me the div class I am searching for but removes all the text. Any idea why? Heres a example of what its outputting...
<div class="marquee"><img src="logo.png" /></div>
<div id="joke">
<div id="setup" class="exit-left"></div>
<div id="punchline">
<div class="question"></div>
<div id="zing" class="exit-right"></div>
</div>
</div>
<div id="buttons">
<input id="tell-me" class="exit-bottom no-select" type="button" value="Tell Me!" />
<!--<input id="another" class="exit-bottom" type="button" value="Another!" />-->
<table class="another exit-bottom no-select">
<tr>
<td class="another" colspan="3">Another</td>
<td class="share"><div class="share-img"></div>Share</td>
</tr>
</table>
</div>
And the innertext is not shown at all...
And here is my code is VS.
var doc = new HtmlAgilityPack.HtmlDocument();
HtmlAgilityPack.HtmlNode.ElementsFlags["br"] = HtmlAgilityPack.HtmlElementFlag.Empty;
doc.OptionWriteEmptyNodes = true;
try
{
var webRequest = HttpWebRequest.Create("http://dadjokegenerator.com/");
Stream stream = webRequest.GetResponse().GetResponseStream();
doc.Load(stream);
stream.Close();
}
catch (System.UriFormatException uex)
{
throw;
}
catch (System.Net.WebException wex)
{
throw;
}
//get the div by id and then get the inner text
doc.GetElementbyId("content").InnerHtml;
await e.Channel.SendMessage("test " + divString); `
Although your code correctly downloads content of page http://dadjokegenerator.com/, InnerHtml is empty, because this page actually doesn't contain joke you are looking for (you can see that, if you display source code of page in you web browser - e.g. in Firefox press CTRL+U). Joke is added to this page later by javascript. If you look at source code of this Javascript at http://dadjokegenerator.com/js/main.js, you can see that individual jokes are downloaded from URL http://dadjokegenerator.com/api/api.php?a=j<=r&vj=0
Here is minimal sample to download joke from this URL. I ommited all error-checks for simplicity and I used free Json.NET library for JSON deserialization:
public class Joke
{
public int Id;
public string Setup;
public string Punchline;
public override string ToString()
{
return Setup + " " + Punchline;
}
}
public static Joke GetJoke()
{
var request = HttpWebRequest.Create("http://dadjokegenerator.com/api/api.php?a=j<=r&vj=0");
using (var response = request.GetResponse())
{
using (var stream = response.GetResponseStream())
{
using (var reader = new StreamReader(stream))
{
var jokeString = reader.ReadToEnd();
Joke[] jokes = JsonConvert.DeserializeObject<Joke[]>(jokeString);
return jokes.FirstOrDefault();
}
}
}
}
Usage is e.g.
GetJoke().ToString();
These links show how to read a web page.
Html Agility Pack. Load and scrape webpage
Get HTML code from website in C#
I am learning to write web crawler and found some great examples to get me started but since I am new to this, I have a few questions in regards to the coding method.
The search result for example can be found here: Search Result
When I look at the HTML source for the result I can see the following:
<HR><CENTER><H3>License Information *</H3></CENTER><HR>
<P>
<CENTER> 06/03/2014 </CENTER> <BR>
<B>Name : </B> WILLIAMS AJAYA L <BR>
<B>Address : </B> NEW YORK NY <BR>
<B>Profession : </B> ATHLETIC TRAINER <BR>
<B>License No: </B> 001475 <BR>
<B>Date of Licensure : </B> 01/12/07 <BR>
<B>Additional Qualification : </B> Not applicable in this profession <BR>
<B> Status :</B> REGISTERED <BR>
<B>Registered through last day of : </B> 08/15 <BR>
How can I use the HTMLAgilityPack to scrap those data from the site?
I was trying to implement an example as shown below, but not sure where to make the edit to get it working to crawl the page:
private void btnCrawl_Click(object sender, EventArgs e)
{
foreach (SHDocVw.InternetExplorer ie in shellWindows)
{
filename = Path.GetFileNameWithoutExtension( ie.FullName ).ToLower();
if ( filename.Equals( "iexplore" ) )
txtURL.Text = "Now Crawling: " + ie.LocationURL.ToString();
}
string url = ie.LocationURL.ToString();
string xmlns = "{http://www.w3.org/1999/xhtml}";
Crawler cl = new Crawler(url);
XDocument xdoc = cl.GetXDocument();
var res = from item in xdoc.Descendants(xmlns + "div")
where item.Attribute("class") != null && item.Attribute("class").Value == "folder-news"
&& item.Element(xmlns + "a") != null
//select item;
select new
{
Link = item.Element(xmlns + "a").Attribute("href").Value,
Image = item.Element(xmlns + "a").Element(xmlns + "img").Attribute("src").Value,
Title = item.Elements(xmlns + "p").ElementAt(0).Element(xmlns + "a").Value,
Desc = item.Elements(xmlns + "p").ElementAt(1).Value
};
foreach (var node in res)
{
MessageBox.Show(node.ToString());
tb.Text = node + "\n";
}
//Console.ReadKey();
}
The Crawler helper class:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Xml.Linq;
namespace CrawlerWeb
{
public class Crawler
{
public string Url
{
get;
set;
}
public Crawler() { }
public Crawler(string Url)
{
this.Url = Url;
}
public XDocument GetXDocument()
{
HtmlAgilityPack.HtmlWeb doc1 = new HtmlAgilityPack.HtmlWeb();
doc1.UserAgent = "Mozilla/4.0 (conpatible; MSIE 7.0; Windows NT 5.1)";
HtmlAgilityPack.HtmlDocument doc2 = doc1.Load(Url);
doc2.OptionOutputAsXml = true;
doc2.OptionAutoCloseOnEnd = true;
doc2.OptionDefaultStreamEncoding = System.Text.Encoding.UTF8;
XDocument xdoc = XDocument.Parse(doc2.DocumentNode.SelectSingleNode("html").OuterHtml);
return xdoc;
}
}
}
tb is a multiline textbox... So I would like it to display the following:
Name WILLIAMS AJAYA L
Address NEW YORK NY
Profession ATHLETIC TRAINER
License No 001475
Date of Licensure 1/12/07
Additional Qualification Not applicable in this profession
Status REGISTERED
Registered through last day of 08/15
I would like the second argument to be added to an array because next step would be to write to a SQL database...
I am able to get the URL from the IE which has the search result but how can I code it in my script?
This little snippet should get you started:
HtmlDocument doc = new HtmlDocument();
WebClient client = new WebClient();
string html = client.DownloadString("http://www.nysed.gov/coms/op001/opsc2a?profcd=67&plicno=001475&namechk=WIL");
doc.LoadHtml(html);
HtmlNodeCollection nodes = doc.DocumentNode.SelectNodes("//div");
You basically use the WebClient class to download the HTML file and then you load that HTML into the HtmlDocument object. Then you need to use XPath to query the DOM tree and search for nodes. In the above example "nodes" will include all the div elements in the document.
Here's a quick reference about the XPath syntax: http://msdn.microsoft.com/en-us/library/ms256086(v=vs.110).aspx