Develop Reference

Remove duplicate rel attributes in HTML using c#

I have an HTML document with a number of links. Some of these links contain two identical rel attributes, and I would like to iterate over all of the a-tags, check if there are more than one rel attribute and remove one if there is, but I can't figure out how to do it.
Example:
<a rel="nofollow" rel="nofollow" href="blah"> link<a>
Should be:
<a rel="nofollow" href="blah"> link<a>
Thanks for any help.

If are you attributes are identical, this is a possible solution using nuget HtmlAgilityPack
using System.IO;
using System.Text;
using System.Linq;
using HtmlAgilityPack;
public class Program
{
public static void Main()
{
var html =
#"
<div>
<a rel='nofollow' rel='nofollow' href='lah'> link1</a>
<a rel='qwerty' rel='qwerty' href='lah'> link2</a>
<a rel='asdf' rel='asdf' href='lah'> link3</a>
</div>";
var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(html);
var htmlNodes = htmlDoc.DocumentNode.SelectNodes("//a");
foreach (var node in htmlNodes)
{
var attr = node.Attributes["rel"];
while(node.Attributes.Contains("rel")) node.Attributes.Remove("rel");
node.Attributes.Add("rel", attr.Value);
}
using var ms = new MemoryStream();
htmlDoc.Save(ms, Encoding.UTF8);
var result = Encoding.UTF8.GetString(ms.ToArray());
System.Console.WriteLine(result);
}
}

c# crawling rule didn't work to cnn web site

I'm beginner to C# crawling
I have tried to crawl CNN headlines news from (https://edition.cnn.com/)
But I have failed to get the head line texts.
target is looks like below html (sorry I'm not good at asking questions containing source code, newbie T.T)
<div class="cd__wrapper" data-analytics="_list-hierarchical-xs_article_">
<div class="cd__content">
<h3 class="cd__headline" data-analytics="_list-hierarchical-xs_article_">
<a href="/travel/article/cruise-ship-passengers-stranded-coronavirus/index.html">
<span class="cd__headline-text">At least 30 cruise ships are at sea. Here's what it's like on board.</span><span class="cd__headline-icon cnn-icon"></span></a></h3></div></div>
First I tried to crawl to all html codes
then convert to string
(my target is get head line text with href link for crawling child pages)
with below c# codes
public async void GetCnnAsync()
{
var url = "https://edition.cnn.com/";
var httpClient = new HttpClient();
var html = await httpClient.GetStringAsync(url);
var htmlDocument = new Hp.HtmlDocument();
htmlDocument.LoadHtml(html);
var headLineHtmlList = htmlDocument.DocumentNode.Descendants("div")
.Where(node => node.GetAttributeValue("class", "")
.Contains("cd__headline")).ToList();
Console.ReadLine();
}
but It didn't work just get null headLineHtmlList
I don't know why I failed to get result. because chrome page inspecter source have that elements
On the other hand when I tried it to stackoverflow site.
I was able to get question list with below codes
public async void GetHtmlAsync()
{
var url = "https://stackoverflow.com/questions";
var httpClient = new HttpClient();
var html = await httpClient.GetStringAsync(url);
var htmlDocument = new Hp.HtmlDocument();
htmlDocument.LoadHtml(html);
var questionsHtml = htmlDocument.DocumentNode.Descendants("div")
.Where(node => node.GetAttributeValue("id", "")
.Equals("questions")).ToList();
var questionList = questionsHtml[0].Descendants("div")
.Where(node => node.GetAttributeValue("id", "")
.Contains("question-summary")).ToList();
}
It was able to get question list.
Now I really really want to get result from CNN website
please help me
Thanks in advance
add more test codes
create WebBrowser control
then navigate then get WebBrowser_DocumentCompleted callback
but I didn't get result again
so, I tried it again with documentCompleted
but I didn't get it
WebBrowser webBrowser;
Control parent;
WebNewsCallback newsCallback;
public WebNewsCrawler(Control parent, WebNewsCallback newsCallback) {
this.parent = parent;
this.newsCallback = newsCallback;
if (webBrowser == null) {
webBrowser = new WebBrowser {
Visible = false,
ScriptErrorsSuppressed = true
};
}
parent.Controls.Add(webBrowser);
webBrowser.DocumentCompleted += WebBrowser_DocumentCompleted;
}
public void doWork(string address) {
webBrowser.Navigate(address);
}
int count = 0;
private void WebBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e) {
if (webBrowser.ReadyState != WebBrowserReadyState.Complete) return;
newsCallback(webBrowser.DocumentStream);
GetCnn(webBrowser.DocumentStream);
Console.WriteLine(count.ToString());
count++;
}
public void GetCnn(Stream stream) {
var doc = new Hp.HtmlDocument();
doc.Load(stream, Encoding.UTF8);
var nodes = doc.DocumentNode.SelectNodes("/html/body/div[7]/section[2]/div[2]/div/div[1]/ul/li[4]/article/div/div/h3/a/span[1]");
if(nodes != null) {
Console.WriteLine("xpath nodes not null");
}
var headLineHtmlList = doc.DocumentNode.Descendants("h3").ToList();
if (headLineHtmlList != null) {
Console.WriteLine("headLineCount " +headLineHtmlList.Count.ToString());
}
}
headLineCount is 0 and xPath result is zero(xpath or xpath full path same result)

Are you sure that the selector you're using is correct? You said:
var headLineHtmlList = htmlDocument.DocumentNode.Descendants("div")
.Where(node => node.GetAttributeValue("class", "")
.Contains("cd__headline")).ToList();
Isn't that saying "Give me all the descendants with tag <div> and a CSS class of cd__headline"?
But you're not looking for a div with a class of cd__headline. You're looking for <a> tags occurring inside <h3> tags that have a CSS class of cd__headline.
I could be wrong, but if I'm right it would be an easy fix! Good luck.

Unable to get html element by using X-Path in HtmlAgilityPack C#

I am trying to get element by using x-path tree element but showing null, and this type of x-path work for other site for me, only 2% site this types of X-Path not working, also i tried x-path from chrome also but when my x-path not work that time chrome x-path also not work.
public static void Main()
{
string url = "http://www.ndrf.gov.in/tender";
HtmlWeb web = new HtmlWeb();
var htmlDoc = web.Load(url);
var nodetest1 = htmlDoc.DocumentNode.SelectSingleNode("/html[1]/body[1]/section[2]/div[1]/div[1]/div[1]/div[1]/div[2]/table[1]"); // i want this type // not wroking
//var nodetest2 = htmlDoc.DocumentNode.SelectSingleNode("//*[#id=\"content\"]/div/div[1]/div[2]/table"); // from Google chrome // not wroking
//var nodetest3 = htmlDoc.DocumentNode.SelectSingleNode("//*[#id=\"content\"]"); // by ID but i don't want this type // wroking
Console.WriteLine(nodetest1.InnerText); //fail
//Console.WriteLine(nodetest2.InnerText); //fail
//Console.WriteLine(nodetest3.InnerText); //proper but I don't want this type
}

The answer that #QHarr suggested works perfectly, But the reason you get null with a correct x-path, is that there is a javascript file in the header of the site, that adds a wrapper div around the table, and since getting result in HtmlAgilityPack seems not loading or executing js, the x-path returns null.
what you observe, after that js runs is:
<div class="view-content">
<div class="guide-text">
...
</div>
<div class="scroll-table1">
<!-- Your table is here -->
</div>
</div>
but what actually you get whithout that js, is:
<div class="view-content">
<!-- Your table is here -->
</div>
thus your x-path should be:
var nodetest1 = htmlDoc.DocumentNode.SelectSingleNode("/html[1]/body[1]/section[2]/div[1]/div[1]/div[1]/div[1]/table[1]");

Your xpath when used in browser selects for entire table. You can shorten and use as follows (fiddle):
using System;
using HtmlAgilityPack;
public class Program
{
public static void Main()
{
string url = "http://www.ndrf.gov.in/tender";
HtmlWeb web = new HtmlWeb();
var htmlDoc = web.Load(url);
var nodetest1 = htmlDoc.DocumentNode.SelectSingleNode("//table");
Console.WriteLine(nodetest1.InnerText);
}
}

Use Fizzler.Systems.HtmlAgilityPack
details here : https://www.nuget.org/packages/Fizzler.Systems.HtmlAgilityPack/
This library adds extension methods called QuerySelector and QuerySelectorAll, that takes CSS Selector not XPath.

Ali Bordbar caught perfect, This Url adds a wrapper div when I navigating URL in WebBrowser control in this all JavaScript file are loaded,
but when i load URL using HtmlWeb there is none of the JavaScript file loaded.
The HtmlWeb retrieves the static HTML response that the server sends, and does not execute any javascript, whereas a WebBrowser would.
So WebBrowser control HTML DOM data XPath and HtmlWeb HTML DOM data XPath not match.
My below code work perfect for this switchvation
HtmlWeb web = new HtmlWeb();
web.AutoDetectEncoding = true;
HtmlAgilityPack.HtmlDocument theDoc1 = web.Load("http://www.ndrf.gov.in/tender");
var HtmlDoc = new HtmlAgilityPack.HtmlDocument();
var bodytag = theDoc1.DocumentNode.SelectSingleNode("//html");
HtmlDoc.LoadHtml(bodytag.OuterHtml);
var xpathHtmldata = HtmlDoc.DocumentNode.SelectSingleNode(savexpath); //savexpath is my first xpath make from HTML DOM data of WebBrowser control which is work for most url.
if (xpathHtmldata == null)
{
//take last tag name from first xpath
string mainele = savexpath.Substring(savexpath.LastIndexOf("/") + 1);
if (mainele.Contains("[")) { mainele = mainele.Remove(mainele.IndexOf("[")); }
//collect all tag name with name of which is sotre in mainele variable
var taglist = HtmlDoc.DocumentNode.SelectNodes("//" + mainele);
foreach (var ele in taglist) //check one by one element
{
string htmltext1 = ele.InnerText;
htmltext1 = Regex.Replace(htmltext1, #"\s", "");
htmltext1 = htmltext1.Replace("&", "&").Trim();
htmltext1 = htmltext1.Replace(" ", "").Trim();
string htmltext2 = saveInnerText; // my previus xpath text from HTML DOM data of WebBrowser control
htmltext2 = Regex.Replace(htmltext2, #"\s", "");
if (htmltext1 == htmltext2) // check equality to my previus xpath text..if it is equal thats my new xpath
{
savexpath = ele.XPath;
break;
}
}
}

Can't get innertext html c#

Ok. So I found this code online everything is working on it but it shows me the div class I am searching for but removes all the text. Any idea why? Heres a example of what its outputting...
<div class="marquee"><img src="logo.png" /></div>
<div id="joke">
<div id="setup" class="exit-left"></div>
<div id="punchline">
<div class="question"></div>
<div id="zing" class="exit-right"></div>
</div>
</div>
<div id="buttons">
<input id="tell-me" class="exit-bottom no-select" type="button" value="Tell Me!" />
<!--<input id="another" class="exit-bottom" type="button" value="Another!" />-->
<table class="another exit-bottom no-select">
<tr>
<td class="another" colspan="3">Another</td>
<td class="share"><div class="share-img"></div>Share</td>
</tr>
</table>
</div>
And the innertext is not shown at all...
And here is my code is VS.
var doc = new HtmlAgilityPack.HtmlDocument();
HtmlAgilityPack.HtmlNode.ElementsFlags["br"] = HtmlAgilityPack.HtmlElementFlag.Empty;
doc.OptionWriteEmptyNodes = true;
try
{
var webRequest = HttpWebRequest.Create("http://dadjokegenerator.com/");
Stream stream = webRequest.GetResponse().GetResponseStream();
doc.Load(stream);
stream.Close();
}
catch (System.UriFormatException uex)
{
throw;
}
catch (System.Net.WebException wex)
{
throw;
}
//get the div by id and then get the inner text
doc.GetElementbyId("content").InnerHtml;
await e.Channel.SendMessage("test " + divString); `

Although your code correctly downloads content of page http://dadjokegenerator.com/, InnerHtml is empty, because this page actually doesn't contain joke you are looking for (you can see that, if you display source code of page in you web browser - e.g. in Firefox press CTRL+U). Joke is added to this page later by javascript. If you look at source code of this Javascript at http://dadjokegenerator.com/js/main.js, you can see that individual jokes are downloaded from URL http://dadjokegenerator.com/api/api.php?a=j&lt=r&vj=0
Here is minimal sample to download joke from this URL. I ommited all error-checks for simplicity and I used free Json.NET library for JSON deserialization:
public class Joke
{
public int Id;
public string Setup;
public string Punchline;
public override string ToString()
{
return Setup + " " + Punchline;
}
}
public static Joke GetJoke()
{
var request = HttpWebRequest.Create("http://dadjokegenerator.com/api/api.php?a=j&lt=r&vj=0");
using (var response = request.GetResponse())
{
using (var stream = response.GetResponseStream())
{
using (var reader = new StreamReader(stream))
{
var jokeString = reader.ReadToEnd();
Joke[] jokes = JsonConvert.DeserializeObject<Joke[]>(jokeString);
return jokes.FirstOrDefault();
}
}
}
}
Usage is e.g.
GetJoke().ToString();

These links show how to read a web page.
Html Agility Pack. Load and scrape webpage
Get HTML code from website in C#

How can i loop over a string and get the links between href that end with jpg?

I'm navigating to a web site using webBrowser in the completed event i did:
void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
mshtml.HTMLDocument objHtmlDoc = (mshtml.HTMLDocument)webBrowser1.Document.DomDocument;
string pageSource = objHtmlDoc.documentElement.innerHTML;
}
Now in the pageSource i have the whole page source.
I tried to make
string[] lines = File.ReadAllLines(pageSource);
But it give me exception:
Illegal characters in path
Then i tried this line:
var aContents = Regex.Matches(pageSource, #"<a [^>]*>(.*?)</a>").Cast<Match>().Select(m => m.Groups[1].Value);
But i there are no href lines in the aContents

Use htmlagilitypack http://html-agility-pack.net
and you can use the library method to load from url - and then check the node to see if it contains the ext and store it in a collection.
List<string> alljpgHref = new List<string>;
HtmlWeb hw = new HtmlWeb();
HtmlDocument doc = hw.Load(/* url */);
foreach(HtmlNode link in doc.DocumentNode.SelectNodes("//a[#href]"))
{
string hrefValue = link.GetAttributeValue( "href", string.Empty );
if (hrefValue.contains(".jpg")) alljpgHref.add(hrefValue);
}

or just query the links:
string[] hrefs = this.webBrowser1.Document.Links.Cast<HtmlElement>()
.Select(a => a.GetAttribute("href")).Where(h => h.Contains(".jpg")).ToArray();