Hello i am pretty new in c# sphere. I want to make a little program that will fetch data from the given page.
It is a fragment of website:
<h3 class="filmInfo__header cloneToCast cloneToOtherInfo" data-type="directing-header">reżyseria</h3>
<div class="filmInfo__info cloneToCast cloneToOtherInfo" data-type="directing-info"> <span itemprop="url" content="/person/Rupert+Sanders-1121101"></span> <span itemprop="name">Rupert Sanders</span> </div>
I want to get data from "Data-type="Directing-info" and get a result from title="Rupert Sanders"
Somebody can help me ?
My very simple code:
private void button1_Click(object sender, EventArgs e)
{
var url = "https://www.filmweb.pl/film/Kr%C3%B3lewna+%C5%9Anie%C5%BCka+i+%C5%81owca-2012-600541";
var httpClient = new HttpClient();
var html = httpClient.GetStringAsync(url);
textBox1.Text = (html.Result);
}
C# or .NET does not offer native HTML parsing functionality. However, there are a handful of libraries which provides HTML parsing functionality. For example, you can use Html Agility Pack.
First, you need to install it into your project. You can easily install it with NuGet Package Manager in Visual Studio if you use it.
After that, you can use it like this with your input HTML:
private void button1_Click(object sender, EventArgs e)
{
var url = "https://www.filmweb.pl/film/Kr%C3%B3lewna+%C5%9Anie%C5%BCka+i+%C5%81owca-2012-600541";
var httpClient = new HttpClient();
var html = httpClient.GetStringAsync(url);
// Create a HtmlDocument and load your HTML into it.
var htmlDocument = new HtmlDocument();
htmlDocument.LoadHtml(html.Result);
// Find your desired node inside it.
HtmlNode directingInfoNode = htmlDocument.DocumentNode.SelectSingleNode("//div[#data-type='directing-info']/a");
// Get the title attribute of that node.
HtmlAttribute titleAttribute = directingInfoNode.Attributes["title"];
textBox1.Text = (titleAttribute.Value);
}
Of course, you will need to put necessary using statement to the top of your file:
using HtmlAgilityPack;
Related
I'm beginner to C# crawling
I have tried to crawl CNN headlines news from (https://edition.cnn.com/)
But I have failed to get the head line texts.
target is looks like below html (sorry I'm not good at asking questions containing source code, newbie T.T)
<div class="cd__wrapper" data-analytics="_list-hierarchical-xs_article_">
<div class="cd__content">
<h3 class="cd__headline" data-analytics="_list-hierarchical-xs_article_">
<a href="/travel/article/cruise-ship-passengers-stranded-coronavirus/index.html">
<span class="cd__headline-text">At least 30 cruise ships are at sea. Here's what it's like on board.</span><span class="cd__headline-icon cnn-icon"></span></a></h3></div></div>
First I tried to crawl to all html codes
then convert to string
(my target is get head line text with href link for crawling child pages)
with below c# codes
public async void GetCnnAsync()
{
var url = "https://edition.cnn.com/";
var httpClient = new HttpClient();
var html = await httpClient.GetStringAsync(url);
var htmlDocument = new Hp.HtmlDocument();
htmlDocument.LoadHtml(html);
var headLineHtmlList = htmlDocument.DocumentNode.Descendants("div")
.Where(node => node.GetAttributeValue("class", "")
.Contains("cd__headline")).ToList();
Console.ReadLine();
}
but It didn't work just get null headLineHtmlList
I don't know why I failed to get result. because chrome page inspecter source have that elements
On the other hand when I tried it to stackoverflow site.
I was able to get question list with below codes
public async void GetHtmlAsync()
{
var url = "https://stackoverflow.com/questions";
var httpClient = new HttpClient();
var html = await httpClient.GetStringAsync(url);
var htmlDocument = new Hp.HtmlDocument();
htmlDocument.LoadHtml(html);
var questionsHtml = htmlDocument.DocumentNode.Descendants("div")
.Where(node => node.GetAttributeValue("id", "")
.Equals("questions")).ToList();
var questionList = questionsHtml[0].Descendants("div")
.Where(node => node.GetAttributeValue("id", "")
.Contains("question-summary")).ToList();
}
It was able to get question list.
Now I really really want to get result from CNN website
please help me
Thanks in advance
add more test codes
create WebBrowser control
then navigate then get WebBrowser_DocumentCompleted callback
but I didn't get result again
so, I tried it again with documentCompleted
but I didn't get it
WebBrowser webBrowser;
Control parent;
WebNewsCallback newsCallback;
public WebNewsCrawler(Control parent, WebNewsCallback newsCallback) {
this.parent = parent;
this.newsCallback = newsCallback;
if (webBrowser == null) {
webBrowser = new WebBrowser {
Visible = false,
ScriptErrorsSuppressed = true
};
}
parent.Controls.Add(webBrowser);
webBrowser.DocumentCompleted += WebBrowser_DocumentCompleted;
}
public void doWork(string address) {
webBrowser.Navigate(address);
}
int count = 0;
private void WebBrowser_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e) {
if (webBrowser.ReadyState != WebBrowserReadyState.Complete) return;
newsCallback(webBrowser.DocumentStream);
GetCnn(webBrowser.DocumentStream);
Console.WriteLine(count.ToString());
count++;
}
public void GetCnn(Stream stream) {
var doc = new Hp.HtmlDocument();
doc.Load(stream, Encoding.UTF8);
var nodes = doc.DocumentNode.SelectNodes("/html/body/div[7]/section[2]/div[2]/div/div[1]/ul/li[4]/article/div/div/h3/a/span[1]");
if(nodes != null) {
Console.WriteLine("xpath nodes not null");
}
var headLineHtmlList = doc.DocumentNode.Descendants("h3").ToList();
if (headLineHtmlList != null) {
Console.WriteLine("headLineCount " +headLineHtmlList.Count.ToString());
}
}
headLineCount is 0 and xPath result is zero(xpath or xpath full path same result)
Are you sure that the selector you're using is correct? You said:
var headLineHtmlList = htmlDocument.DocumentNode.Descendants("div")
.Where(node => node.GetAttributeValue("class", "")
.Contains("cd__headline")).ToList();
Isn't that saying "Give me all the descendants with tag <div> and a CSS class of cd__headline"?
But you're not looking for a div with a class of cd__headline. You're looking for <a> tags occurring inside <h3> tags that have a CSS class of cd__headline.
I could be wrong, but if I'm right it would be an easy fix! Good luck.
I am trying to get element by using x-path tree element but showing null, and this type of x-path work for other site for me, only 2% site this types of X-Path not working, also i tried x-path from chrome also but when my x-path not work that time chrome x-path also not work.
public static void Main()
{
string url = "http://www.ndrf.gov.in/tender";
HtmlWeb web = new HtmlWeb();
var htmlDoc = web.Load(url);
var nodetest1 = htmlDoc.DocumentNode.SelectSingleNode("/html[1]/body[1]/section[2]/div[1]/div[1]/div[1]/div[1]/div[2]/table[1]"); // i want this type // not wroking
//var nodetest2 = htmlDoc.DocumentNode.SelectSingleNode("//*[#id=\"content\"]/div/div[1]/div[2]/table"); // from Google chrome // not wroking
//var nodetest3 = htmlDoc.DocumentNode.SelectSingleNode("//*[#id=\"content\"]"); // by ID but i don't want this type // wroking
Console.WriteLine(nodetest1.InnerText); //fail
//Console.WriteLine(nodetest2.InnerText); //fail
//Console.WriteLine(nodetest3.InnerText); //proper but I don't want this type
}
The answer that #QHarr suggested works perfectly, But the reason you get null with a correct x-path, is that there is a javascript file in the header of the site, that adds a wrapper div around the table, and since getting result in HtmlAgilityPack seems not loading or executing js, the x-path returns null.
what you observe, after that js runs is:
<div class="view-content">
<div class="guide-text">
...
</div>
<div class="scroll-table1">
<!-- Your table is here -->
</div>
</div>
but what actually you get whithout that js, is:
<div class="view-content">
<!-- Your table is here -->
</div>
thus your x-path should be:
var nodetest1 = htmlDoc.DocumentNode.SelectSingleNode("/html[1]/body[1]/section[2]/div[1]/div[1]/div[1]/div[1]/table[1]");
Your xpath when used in browser selects for entire table. You can shorten and use as follows (fiddle):
using System;
using HtmlAgilityPack;
public class Program
{
public static void Main()
{
string url = "http://www.ndrf.gov.in/tender";
HtmlWeb web = new HtmlWeb();
var htmlDoc = web.Load(url);
var nodetest1 = htmlDoc.DocumentNode.SelectSingleNode("//table");
Console.WriteLine(nodetest1.InnerText);
}
}
Use Fizzler.Systems.HtmlAgilityPack
details here : https://www.nuget.org/packages/Fizzler.Systems.HtmlAgilityPack/
This library adds extension methods called QuerySelector and QuerySelectorAll, that takes CSS Selector not XPath.
Ali Bordbar caught perfect, This Url adds a wrapper div when I navigating URL in WebBrowser control in this all JavaScript file are loaded,
but when i load URL using HtmlWeb there is none of the JavaScript file loaded.
The HtmlWeb retrieves the static HTML response that the server sends, and does not execute any javascript, whereas a WebBrowser would.
So WebBrowser control HTML DOM data XPath and HtmlWeb HTML DOM data XPath not match.
My below code work perfect for this switchvation
HtmlWeb web = new HtmlWeb();
web.AutoDetectEncoding = true;
HtmlAgilityPack.HtmlDocument theDoc1 = web.Load("http://www.ndrf.gov.in/tender");
var HtmlDoc = new HtmlAgilityPack.HtmlDocument();
var bodytag = theDoc1.DocumentNode.SelectSingleNode("//html");
HtmlDoc.LoadHtml(bodytag.OuterHtml);
var xpathHtmldata = HtmlDoc.DocumentNode.SelectSingleNode(savexpath); //savexpath is my first xpath make from HTML DOM data of WebBrowser control which is work for most url.
if (xpathHtmldata == null)
{
//take last tag name from first xpath
string mainele = savexpath.Substring(savexpath.LastIndexOf("/") + 1);
if (mainele.Contains("[")) { mainele = mainele.Remove(mainele.IndexOf("[")); }
//collect all tag name with name of which is sotre in mainele variable
var taglist = HtmlDoc.DocumentNode.SelectNodes("//" + mainele);
foreach (var ele in taglist) //check one by one element
{
string htmltext1 = ele.InnerText;
htmltext1 = Regex.Replace(htmltext1, #"\s", "");
htmltext1 = htmltext1.Replace("&", "&").Trim();
htmltext1 = htmltext1.Replace(" ", "").Trim();
string htmltext2 = saveInnerText; // my previus xpath text from HTML DOM data of WebBrowser control
htmltext2 = Regex.Replace(htmltext2, #"\s", "");
if (htmltext1 == htmltext2) // check equality to my previus xpath text..if it is equal thats my new xpath
{
savexpath = ele.XPath;
break;
}
}
}
I'm try to develop a UWP app that get data from my University website
I'm able to login with WebView and now I'm try to extract any text that show up after login
But I try to extract web content with HtmlAgilityPack but it got redirect to login page instead.
Here is what I did
private async void GetInfo(WebView sender, WebViewDOMContentLoadedEventArgs args)
{
HtmlDocument doc = new HtmlDocument();
HtmlWeb web = new HtmlWeb();
doc = await web.LoadFromWebAsync(sender.Source.AbsoluteUri);
var chunks = new List<string>();
foreach (var item in doc.DocumentNode.DescendantsAndSelf())
{
if (item.NodeType == HtmlNodeType.Text)
{
if (item.InnerText.Trim() != "")
{
chunks.Add(item.InnerText.Trim());
}
}
}
}
As I check with Visual studio, all texts in chunks list is from login page not after login page
You can use InvokeScriptAsync in WebView to get Html. Call the below code after login using WebView
await webView.InvokeScriptAsync("eval", new string[] { "document.documentElement.outerHTML;" });
For more info:
Check out this answer
Sample app for LogIn to StackOverflow
I have been trying to download an public Instagram profile to the fetch stats such as followers and bio. I have been doing this in a c# console application and downloading the HTML using HTML Agility Pack.
Code:
string url = #"https://www.instagram.com/" + Console.ReadLine() + #"/?hl=en";
Console.WriteLine();
HtmlWeb web = new HtmlWeb();
HtmlDocument document = web.Load(url);
document.Save(path1);
When I save it though all I get is a bunch of scripts and a blank screen:
I was wondering how to save the html once all the scripts had run and formed the content
When you retrieve content using a web request, it returns a HTML document which is then rendered by the browser to display the content.
Right now, you're saving the HTML document given to you by the server. Instead of doing this, you need to render it before getting the details. One way to do this is using a web browser control. If you set the URL to the instragram URL, let the rendering engine handle it and once the load event is fired by the control, you can get the rendered HTML output.
From there, you can deserialize as an XmlDocument and identify exactly what details you need to retrieve from the rendered output.
public MainWindow()
{
InitializeComponent();
WB_1.Navigate(#"https://www.instagram.com/" + Console.ReadLine() + #"/?hl=en");
WB_1.LoadCompleted += wb_LoadCompleted;
}
void wb_LoadCompleted(object sender, NavigationEventArgs e)
{
dynamic doc = WB_1.Document;
string htmlText = doc.documentElement.InnerHtml;
}
ANSWER
Thanks for the suggestions on how to download the HTML! I managed to return some instagram information in the end. Here is the code:
//(This was done using HTML Agility Pack)
string url = #"https://www.instagram.com/" + Console.ReadLine() + #"/?hl=en";
HtmlWeb web = new HtmlWeb();
HtmlDocument document = web.Load(url);
var metas = document.DocumentNode.Descendants("meta");
var followers = metas.FirstOrDefault(_ => _.HasProperty("name", "description"));
if (followers == null) { Console.WriteLine("Sorry, Can't Find Profile :("); return; }
var content = followers.Attributes["content"].Value.StopAt('-');
Console.WriteLine(content);
And HasProperty() & StopAt()
public static bool HasProperty(this HtmlNode node, string property, params string[] valueArray)
{
var propertyValue = node.GetAttributeValue(property, "");
var propertyValues = propertyValue.Split(' ');
return valueArray.All(c => propertyValues.Contains(c));
}
public static string StopAt(this string input, char stopAt)
{
int x = input.IndexOf(stopAt);
return input.Substring(0, x);
}
NOTE:
However this is still not the answer I am looking for. I still have a wreck of HTML which is not structred the same as the HTML I recieve when I look at it in Google Chrome. Doing some searching in the HTML I managed to scalvage the content-less html for a meta tag which contains the content. This is okay for this but if I going to continue this method of finding HTML content then it may not be the same :(
I'm trying to get values from a page, the code I'm using is this :
HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("url")
var nodi = doc.DocumentNode.SelectNodes("//*[#id='hotellist_inner']/div");
string code = "";
if (nodi != null){
foreach (var item in nodi){
var prova = item.InnerText;
Console.WriteLine(code);
}
}
It all works fine as I can get most of the informations I need except for the price.
The url is This.
The Xpath of the info I'm trying to access is
//*[#id="hotellist_inner"]/div[1]/div[2]/div[3]/div[2]/table/tbody/tr/td[4]/div/strong/b
But it looks like my code ignores anything in this section of the HTML.
Can anyone tell me what am I doing wrong?
Thanks in advance!