i started develop application. Me need take some informations from website after download it in database, and after I need treatment this informations.
Well, I don't have enough experience and be grateful any of your recomendation.
for example - I'll work with sport site. (https://terrikon.com/football/spain/championship/)
I need receive informations from table and download this data in DB.
I tried some ways download data and understand that best way - use "htmlagilitypack".
I read documentation about work with this library and best that i did:
using System;
using System.Xml;
using HtmlAgilityPack;
namespace Parser
{
class Program
{
static void Main(string[] args)
{
var html = #"https://terrikon.com/football/spain/championship/";
HtmlWeb web = new HtmlWeb();
var htmlDoc = web.Load(html);
var node = htmlDoc.DocumentNode.SelectSingleNode("//head/title");
var table = htmlDoc.QuerySelector("#champs-table > table");
var tableRows = table.QuerySelectorAll("tr");
foreach (var row in tableRows)
{
var team = row.QuerySelector(".team");
var win = row.QuerySelector(".win");
var draw = row.QuerySelector(".draw");
var lose = row.QuerySelector(".lose");
Console.WriteLine(team.OuterHtml );
};
}
}
}
I can receive title of website or all informations, if I'll change this string
var node = htmlDoc.DocumentNode.SelectSingleNode("//head");
Could you give me advice how can I get informations only from table?
thanks for tour attention
I would recommend to also install a Css selector extension for HtmlAgilityPack which can be found here:
https://github.com/hcesar/HtmlAgilityPack.CssSelector
With that you can Query your nodes with css selectors.
To get the information from that table you will have to know the CSS selector for it.
In this case its:
#champs-table > table
So to get whole table you could do it like this:
var table = htmlDoc.QuerySelector("#champs-table > table");
// then query rows of that table:
var tableRows = table.QuerySelectorAll("tr");
// Now each element it tableRows is a <tr> from that html table
// you could access every value in a for each loop
foreach(var row in tableRows)
{
var team = row.QuerySelector(".team"); // "team" is a css class applied to <td> containing the team name
var win = row.QuerySelector(".win");
var draw = row.QuerySelector(".draw");
var lose = row.QuerySelector(".lose");
}
Related
Im trying to get stock data from a website with webcrawler as a hobby project. I got the link to work, i got the Name of the stock but i cant get the price... I dont know how to handle the html code. Here is my code,
var htmlDocument = new HtmlDocument();
htmlDocument.LoadHtml(html);
var divs = htmlDocument.DocumentNode.Descendants("div").Where(n => n.GetAttributeValue("class", "").Equals("Flexbox__StyledFlexbox-sc-1ob4g1e-0 eYavUv Row__StyledRow-sc-1iamenj-0 foFHXj Rows__AlignedRow-sc-1udgki9-0 dnLFDN")).ToList();
var stocks = new List<Stock>();
foreach (var div in divs)
{
var stock = new Stock()
{
Name = div.Descendants("a").Where(a=>a.GetAttributeValue("class","").Equals("Link__StyledLink-sc-apj04t-0 foCaAq NameCell__StyledLink-sc-qgec4s-0 hZYbiE")).FirstOrDefault().InnerText,
changeInPercent = div.Descendants("span").Where((a)=>a.GetAttributeValue("class", "").Equals("Development__StyledDevelopment-sc-hnn1ri-0 kJLDzW")).FirstOrDefault()?.InnerText
};
stocks.Add(stock);
}
foreach (var stock in stocks)
{
Console.WriteLine(stock.Name + " ");
}
I got the Name correct, but i dont really know how the get the ChangeInPercent.... I will past in the html code below,
The top highlight show where i got the name from, and the second one is the "span" i want. I want the -4.70
Im a litle bit confused when it comes to get the data with my code. I tried everything. My changeInPercent property is a string.
it has to be the code somehow...
There's probably an easier to select a single attribute/node than the way you're doing it right now:
If you know the exact XPath expression to select the node you're looking for, then you can do the following:
var htmlDocument = new HtmlDocument();
htmlDocument.LoadHtml(html);
var changeInPercent = htmlDocument.DocumentNode
.SelectSingleNode("//foo/bar")
.InnerText;
Getting the right XPath expression (the //foo/bar example above) is the tricky part. But this can be found quite easy using your browser's dev tools. You can navigate to the desired element and just copy it's XPath expression - simple as that! See here for a sample on how to copy the expression.
I'm a beginner programmer working on a small webscraper in C#. The purpose is to take a hospital's public website, grab the data for each doctor, their department, phone and diploma info, and display it in a Data Grid View. It's a public website, and as far as I'm concerned, the website's robots.txt allows this, so I left everything in the code as it is.
I am able to grab each data (name, department, phone, diploma) separately, and can successfully display them in a text box.
// THIS WORKS:
string text = "";
foreach (var nodes in full)
{
text += nodes.InnerText + "\r\n";
}
textBox1.Text = text;
However, when I try to pass the data on to the data grid view using a class, the foreach loop only goes through the first name and fills the data grid with that.
foreach (var nodes in full)
{
var Doctor = new Doctor
{
Col1 = full[0].InnerText,
Col2 = full[1].InnerText,
Col3 = full[2].InnerText,
Col4 = full[3].InnerText,
};
Doctors.Add(Doctor);
}
I spent a good few hours looking for solutions but none of what I've found have been working, and I'm at the point where I can't decide if I messed up the foreach loop somehow, or if I'm not doing something according to HTML Agility Pack's rules. It lets me iterate through for the textbox, but not the foreach. Changing full[0] to nodes[0] or nodes.InnerText doesn't seem to solve it either.
link to public gist file (where you can see my whole code)
screenshot
Thank you for the help in advance!
The problem is how you're selecting the nodes from the page. full contains all individual names, departments etc. in a flat list, which means full[0] is the name of the first doctor while full[4] is the name of the next. Your for-loop doesn't take that into account, as you (for every node) always access full[0] to full[3] - so, only the properties of the first doctor.
To make your code more readable I'd split it up a bit to first make a list of all the card-elements for each doctor and then select the individual parts within the loop:
HtmlWeb web = new HtmlWeb();
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc = web.Load("https://klinikaikozpont.unideb.hu/doctor_finder");
const string doctorListItem = "div[contains(#class, 'doctor-list-item-model')]";
const string cardContent = "div[contains(#class, 'card-content')]";
var doctorCards = doc.DocumentNode.SelectNodes($"//{doctorListItem}/{cardContent}");
var doctors = new List<Doctor>();
foreach (var card in doctorCards)
{
var name = card.SelectSingleNode("./h3")?.InnerText;
const string departmentNode = "div[contains(#class, 'department-name')]";
var department = card.SelectSingleNode($"./{departmentNode}/p")?.InnerText;
// other proprties...
doctors.Add(new Doctor{NameAndTitle = name, Department = department});
}
// I took the liberty to make this class easier to understand
public class Doctor
{
public string NameAndTitle { get; set; }
public string Department { get; set; }
// Add other properties
}
Check out the code in action.
I am putting together a map of all the inline styles on elements in a large project. I would like to show the line number where they are located similar the example below.
Is it possible to get the line number of an element in AngleSharp?
foreach (var file in allFiles)
{
string source = File.ReadAllText(file.FullName);
var parser = new HtmlParser();
var doc = parser.ParseDocument(source);
var items = doc.QuerySelectorAll("*[style]");
sb.AppendLine($"{file.Name} - inline styles({items.Count()})");
foreach (var item in items)
{
sb.AppendLine($"\t\tstyle (**{item.LineNumber}**): {item.GetAttribute("style")}");
}
}
Yes this is possible.
Quick example:
var parser = new HtmlParser(new HtmlParserOptions
{
IsKeepingSourceReferences = true,
});
var document = parser.ParseDocument("<html><head></head><body>&foo</body></html>");
document.QuerySelector("body").SourceReference?.Position.Dump();
The output looks as follows:
The important part is to use the IsKeepingSourceReferences option, as this will allow you to use SourceReference. Some (by the parser / spec inserted) elements may not have a source reference, so keep in mind that this may be null.
so when I'm trying to do is parse a HTML document using Html Agility Pack. I load the html doc and it works. The issue lies when I try to parse it using XPath. I get a "System.NullReferenceException: 'Object reference not set to an instance of an object.'" Error.
To get my xpath I use the Chrome Development window and highlight the whole table that has the rows which contains the data that I want to parse, right click it and copy Xpath.
Here's my code
string url = "https://www.ctbiglist.com/index.asp";
string myPara = "LastName=Smith&FirstName=James&PropertyID=&Submit=Search+Properties";
string htmlResult;
// Get the raw HTML from the website
using (WebClient client = new WebClient())
{
client.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
// Send in the link along with the FirstName, LastName, and Submit POST request
htmlResult = client.UploadString(url, myPara);
//Console.WriteLine(htmlResult);
}
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(htmlResult);
HtmlNodeCollection table = doc.DocumentNode.SelectNodes("//*[#id=\"Table2\"]/tbody/tr[2]/td/table/tbody/tr/td/div[2]/table/tbody/tr[2]/td/table/tbody/tr[2]/td/form/div/table[1]/tbody/tr");
Console.WriteLine(table.Count);
When I run this code it works but grabs all the tables in the HTML document.
var query = from table in doc.DocumentNode.SelectNodes("//table").Cast<HtmlNode>()
from row in table.SelectNodes("//tr").Cast<HtmlNode>()
from cell in row.SelectNodes("//th|td").Cast<HtmlNode>()
select new { Table = table.Id, CellText = cell.InnerText };
foreach (var cell in query)
{
Console.WriteLine("{0}: {1}", cell.Table, cell.CellText);
}
What I want is a specific table that holds all the tables rows that has the data I want to parse into objects.
Thanks for the help!!!
Change the line
from table in doc.DocumentNode.SelectNodes("//table").Cast<HtmlNode>()
to
from table in doc.DocumentNode.SelectNodes("//table[#id=\"Table2\"]").Cast<HtmlNode()
This will only select specific table with given Id. But if you have nested Tables then you have change your xpath accordingly to get the nested table rows.
I want to use the HTML agility pack to parse tables from complex web pages, but I am somehow lost in the object model.
I looked at the link example, but did not find any table data this way.
Can I use XPath to get the tables? I am basically lost after having loaded the data as to how to get the tables. I have done this in Perl before and it was a bit clumsy, but worked. (HTML::TableParser).
I am also happy if one can just shed a light on the right object order for the parsing.
How about something like:
Using HTML Agility Pack
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(#"<html><body><p><table id=""foo""><tr><th>hello</th></tr><tr><td>world</td></tr></table></body></html>");
foreach (HtmlNode table in doc.DocumentNode.SelectNodes("//table")) {
Console.WriteLine("Found: " + table.Id);
foreach (HtmlNode row in table.SelectNodes("tr")) {
Console.WriteLine("row");
foreach (HtmlNode cell in row.SelectNodes("th|td")) {
Console.WriteLine("cell: " + cell.InnerText);
}
}
}
Note that you can make it prettier with LINQ-to-Objects if you want:
var query = from table in doc.DocumentNode.SelectNodes("//table").Cast<HtmlNode>()
from row in table.SelectNodes("tr").Cast<HtmlNode>()
from cell in row.SelectNodes("th|td").Cast<HtmlNode>()
select new {Table = table.Id, CellText = cell.InnerText};
foreach(var cell in query) {
Console.WriteLine("{0}: {1}", cell.Table, cell.CellText);
}
The most simple what I've found to get the XPath for a particular Element is to install FireBug extension for Firefox go to the site/webpage press F12 to bring up firebug; right select and right click the element on the page that you want to query and select "Inspect Element" Firebug will select the element in its IDE then right click the Element in Firebug and choose "Copy XPath" this function will give you the exact XPath Query you need to get the element you want using HTML Agility Library.
I know this is a pretty old question but this was my solution that helped with visualizing the table so you can create a class structure. This is also using the HTML Agility Pack
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(#"<html><body><p><table id=""foo""><tr><th>hello</th></tr><tr><td>world</td></tr></table></body></html>");
var table = doc.DocumentNode.SelectSingleNode("//table");
var tableRows = table.SelectNodes("tr");
var columns = tableRows[0].SelectNodes("th/text()");
for (int i = 1; i < tableRows.Count; i++)
{
for (int e = 0; e < columns.Count; e++)
{
var value = tableRows[i].SelectSingleNode($"td[{e + 1}]");
Console.Write(columns[e].InnerText + ":" + value.InnerText);
}
Console.WriteLine();
}
In my case, there is a single table which happens to be a device list from a router. If you wish to read the table using TR/TH/TD (row, header, data) instead of a matrix as mentioned above, you can do something like the following:
List<TableRow> deviceTable = (from table in document.DocumentNode.SelectNodes(XPathQueries.SELECT_TABLE)
from row in table?.SelectNodes(HtmlBody.TR)
let rows = row.SelectSingleNode(HtmlBody.TR)
where row.FirstChild.OriginalName != null && row.FirstChild.OriginalName.Equals(HtmlBody.T_HEADER)
select new TableRow
{
Header = row.SelectSingleNode(HtmlBody.T_HEADER)?.InnerText,
Data = row.SelectSingleNode(HtmlBody.T_DATA)?.InnerText}).ToList();
}
TableRow is just a simple object with Header and Data as properties.
The approach takes care of null-ness and this case:
<tr>
<td width="28%"> </td>
</tr>
which is row without a header. The HtmlBody object with the constants hanging off of it are probably readily deduced but I apologize for it even still. I came from the world where if you have " in your code, it should either be constant or localizable.
Line from above answer:
HtmlDocument doc = new HtmlDocument();
This doesn't work in VS 2015 C#. You cannot construct an HtmlDocument any more.
Another MS "feature" that makes things more difficult to use. Try HtmlAgilityPack.HtmlWeb and check out this link for some sample code.