Develop Reference

Html Agility Pack Xpath not working

so when I'm trying to do is parse a HTML document using Html Agility Pack. I load the html doc and it works. The issue lies when I try to parse it using XPath. I get a "System.NullReferenceException: 'Object reference not set to an instance of an object.'" Error.
To get my xpath I use the Chrome Development window and highlight the whole table that has the rows which contains the data that I want to parse, right click it and copy Xpath.
Here's my code
string url = "https://www.ctbiglist.com/index.asp";
string myPara = "LastName=Smith&FirstName=James&PropertyID=&Submit=Search+Properties";
string htmlResult;
// Get the raw HTML from the website
using (WebClient client = new WebClient())
{
client.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
// Send in the link along with the FirstName, LastName, and Submit POST request
htmlResult = client.UploadString(url, myPara);
//Console.WriteLine(htmlResult);
}
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(htmlResult);
HtmlNodeCollection table = doc.DocumentNode.SelectNodes("//*[#id=\"Table2\"]/tbody/tr[2]/td/table/tbody/tr/td/div[2]/table/tbody/tr[2]/td/table/tbody/tr[2]/td/form/div/table[1]/tbody/tr");
Console.WriteLine(table.Count);
When I run this code it works but grabs all the tables in the HTML document.
var query = from table in doc.DocumentNode.SelectNodes("//table").Cast<HtmlNode>()
from row in table.SelectNodes("//tr").Cast<HtmlNode>()
from cell in row.SelectNodes("//th|td").Cast<HtmlNode>()
select new { Table = table.Id, CellText = cell.InnerText };
foreach (var cell in query)
{
Console.WriteLine("{0}: {1}", cell.Table, cell.CellText);
}
What I want is a specific table that holds all the tables rows that has the data I want to parse into objects.
Thanks for the help!!!

Change the line
from table in doc.DocumentNode.SelectNodes("//table").Cast<HtmlNode>()
to
from table in doc.DocumentNode.SelectNodes("//table[#id=\"Table2\"]").Cast<HtmlNode()
This will only select specific table with given Id. But if you have nested Tables then you have change your xpath accordingly to get the nested table rows.

FileHelpers - FieldOptional doesn't seem to work when using ReadStringAsDT

I'm using DelimitedFieldBuilder and marked my fields as FieldOptional = true.
I later create my class builder like so:
classBuilder = ClassBuilder.LoadFromXmlString(ColumnMappings);
I create my engine:
engine = new FileHelperEngine((classBuilder as DelimitedClassBuilder).CreateRecordClass());
I then populate the dataset like so:
var myString = "01,122242843,456183,160823,0716,84,80,1\n02,456183,122242843,1,160822,,USD,1/\n03,008066662,USD,010,0,,,015\n88,125,,450,1134403,,,570/";
using(var dt = engine.ReadStringAsDT(myString))
{
using (var ds = new DataSet())
{
ds.Tables.Add(dt);
var myXml = ds.GetXml());
}
}
This seemed pretty straight forward. However the fields that lack values are still outputted by the ds.GetXml() call like so:
<NewDataSet>
<Table1>
<ID>01</ID>
<Credit>122242843/</Credit>
<Field3>456183</Field3>
<Field4>160823</Field4>
<Field5>0716</Field5>
<Field6>84</Field6>
<Field7>80</Field7>
<Field8>1</Field8>
<Field9 />
</Table1>
</NewDataSet>
Notice Field9. It is marked using FieldOptional = true. I was under the impression that by marking it as optional, if there was no value, the Field element would not be output in the final XML. Is that not the case or am I missing another property setting perhaps?
Any help would be appreciated.
Kind regards everyone!

XmlDocument to DataSet only returning 1 Row

I have an problematic piece of code and it's something really peculiar and nothing I've ever experienced before!
I'm calling a Sharepoint SOAP function and the results are being returned absolutely fine., many XML records of data are being retruned.
Now I have tried to convert the results into an XmlDocument, which I then use to load into a DataSet.
However when it gets inserted into the DataSet it's only inserting 1 record, which happens to be the very first record of the Xml.
The problematic code is the below:
Lists list = new Lists();
list.Url = URL + "_vti_bin/Lists.asmx";
list.UseDefaultCredentials = true;
//Gets the entire Lists attached to the Sharepoint Site
XmlNode Results = list.GetListCollection();
//Writes the entire results into an XmlDocument.
doc.AppendChild(doc.ImportNode(Results, true));
using (StringReader xmlSR = new StringReader(doc.InnerXml))
{
ds.ReadXml(xmlSR, XmlReadMode.Auto);
}
The Xml from 'doc.InnerXml' is all valid and is pastable into Xml Notepad 2007, so i'm a bit at a lost.
I hope someone can shed some light onto this, be much appreciated

The following example works for me:
Lists list = new Lists(); //SharePoint Lists SOAP service
//Perform request
XmlNode result = list.GetListCollection();
//Process result
var ds = new DataSet("ListsResults");
using (var reader = new StringReader(result.OuterXml))
{
ds.ReadXml(reader, XmlReadMode.Auto);
}
//print List Titles
foreach (DataRow row in ds.Tables[0].Rows)
{
Console.WriteLine(row["Title"]);
}
Another common approach is to utilize LINQ to XML:
Lists list = new Lists(); //SharePoint Lists SOAP service
//Perform request
XmlNode result = list.GetListCollection();
var docResult = XDocument.Parse(result.OuterXml);
XNamespace s = "http://schemas.microsoft.com/sharepoint/soap/";
var listEntries = from le in docResult.Descendants(s + "List")
select new
{
Title = le.Attribute("Title").Value
};
foreach (var e in listEntries)
{
Console.WriteLine(e.Title);
}

FileHelpers - Column mapping

Quick question regarding filehelper library:
I have used file helper engine to read stream, do my validation and if the CSV file has not got a header we need to match/map it to my model: i.e
id, name, age, phone, sex,
but the CSV might not come in this format/order all the time and we need to match them using a drop down list for each column.
Is there any way I can do this?
Thannks,

The short answer, no. BUT you can create a dependent class dynamically:
Since you have the list of possible fields in your JSON file, I would recommend doing a basic System.IO ReadLine for the first data row, and then parse by your delimiter for the individual headers. i.e.:
string headerString;
var headers = new List<String>();
var file = new System.IO.StreamReader("C:\\myFile.txt");
headerString = file.ReadLine();
file.Close();
headers = headerString.Split(',').ToList();
now you have the list of strings for the first row to match against your JSON file. Then you can create your dependent class using System.Reflection.Emit (referenced link below)
typeBuilder.SetParent(typeof(MyFileHelperBaseClass));
// can place the property definitions in a for loop against your headers
foreach(string h in headers){
typeBuilder.DefineProperty("<header/col#>", ..., typeof(System.Int32), null);
}
stackoverflow article 14724822: How Can I add properties to a class on runtime in C#?
File Helpers gets a little finicky at times, so it will take some tweaking.
Hope this helps

You can use File.ReadLines(#"C:\myfile.txt").First() to read the first line and get the headers.
Then you can just use a FileHelpers CodeBuilder to build your runtime class. From the example for a delimited csv file:
DelimitedClassBuilder cb = new DelimitedClassBuilder("Customers", ",");
cb.IgnoreFirstLines = 1;
cb.IgnoreEmptyLines = true;
cb.AddField("BirthDate", typeof(DateTime));
cb.LastField.TrimMode = TrimMode.Both;
cb.LastField.FieldNullValue = DateTime.Today;
cb.AddField("Name", typeof(string));
cb.LastField.FieldQuoted = true;
cb.LastField.QuoteChar = '"';
cb.AddField("Age", typeof(int));
engine = new FileHelperEngine(cb.CreateRecordClass());
DataTable dt = engine.ReadFileAsDT("testCustomers.txt");
Then you can traverse the resulting data table.

HTML Agility pack - parsing tables

I want to use the HTML agility pack to parse tables from complex web pages, but I am somehow lost in the object model.
I looked at the link example, but did not find any table data this way.
Can I use XPath to get the tables? I am basically lost after having loaded the data as to how to get the tables. I have done this in Perl before and it was a bit clumsy, but worked. (HTML::TableParser).
I am also happy if one can just shed a light on the right object order for the parsing.

How about something like:
Using HTML Agility Pack
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(#"<html><body><p><table id=""foo""><tr><th>hello</th></tr><tr><td>world</td></tr></table></body></html>");
foreach (HtmlNode table in doc.DocumentNode.SelectNodes("//table")) {
Console.WriteLine("Found: " + table.Id);
foreach (HtmlNode row in table.SelectNodes("tr")) {
Console.WriteLine("row");
foreach (HtmlNode cell in row.SelectNodes("th|td")) {
Console.WriteLine("cell: " + cell.InnerText);
}
}
}
Note that you can make it prettier with LINQ-to-Objects if you want:
var query = from table in doc.DocumentNode.SelectNodes("//table").Cast<HtmlNode>()
from row in table.SelectNodes("tr").Cast<HtmlNode>()
from cell in row.SelectNodes("th|td").Cast<HtmlNode>()
select new {Table = table.Id, CellText = cell.InnerText};
foreach(var cell in query) {
Console.WriteLine("{0}: {1}", cell.Table, cell.CellText);
}

The most simple what I've found to get the XPath for a particular Element is to install FireBug extension for Firefox go to the site/webpage press F12 to bring up firebug; right select and right click the element on the page that you want to query and select "Inspect Element" Firebug will select the element in its IDE then right click the Element in Firebug and choose "Copy XPath" this function will give you the exact XPath Query you need to get the element you want using HTML Agility Library.

I know this is a pretty old question but this was my solution that helped with visualizing the table so you can create a class structure. This is also using the HTML Agility Pack
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(#"<html><body><p><table id=""foo""><tr><th>hello</th></tr><tr><td>world</td></tr></table></body></html>");
var table = doc.DocumentNode.SelectSingleNode("//table");
var tableRows = table.SelectNodes("tr");
var columns = tableRows[0].SelectNodes("th/text()");
for (int i = 1; i < tableRows.Count; i++)
{
for (int e = 0; e < columns.Count; e++)
{
var value = tableRows[i].SelectSingleNode($"td[{e + 1}]");
Console.Write(columns[e].InnerText + ":" + value.InnerText);
}
Console.WriteLine();
}

In my case, there is a single table which happens to be a device list from a router. If you wish to read the table using TR/TH/TD (row, header, data) instead of a matrix as mentioned above, you can do something like the following:
List<TableRow> deviceTable = (from table in document.DocumentNode.SelectNodes(XPathQueries.SELECT_TABLE)
from row in table?.SelectNodes(HtmlBody.TR)
let rows = row.SelectSingleNode(HtmlBody.TR)
where row.FirstChild.OriginalName != null && row.FirstChild.OriginalName.Equals(HtmlBody.T_HEADER)
select new TableRow
{
Header = row.SelectSingleNode(HtmlBody.T_HEADER)?.InnerText,
Data = row.SelectSingleNode(HtmlBody.T_DATA)?.InnerText}).ToList();
}
TableRow is just a simple object with Header and Data as properties.
The approach takes care of null-ness and this case:
<tr>
<td width="28%"> </td>
</tr>
which is row without a header. The HtmlBody object with the constants hanging off of it are probably readily deduced but I apologize for it even still. I came from the world where if you have " in your code, it should either be constant or localizable.

Line from above answer:
HtmlDocument doc = new HtmlDocument();
This doesn't work in VS 2015 C#. You cannot construct an HtmlDocument any more.
Another MS "feature" that makes things more difficult to use. Try HtmlAgilityPack.HtmlWeb and check out this link for some sample code.