Develop Reference

Using HtmlAgilityPack with C# to find all href links within td elements in html page

I am attempting to use HtmlAgilityPack package to find each of the href links within td tags throughout an entire html page. The trick is that these tables start deep down into the html structure. I noticed with HtmlAgilityPack you can't just say get all tds that are within trs on a page. There is a parent div wrapped around each table with a class on it "table-group" that I am not showing in my sample below. Maybe I can use that as a starting point? The biggest trouble that I am dealing with is that there are several parent elements above everything in my sample below, but I want to skip all of that and start here.
Here is a sample of the structure I am trying to navigate:
<table>
<thead>
</thead>
<tbody>
<tr>
<td>Link 1</td>
<td>1</td>
</tr>
<tr>
<td>Link 2</td>
<td>2</td>
</tr>
<tr>
<td>Link 3</td>
<td>3</td>
</tr>
</tbody>
</table>
<table>
<thead>
</thead>
<tbody>
<tr>
<td>Link 4</td>
<td>4</td>
</tr>
<tr>
<td>Link 5</td>
<td>5</td>
</tr>
<tr>
<td>Link 6</td>
<td>6</td>
</tr>
</tbody>
</table>
I would like my end result to be:
https://path-to-pdf1
https://path-to-pdf2
https://path-to-pdf3
https://path-to-pdf4
https://path-to-pdf5
https://path-to-pdf6
Here is what I have tried:
var html = #"https://myurl.com";
HtmlWeb web = new HtmlWeb();
var htmlDoc = web.Load(html);
var nodes = htmlDoc.DocumentNode.SelectNodes("//table/tbody/tr/td/a[0]");
foreach (var item in nodes)
{
Console.WriteLine(item.Attributes["href"].Value);
}
Console.ReadKey();

Modify
var nodes = htmlDoc.DocumentNode.SelectNodes("//table/tbody/tr/td/a[0]");
to
var nodes = htmlDoc.DocumentNode.SelectNodes("//table/tbody/tr/td[1]/a");
then you wil get the result you want ,you could read the documents related with XPath for more details
I tried in a MVC project with the same html file:
Update:
I copied the html codes to the html page in my local and get the nodes successfully

How to download linked text - Selenium C#

I am trying to create an automation for downloading files from lined text.
Unfortunatly I can't get it to work. I am new to selenium.
Here is an HTML site code:
Ttnc-18p - 17.34 GB
<table class="table table-stripped">
<thead>
<tr>
<th>Name</th>
<th>Größe</th>
<th>DL</th>
</tr>
</thead>
<tbody>
<tr>
<td>Ttnc-18p.part01.rar
</td><td>500.00 MB</td>
<td><img width="16" height="16" src="//filer.net/media/images/ico_arrow_down.png?2018" alt="DL">
</td></tr>
<tr>
<td>Ttnc-18p.part01.rev
</td><td>500.00 MB</td>
<td><img width="16" height="16" src="//filer.net/media/images/ico_arrow_down.png?2018" alt="DL">
</td></tr>
<tr>
<td>Ttnc-18p.part02.rar
</td><td>500.00 MB</td>
<td><img width="16" height="16" src="//filer.net/media/images/ico_arrow_down.png?2018" alt="DL">
</td></tr>
<tr>
<td>Ttnc-18p.part03.rar
</td><td>500.00 MB</td>
<td><img width="16" height="16" src="//filer.net/media/images/ico_arrow_down.png?2018" alt="DL">
</td></tr>
I want it to download the link from: Ttnc-18p.part01.rar, Ttnc-18p.part02.rar, Ttnc-18p.part03.rar and so on...
I tried this:
ChromeDriver.FindElement(By.XPath("/table[contains(#class,'table table-stripped')]/tbody/tr/td/a[contains(text(),'" + "Ttnc-18p.part01.rar" + "')]")).Click();
It doesn't work and I can't figure out what to do. Any thing else fails.
The second thing I am trying to is that the code will generate an array of links that it need to download so I can feed the code different website that have differend number of linkes.
Please help.

You can try the below, but you would likely want to break the these in to separate methods for reusability:
public IEnumerable<string> DownloadLinks(string url)
{
// for storing each href value as it is retrieved from the links
var listOfLinks = new List<string>();
// start your instance of ChromeDriver
var driver = new ChromeDriver();
// Navigate to the url you passed in
driver.Navigate().GoToUrl(url);
// Get a collection of all anchor ("a") tags.
var anchorTags= driver.FindElements(By.TagName("a"));
// Now for each anchor tag...
foreach(var link in anchorTags)
{
// ...retrieve the value of its 'href' attribute (i.e. your link)...
var l = link.GetAttribute("href");
// ...add the link path to your listOfLinks
// and append your url to the href since the href is only a partial
// ( /get/vobqwunyrrxl2oo5 becomes https://yourwebsite.com/get/vobqwunyrrxl2oo5)
listOfLinks.Add(url + l);
// now click your link to simulate clicking the link and downloading the file
link.Click();
}
// and finally return your list of links
return listOfLinks;
}

Why is my xpath null? Why is it not finding the nodes

I am just getting into traversing through XML documents to learn how to use xpath.
I have stumbled on to a issue. Everytime I try to execute my xpath it returns null as if it didnt find anything.
I've tried the xpath out in XMLQuire and it worked there.
class Program
{
private static string URL = "https://www.kijiji.ca/b-renovation-contracting-handyman/ontario/home-renovations/k0c753l9004";
private static HtmlWeb client = new HtmlWeb();
static void Main(string[] args)
{
var DOM = client.Load(URL); // //table/tbody/tr/td[#class = 'description']/p
var Featured = DOM.DocumentNode.SelectNodes("//table[contains(#class,'top-feature')]/tbody/tr/td/a");
foreach (var Listing in Featured)
{
}
}
}
I commented out the other xpath I tried, I've tried those two and both are returning null why is that?
Here is a image showing the part of the DOM I want to access.
<table class="top-feature js-hover" data-ad-id="1299717863" data-vip-url="/v-renovation-contracting-handyman/sudbury/c-l-contracting-any-job-big-or-small/1299717863">
<tbody><tr>
<td class="watchlist">
<div class="watch js-hover p-vap-lnk-actn-addwtch" data-action="add" data-adid="1299717863" title="Click to add to My Favourites"><div class="icon"></div></div>
<input id="watchlistXsrf" name="ca.kijiji.xsrf.token" value="1527418405414.9b71d1309fdd8a315258ea5a3dac1a09e4a99ec7f32041df88307c46e26a5b1b" type="hidden">
</td>
<td class="image">
<div class="multiple-images"><img src="https://i.ebayimg.com/00/s/NjAwWDgwMA==/z/fXEAAOSwaZdZxTv~/$_2.JPG" alt="C.L. Contracting. Any job big or small."></div>
</td>
<td class="description">
<a href="/v-renovation-contracting-handyman/sudbury/c-l-contracting-any-job-big-or-small/1299717863" class="title ">
C.L. Contracting. Any job big or small.</a>
<p>
Contractor handyman home renovations and repairs. Contractor for Dollarama, Rexall, LaSenza and more. Fully licensed and insured. Able to do drywall, decks, framing, plumbing, flooring windows, ...</p>
<p class="details">
</p>
</td>
<td class="posted">
</td>
</tr>
</tbody></table>
My solution (Need help making my xpath into 1 line instead of traversing through with a bunch of loops.)
private static string URL = "https://www.kijiji.ca/b-renovation-contracting-handyman/ontario/home-renovations/k0c753l9004";
private static HtmlWeb client = new HtmlWeb();
static void Main(string[] args)
{
var DOM = client.Load(URL); // //table/tbody/tr/td[#class = 'description']/p
var Featured = DOM.DocumentNode.SelectNodes("//table[contains(#class,'top-feature')]/tbody/tr/td/a");
foreach (var table in DOM.DocumentNode.SelectNodes("//table[contains(#class, 'top-feature')]"))
{
Console.WriteLine($"Found: {table}");
foreach (var rows in table.SelectNodes("tr"))
{
Console.WriteLine(rows);
foreach (var cell in rows.SelectNodes("td[#class='description']/a"))
{
Console.WriteLine(cell.InnerText.Trim());
}
}
}
Console.ReadKey();
I've managed to fix it, however I ams till curious to why this xpath works
//table[contains(#class, 'top-feature')]/tr/td[#class='description']/a
And this one doesnt.
//table[contains(#class,'top-feature')]/tbody/tr/td/a

As mentioned in the comment, the <tbody> element is generated by a browser developer tool.
If you look at your var DOM object during runtime with the debugger, you can see the InnerHtml property.
<table class="regular-ad js-hover" data-ad-id=".." data-vip-url="..">
<tr>
<td class="watchlist">
...
</td>
<td class="image">
...
</td>
...
</tr>
</table>
No <tbody> element so your XPath has to look like this:
DOM.DocumentNode.SelectNodes("//table[contains(#class,'top-feature')]/tr/td/a");

How to search through html table rows?

Using Windows Forms and C#.
For example...
<table id=tbl>
<tbody>
<tr>
<td>HELLO</td>
<td>YES</td>
<td>TEST</td>
</tr>
<tr>
<td>BLAH BLAH</td>
<td>YES</td>
<td>TEST</td>
</tr>
</tbody>
</table>
I load the page using the WebBrowser Control. The page loads perfectly.
The next thing I want to do is search through all the rows in the table and check if they contain a specific value ; for example in this instance YES.
If they contain it I want the row to be passed on to me so I can store it as string.
But I want the row to be in HTML form. (containing the tags).
How can I accomplish this ?
Please help me.

You can use the HtmlAgilityPack to easily parse the html. For example, to get all of the TD elements, you can do this:
string value = #" <table id=tbl>
<tbody>
<tr>
<td>HELLO</td>
<td>YES</td>
<td>TEST</td>
</tr>
<tr>
<td>BLAH BLAH</td>
<td>YES</td>
<td>TEST</td>
</tr>
</tbody>
</table>";
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(value);
var nodes = doc.GetElementbyId("tbl").SelectNodes("tbody/tr/td");
foreach (var node in nodes)
{
Debug.WriteLine(node.InnerText);
}

You can use this: http://simplehtmldom.sourceforge.net/ , its really simple way how to search in HTML files
Just include simple_html_dom.php to your file and then just follow this manual
http://simplehtmldom.sourceforge.net/manual.htm
and your php code will looks like
$html = file_get_html('File.html');
foreach($html->find('td') as $element)
echo $element->text. '<br>';

Finding the child of a parent's sibling element with WatiN

The scenario that I am looking at is that we have a table with multiple columns. One of those columns has a name, another has a dropdown list. I need to manipulate the dropdown for a row that contains a particular name. I looked at the source output, and tried getting the element's grandparent (the table row) so that I could search for the list. However, there was no such search functionality when I used the parent object.
It seems like there would be a lot of this kind of scenario in automating/testing a site, but I have not found anything after searching for a couple of hours. Any help would be appreciated.
EDIT: The application in question is an ASP.NET, and the output HTML is gnarly at best. However, here is a cleaned up example of what the HTML being searched looks like:
<table class="myGrid" cellspacing="0" cellpadding="3" rules="all" border="1" id="ctl00_content_MyRpt_ctl01_MyGrid" style="border-collapse:collapse;">
<tr align="left" style="color:Black;background-color:#DFDBDB;">
<th scope="col">Name</th><th scope="col">Unit</th><th scope="col">Status</th><th scope="col">Action</th>
</tr>
<tr>
<td>
<span id="ctl00_content_MyRpt_ctl01_MyGrid_ctl02_Name">JOHN DOE</span>
</td>
<td>
<span id="ctl00_content_MyRpt_ctl01_MyGrid_ctl02_UnitType">Region</span>
<span id="ctl00_content_MyRpt_ctl01_MyGrid_ctl02_UnitNum">1</span>
</td>
<td>
<span id="ctl00_content_MyRpt_ctl01_MyGrid_ctl02_Status">Complete</span>
</td>
<td class="dropdown">
<select name="ctl00$content$MyRpt$ctl01$MyGrid$ctl02$ActionDropDown" onchange="javascript:setTimeout('__doPostBack(\'ctl00$content$MyRpt$ctl01$MyGrid$ctl02$ActionDropDown\',\'\')', 0)" id="ctl00_content_MyRpt_ctl01_MyGrid_ctl02_ActionDropDown" class="dropdown">
<option value="123456">I want to...</option>
<option value="Details.aspx">View Details</option>
<option value="Summary.aspx">View Summary</option>
<option value="DirectReports.aspx">View Direct Reports</option>
</select>
</td>
</tr>
<tr>
...
</tr>
</table>

I found a way to do what I wanted. It is probably not the best or most elegant solution, but it works (it is not production code).
private void btnStart_Click(object sender, EventArgs e)
{
using (var browser = new IE("http://godev/review"))
{
browser.Link(Find.ByText("My Direct Reports")).Click();
TableRow tr = browser.Span(Find.ByText("JOHN DOE")).Parent.Parent as TableRow;
SelectList objSL = null;
if (tr.Exists)
{
foreach (var td in tr.TableCells)
{
objSL = td.ChildOfType<SelectList>(Find.Any) as SelectList;
if (objSL.Exists) break;
}
if (objSL != null && objSL.Exists)
{
Option o = objSL.Option(Find.ByText("View Direct Reports"));
if (o.Exists) o.Select();
}
}
}
}
Hopefully this saves someone a little time and effort. Also, I would love to see if someone has a better solution.