exclude certain element in xPath c# - c#

My html structure like this:
<table id = "searchResultsTable">
<tbody class="searchResultsRowClass">
<tr>
<td>....</td>
<td>....</td>
</tr>
<tr>
<td>....</td>
<td>....</td>
</tr>
<!--it repeats 21 times in every page -->
</tbody>
</table>
My C# code:
private void button1_Click(object sender, EventArgs e)
{
var url = "url";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader sr = new StreamReader(response.GetResponseStream());
string sourceCode = sr.ReadToEnd();
HtmlAgilityPack.HtmlDocument document = new HtmlAgilityPack.HtmlDocument();
document.LoadHtml(sourceCode);
var rows = document.DocumentNode.SelectNodes("//*[#id='searchResultsTable']/tbody/tr");
foreach (var row in rows)
{
if (row.ChildNodes.Count > 0)
{
var name = row.SelectSingleNode("td[2]/a[1]").InnerText;
var year = row.SelectSingleNode("td[3]").InnerText;
var km = row.SelectSingleNode("td[4]").InnerText;
var color = row.SelectSingleNode("td[5]").InnerText;
var price = row.SelectSingleNode("td[6]").InnerText;
var date = row.SelectSingleNode("td[7]").InnerText;
var location = row.SelectSingleNode("td[8]").InnerText;
Console.WriteLine("name" + name + "\nyear" + year + "\nkm" + km + "\ncolor" + color + "\nprice" + price + "\ndate" + date + "\nlocation" + location);
}
}
In my html,tr[5] is empty. Therefore, I have to exclude it.
I tried /root/*[not(self::a)] like /tr/*[not(self::tr[5])]. But it didn't work.
Now, I can get only first 4 tr elements. Then,
NullReferenceException occurred.
How can I exclude one of tr element in xpath?
<table id="searchResultsTable" class="">
<thead>
<tr>
<td class="searchResultsFirstColumn"> </td>
<td class="">İlan Başlığı</td>
<td>
<a href="/otomobil?sorting=a5_asc&price_min=40000&price_max=40000">
Yıl</a>
</td>
<td>
<a href="/otomobil?sorting=a4_asc&price_min=40000&price_max=40000">
Km</a>
</td>
<td>Renk</td>
<td class="searchResultsPriceHeader">
<a href="/otomobil?sorting=price_asc&price_min=40000&price_max=40000">
Fiyat</a>
</td>
<td class="searchResultsDateHeader">
<a href="/otomobil?sorting=date_desc&price_min=40000&price_max=40000">
İlan Tarihi</a>
</td>
<td class="searchResultsLastColumn searchResultsLocationHeader">
<a href="/otomobil?sorting=address_desc&price_min=40000&price_max=40000">
İl / İlçe</a>
</td>
<td class="searchResultsIgnoredColumn"></td>
</tr>
</thead>
<tbody class="searchResultsRowClass">
<tr data-id="464336919" class="searchResultsItem ">
<td class="searchResultsLargeThumbnail">
<a href="/ilan/vasita-otomobil-fiat-automobilworld-den-2015-linea-pop-89.000km-degisensiz-faturali-464336919/detay">
<img src="https://image5.sahibinden.com/photos/33/69/19/thmb_464336919waj.jpg" alt="AUTOMOBİLWORLD'DEN 2015 LINEA POP 89.000KM DEĞİŞENSİZ FATURALI #464336919" title="AUTOMOBİLWORLD'DEN 2015 LINEA POP 89.000KM DEĞİŞENSİZ FATURALI">
</a></td>
<td class="searchResultsTitleValue ">
<input id="favoriteClassifiedsVisibility" type="hidden" value="true">
<div class="action-wrapper" data-classified-id="464336919">
<div class="add-to-favorites last favorite">
<a href="#" class="action classifiedAddFavorite trackClick trackId_favorite hidden">
Favorilerime Ekle</a>
<a href="#" class="action classifiedRemoveFavorite trackClick trackId_favorite disable">
Favorilerimde</a>
</div>
<div class="compare hidden">
<a class="facetedCheckbox action compare-classified">
<i></i>Karşılaştır</a>
</div>
</div>
<a class="classifiedTitle" href="/ilan/vasita-otomobil-fiat-automobilworld-den-2015-linea-pop-89.000km-degisensiz-faturali-464336919/detay">
AUTOMOBİLWORLD'DEN 2015 LINEA POP 89.000KM DEĞİŞENSİZ FATURALI</a>
<a class="titleIcon store-icon" href="https://automobilworld.sahibinden.com/" title="AUTOMOBIL WORLD" style="visibility: visible;">
<img class="titleIcon" src="https://s0.shbdn.com/assets/images/iconStore:e98c183976843a1e5b3d4e580d614009.png" alt="AUTOMOBIL WORLD" title="AUTOMOBIL WORLD" style="visibility: visible;">
</a>
<img class="titleIcon" alt="Haritalı İlan" title="Haritalı İlan" src="https://s0.shbdn.com/assets/images/iconHasMap:1f5f8f9b79e391584fe00304345baa05.png" style="visibility: visible;">
<br>
<div class="classifiedSubtitle " style="visibility: visible;">
Fiat > Linea > 1.3 Multijet Pop</div>
</td>
<td class="searchResultsAttributeValue">
2015</td>
<td class="searchResultsAttributeValue">
89.000</td>
<td class="searchResultsAttributeValue">
Beyaz</td>
<td class="searchResultsPriceValue">
<div> 40.000 TL</div></td>
<td class="searchResultsDateValue">
<span>21 Temmuz</span>
<br>
<span>2017</span>
</td>
<td class="searchResultsLocationValue">
İstanbul<br>Büyükçekmece</td>
<td class="ignore-me">
<a href="#" class="mark-as-not-ignored disable">
Göster</a>
</td>
</tr>
<tr data-id="460187522" class="searchResultsItem ">
<td class="searchResultsLargeThumbnail">
<a href="/ilan/vasita-otomobil-volkswagen-orjinal-kilometre-124800km-lpg-460187522/detay">
<img src="https://image5.sahibinden.com/photos/18/75/22/thmb_460187522s6x.jpg" alt="ORJİNAL KİLOMETRE 124800KM LPG #460187522" title="ORJİNAL KİLOMETRE 124800KM LPG">
</a></td>
<td class="searchResultsTitleValue ">
<input id="favoriteClassifiedsVisibility" type="hidden" value="true">
<div class="action-wrapper" data-classified-id="460187522">
<div class="add-to-favorites last favorite">
<a href="#" class="action classifiedAddFavorite trackClick trackId_favorite hidden">
Favorilerime Ekle</a>
<a href="#" class="action classifiedRemoveFavorite trackClick trackId_favorite disable">
Favorilerimde</a>
</div>
<div class="compare hidden">
<a class="facetedCheckbox action compare-classified">
<i></i>Karşılaştır</a>
</div>
</div>
<a class="classifiedTitle" href="/ilan/vasita-otomobil-volkswagen-orjinal-kilometre-124800km-lpg-460187522/detay">
ORJİNAL KİLOMETRE 124800KM LPG</a>
<a class="titleIcon store-icon" href="https://42.sahibinden.com/" title="HÜSEYİN ÖRNEK" style="visibility: visible;">
<img class="titleIcon" src="https://s0.shbdn.com/assets/images/iconStore:e98c183976843a1e5b3d4e580d614009.png" alt="HÜSEYİN ÖRNEK" title="HÜSEYİN ÖRNEK" style="visibility: visible;">
</a>
<img class="titleIcon" alt="Haritalı İlan" title="Haritalı İlan" src="https://s0.shbdn.com/assets/images/iconHasMap:1f5f8f9b79e391584fe00304345baa05.png" style="visibility: visible;">
<br>
<div class="classifiedSubtitle " style="visibility: visible;">
Volkswagen > Passat > 1.6 Comfortline</div>
</td>
<td class="searchResultsAttributeValue">
2002</td>
<td class="searchResultsAttributeValue">
124.800</td>
<td class="searchResultsAttributeValue">
Beyaz</td>
<td class="searchResultsPriceValue">
<div> 40.000 TL</div></td>
<td class="searchResultsDateValue">
<span>10 Temmuz</span>
<br>
<span>2017</span>
</td>
<td class="searchResultsLocationValue">
Konya<br>Selçuklu</td>
<td class="ignore-me">
<a href="#" class="mark-as-not-ignored disable">
Göster</a>
</td>
</tr>
<tr data-id="397435322" class="searchResultsItem ">
<td class="searchResultsLargeThumbnail">
<a href="/ilan/vasita-otomobil-renault-2013-renault-clio-4-touch-paket-boyasizzzzzzzzzzzzzzzzzz-397435322/detay">
<img src="https://image5.sahibinden.com/photos/43/53/22/thmb_397435322ao8.jpg" alt="2013 RENAULT CLİO 4 TOUCH PAKET ___BOYASIZZZZZZZZZZZZZZZZZZ___ #397435322" title="2013 RENAULT CLİO 4 TOUCH PAKET ___BOYASIZZZZZZZZZZZZZZZZZZ___">
</a></td>
<td class="searchResultsTitleValue ">
<input id="favoriteClassifiedsVisibility" type="hidden" value="true">
<div class="action-wrapper" data-classified-id="397435322">
<div class="add-to-favorites last favorite">
<a href="#" class="action classifiedAddFavorite trackClick trackId_favorite hidden">
Favorilerime Ekle</a>
<a href="#" class="action classifiedRemoveFavorite trackClick trackId_favorite disable">
Favorilerimde</a>
</div>
<div class="compare hidden">
<a class="facetedCheckbox action compare-classified">
<i></i>Karşılaştır</a>
</div>
</div>
<a class="classifiedTitle" href="/ilan/vasita-otomobil-renault-2013-renault-clio-4-touch-paket-boyasizzzzzzzzzzzzzzzzzz-397435322/detay">
2013 RENAULT CLİO 4 TOUCH PAKET ___BOYASIZZZZZZ­ZZZZZZZZZZZZ___</a>
<a class="titleIcon store-icon" href="https://guvenototarsus.sahibinden.com/" title="GÜVEN OTOMOTİV" style="visibility: visible;">
<img class="titleIcon" src="https://s0.shbdn.com/assets/images/iconStore:e98c183976843a1e5b3d4e580d614009.png" alt="GÜVEN OTOMOTİV" title="GÜVEN OTOMOTİV" style="visibility: visible;">
</a>
<img class="titleIcon" alt="Haritalı İlan" title="Haritalı İlan" src="https://s0.shbdn.com/assets/images/iconHasMap:1f5f8f9b79e391584fe00304345baa05.png" style="visibility: visible;">
<br>
<div class="classifiedSubtitle " style="visibility: visible;">
Renault > Clio > 1.2 Touch</div>
</td>
<td class="searchResultsAttributeValue">
2013</td>
<td class="searchResultsAttributeValue">
74.000</td>
<td class="searchResultsAttributeValue">
Siyah</td>
<td class="searchResultsPriceValue">
<div> 40.000 TL</div></td>
<td class="searchResultsDateValue">
<span>08 Temmuz</span>
<br>
<span>2017</span>
</td>
<td class="searchResultsLocationValue">
Mersin<br>Tarsus</td>
<td class="ignore-me">
<a href="#" class="mark-as-not-ignored disable">
Göster</a>
</td>
</tr>
<tr data-id="458875511" class="searchResultsItem ">
<td class="searchResultsLargeThumbnail">
<a href="/ilan/vasita-otomobil-ford-2011-model-40.binde-otomatik-458875511/detay">
<img src="https://image5.sahibinden.com/photos/87/55/11/thmb_458875511431.jpg" alt="2011 MODEL 40.binde OTOMATİK #458875511" title="2011 MODEL 40.binde OTOMATİK">
</a></td>
<td class="searchResultsTitleValue ">
<input id="favoriteClassifiedsVisibility" type="hidden" value="true">
<div class="action-wrapper" data-classified-id="458875511">
<div class="add-to-favorites last favorite">
<a href="#" class="action classifiedAddFavorite trackClick trackId_favorite hidden">
Favorilerime Ekle</a>
<a href="#" class="action classifiedRemoveFavorite trackClick trackId_favorite disable">
Favorilerimde</a>
</div>
<div class="compare hidden">
<a class="facetedCheckbox action compare-classified">
<i></i>Karşılaştır</a>
</div>
</div>
<a class="classifiedTitle" href="/ilan/vasita-otomobil-ford-2011-model-40.binde-otomatik-458875511/detay">
2011 MODEL 40.binde OTOMATİK</a>
<a class="titleIcon store-icon" href="https://mackaotomotiv.sahibinden.com/" title="MAÇKA OTOMOTİV" style="visibility: visible;">
<img class="titleIcon" src="https://s0.shbdn.com/assets/images/iconStore:e98c183976843a1e5b3d4e580d614009.png" alt="MAÇKA OTOMOTİV" title="MAÇKA OTOMOTİV" style="visibility: visible;">
</a>
<img class="titleIcon" alt="Haritalı İlan" title="Haritalı İlan" src="https://s0.shbdn.com/assets/images/iconHasMap:1f5f8f9b79e391584fe00304345baa05.png" style="visibility: visible;">
<br>
<div class="classifiedSubtitle " style="visibility: visible;">
Ford > Fiesta > 1.4 Titanium</div>
</td>
<td class="searchResultsAttributeValue">
2011</td>
<td class="searchResultsAttributeValue">
40.000</td>
<td class="searchResultsAttributeValue">
Gümüş Gri</td>
<td class="searchResultsPriceValue">
<div> 40.000 TL</div></td>
<td class="searchResultsDateValue">
<span>07 Temmuz</span>
<br>
<span>2017</span>
</td>
<td class="searchResultsLocationValue">
Düzce<br>Merkez</td>
<td class="ignore-me">
<a href="#" class="mark-as-not-ignored disable">
Göster</a>
</td>
</tr>
<tr class="searchResultsPromoToplist">
<td colspan="12">
<div><strong>Siz de ilanınızın yukarıda yer almasını istiyorsanız <u>tıklayın</u>.</strong></div>
</td>
</tr>
<tr data-id="465252780" class="searchResultsItem ">
<td class="searchResultsLargeThumbnail">
<a href="/ilan/vasita-otomobil-peugeot-multimedia-sistemli-temiz-2014-active-paket-301-465252780/detay">
<img class="searchResultThumbnailPlaceholder otherNoImage" src="https://s0.shbdn.com/assets/images/iconHasMegaPhoto:1d086aab554fd92d49d3762a0542888a.png" alt="Multimedia Sistemli Temiz 2014 Active Paket 301 #465252780" title="Megafotolu ilan">
</a></td>
<td class="searchResultsTitleValue ">
<input id="favoriteClassifiedsVisibility" type="hidden" value="true">
<div class="action-wrapper" data-classified-id="465252780">
<div class="add-to-favorites last favorite">
<a href="#" class="action classifiedAddFavorite trackClick trackId_favorite hidden">
Favorilerime Ekle</a>
<a href="#" class="action classifiedRemoveFavorite trackClick trackId_favorite disable">
Favorilerimde</a>
</div>
<div class="compare hidden">
<a class="facetedCheckbox action compare-classified">
<i></i>Karşılaştır</a>
</div>
</div>
<img class="titleIcon" src="https://s0.shbdn.com/assets/images/iconNew:c9b443de96056beb84b4cdc03ca5051c.png" alt="Yeni İlan" title="Yeni İlan" style="visibility: visible;">
<a class="classifiedTitle" href="/ilan/vasita-otomobil-peugeot-multimedia-sistemli-temiz-2014-active-paket-301-465252780/detay">
Multimedia Sistemli Temiz 2014 Active Paket 301</a>
<img class="titleIcon" alt="Haritalı İlan" title="Haritalı İlan" src="https://s0.shbdn.com/assets/images/iconHasMap:1f5f8f9b79e391584fe00304345baa05.png" style="visibility: visible;">
<br>
<div class="classifiedSubtitle " style="visibility: visible;">
Peugeot > 301 > 1.6 HDi Active</div>
</td>
<td class="searchResultsAttributeValue">
2014</td>
<td class="searchResultsAttributeValue">
95.500</td>
<td class="searchResultsAttributeValue">
Beyaz</td>
<td class="searchResultsPriceValue">
<div> 40.000 TL</div></td>
<td class="searchResultsDateValue">
<span>25 Temmuz</span>
<br>
<span>2017</span>
</td>
<td class="searchResultsLocationValue">
İstanbul<br>Kadıköy</td>
<td class="ignore-me">
<a href="#" class="mark-as-not-ignored disable">
Göster</a>
</td>
</tr>
Here is part of the html. I tried to exclude <tr class="searchResultsPromoToplist">

If you only want to match <tr> elements that have <td> child elements, change you XPath from:
//*[#id='searchResultsTable']/tbody/tr
to:
//*[#id='searchResultsTable']/tbody/tr[td]
EDIT
From the HTML you posted it would seem that the <tr> elements you are interested in all have a data-id attribute. If that is the case, change your XPath to:
//*[#id='searchResultsTable']/tbody/tr[#data-id]

Try
string.isnullorempty(yourrowelement)? yourrowelement:string.empty

Related

Unable to locate element inside a table using xpath with selenium

I want to click on a button inside my table each row has a update button I want to click on a specfic button inside my table.
Here is a what my table looks like:
<table _ngcontent-vhp-c82="" datatable="" id="dtOptionsComments" class="display table table-striped table-bordered dt-responsive dataTable dtr-inline" aria-describedby="dtOptionsComments_info" style="width: 100%;" width="100%">
<thead _ngcontent-vhp-c82="">
<tr _ngcontent-vhp-c82="">
<th _ngcontent-vhp-c82="" class="no-marking sorting_disabled" rowspan="1" colspan="1" style="width: 50.4px;" aria-label=""></th>
<th _ngcontent-vhp-c82="" class="sorting sorting_asc" tabindex="0" aria-controls="dtOptionsComments" rowspan="1" colspan="1" style="width: 1109.4px;" aria-label="Comment.Comment Shipping.ShippingDatatable.aria.sortDescending" aria-sort="ascending">Comentario Shipping.Shipping</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td class="no-marking dtr-control">
<a href="javascript:void(0);">
<span data-toggle="modal" data-target="#update-modal" update-comment-text="6 MESES DE GARANTIA" update-comment-id="5" class="material-icons md-18 clickable"> edit </span>
</a>
<a href="javascript:void(0);">
<span data-toggle="modal" data-target="#delete-modal" delete-comment-id="5" class="material-icons clickable">delete</span>
</a>
</td>
<td class="sorting_1">6 MESES DE GARANTIA</td>
</tr>
<!-- MORE ROWS!!! -->
</tbody>
<tfoot _ngcontent-vhp-c82="">
<tr _ngcontent-vhp-c82="">
<td _ngcontent-vhp-c82="" class="no-marking" rowspan="1" colspan="1">
<a _ngcontent-vhp-c82="" href="javascript:void(0);">
<span _ngcontent-vhp-c82="" class="material-icons clickable"> add_box </span>
</a>
</td>
<td _ngcontent-vhp-c82="" rowspan="1" colspan="1">
<input _ngcontent-vhp-c82="" formcontrolname="addComment" type="text" id="addComment" name="addComment" class="form-control ng-untouched ng-pristine ng-invalid">
<!---->
</td>
</tr>
</tfoot>
</table>
Here is my code trials:
IWebElement btnUpdate = _driver.FindElement(By.XPath("//*[update-comment-id='" + commentAction.GetLastQuoteInsertId().ToString() + "']"));
btnUpdate.Click();
I have validated that the function GetLastQuoteInsertId returns the proper value
Why is my xPath selector wrong how can I fix it thank you for your help.
You were almost there. While considering a xpath the attribute_name should be always preceded by a # sign.
Additionally to make the xpath more canonical as the element is a <span> element you can mention //span to start the xpath.
Effectively, your line of code will be:
IWebElement btnUpdate = _driver.FindElement(By.XPath("//span[#update-comment-id='" + commentAction.GetLastQuoteInsertId().ToString() + "']"));
btnUpdate.Click();

I need help to implement XPath

I have the following email from amazon that is from temp-mail.org
Basically amazon has sent me a verification code after i signed up with an email from temp-mail.org. I would like to use XPath to scrape/extract that Verification code and ignore everything else in the email and save it as a variable.
Here is the source for the email:
<div class="col-md-7 col-md-offset-0 col-sm-10 col-sm-offset-2 col-xs-12 ord2">
<div class="content main" style="opacity: 100;">
<div class="mailView" style="display: block">
<div class="pm-ctrl clearfix">
<ul class="reset lcol">
<li class="lcol">
<span class="glyphicon glyphicon-chevron-left"></span> Back to list
</li>
</ul>
<ul class="reset rcol">
<li class="lcol">
<span class="glyphicon glyphicon-download-alt"></span> Download
</li>
<li class="lcol">
<span class="glyphicon glyphicon-remove"></span> Delete
</li>
<li class="lcol">
<span class="glyphicon glyphicon-wrench"></span> Source
</li>
</ul>
</div>
<div class="pm-info">
<h4 class="pm-subject">Amazon password assistance</h4>
<ul class="reset">
<li><span class="glyphicon glyphicon-user grey"></span> From: "Amazon.co.uk" <account-update#amazon.co.uk></li>
<li><span class="glyphicon glyphicon-time grey"></span> Date: 12-01-2019 11:19:27</li>
</ul>
</div>
<div class="pm-text">
<div data-x-div-type="html" xmlns="http://www.w3.org/1999/xhtml">
<div data-x-div-type="body">
<img width="1" height="1" src="https://www.amazon.co.uk/gp/r.html?C=4XBRCDY40J48&M=urn:rtn:msg:20190112112940370f133b15924642ae9c1749c160p0eu&R=3SXE68KQSKHXF&T=O&U=https%3A%2F%2Fimages-eu.ssl-images-amazon.com%2Fimages%2FG%2F01%2Fnav%2Ftransp.gif&H=MSPPWISARSEXNQ35VMWJXUZ01L4A&ref_=pe_1764051_62581901_opens">
<table
align="center" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td>
<table cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td>
<table cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td width="250">
<img src="https://images-na.ssl-images-amazon.com/images/G/01/x-locale/cs/te/logo._CB152417367_.png"></td>
<td width="250" valign="top" align="right">
<p>Password assistance</p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td>
<p>To verify your identity, please use the following code:</p>
<p>451429</p>
</td>
</tr>
<tr>
<td>
<p>Amazon takes your account security very seriously. Amazon will never email you and ask you to disclose or verify your Amazon password, credit card or banking account number. If you receive a suspicious email with a link to
update your account information, do not click on the link—instead, report the email to Amazon for investigation.
</p>
</td>
</tr>
<tr>
<td>
<p>We hope to see you again soon.
</p>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table><img width="1" height="1" src="https://www.amazon.co.uk/gp/r.html?C=4XBRCDY40J48&M=urn:rtn:msg:20190112112940370f133b15924642ae9c1749c160p0eu&R=HE8PCAHPJVPN&T=E&U=https%3A%2F%2Fimages-eu.ssl-images-amazon.com%2Fimages%2FG%2F01%2Fnav%2Ftransp.gif&H=NJAGFLUUE0Y1VUAJCA4S9MABYVAA&ref_=pe_1764051_62581901_open">
</div>
</div>
</div>
</div>
As you can see, the code is
<p>451429</p>
Now how would i go about using XPath, or anything else (I have an open mind), to save that specific Code which will change everytime i run the program, to a variable called "verifyCode" or whatever variable name you please. Thank you!
Using inspect element, i have found the XPath which is:
//*[#cellspacing='0']/tbody/tr/td/table/tbody/tr[2]/td/p[2]
but when i use
var codeverify = driver.FindElement(By.XPath("//[#cellspacing='0']/tbody/tr/td/table/tbody/tr[2]/td/p[2]/text()"));
it tells me XPath is invalid
You can use HtmlAgilityPack nugget package and use below code. Input.html is the HTML which you mentioned in your question.
var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(File.ReadAllText("Input.html"));
var element = htmlDoc.DocumentNode.Descendants()
.Where(x => x.InnerText == "To verify your identity, please use the following code:")
.ToList()
.FirstOrDefault();
while (element.NextSibling.Name != "p")
element = element.NextSibling;
var code= element.NextSibling.InnerText;

Unable to locate input element under div in Selenium

I am trying to located input element "mpcname" in js file which is under div and table elements.
The code I have written to locate the element is as follows but is not detecting the Input element with the name mpcName in it.
Driver.FindElement(By.XPath(".//input[contains(.,'mpcName')]"));
The HTML code looks like below.
<div enable-tooltips="" class="ng-scope">
<form name="InstanceDetailForm" ng-submit="doApply()" novalidate="" class="ng-pristine ng-invalid ng-invalid-required ng-valid-unnamed ng-valid-maxlength ng-valid-not-integer">
<table class="table-condensed" style="width: 100%;">
<tbody><tr>
<td></td>
</tr>
<tr>
<td>
<table style="width: 100%;">
<tbody><tr>
<td class="pull-left">
<h4 class="ng-binding">Controller Configuration<b ng-show="InstanceIsDirty()" class="ng-hide"> *</b></h4>
</td>
<td class="pull-right">
<button id="btnApply" type="submit" class="btn btn-primary" data-toggle="tooltip" data-placement="bottom" title="Save an APC instance">
<span class="glyphicon glyphicon-save"></span> Apply
</button>
<button id="btnExportConfig" type="button" class="btn btn-primary" ng-click="goToInstanceList()" data-toggle="tooltip" data-placement="bottom" title="Cancel save and redirect to APC instance list page.">
<span class="glyphicon glyphicon-remove"></span> Cancel
</button>
</td>
</tr>
</tbody></table>
</td>
</tr>
<tr>
<td>
<table style="width: 100%" class="table-condensed table-striped table-bordered">
<tbody><tr>
<td>Name</td>
<td>
**<input id="mpcname" name="mpcname" ng-model="mpcname" value="" ng-required="true" is-named="" size="50" autocomplete="off" ng-disabled="!allowEditInstanceName(mpcname,state)" class="ng-pristine ng-empty ng-invalid ng-invalid-required ng-valid-unnamed ng-touched" required="required" type="text">**
With xpath ".//input[contains(.,'mpcName')]" you are actually looking for an <input> element with text mpcName. mpcName is actually the id (and name and ng-model) attribute. You can use
Driver.FindElement(By.Id("mpcName"));
Or
Driver.FindElement(By.Name("mpcName"));
following code may be helpful
Driver.FindElement(By.XPath(".//input[contains(#id,'mpc')]");
Driver.FindElement(By.XPath(".//input[contains(#class,'ng-pris')]");

Parsing HTML content with C# Parser

I have the following HTML file, I want to get each H2 (Standard (Flexible Rate).. and Executive (Flexible Rate) ... with the Room Only , Breakfast Included .
Then push the Room Only and Breakfast Include with 2 Prices of each into objects where I have Standard along with 2 prices from Room Only and Breakfast Included, and the same for Executive
I tried Fizzler with AgilityPack however, I couldn't get the correct results, could you please suggest me an idea or one good parser for this case? Thanks
<div id="accordionResizer" style="padding:5px; height:300px; border-radius:6px;" class="ui-widget-content regestancias">
<div id="accordion" class="dias">
<h2>
<a href="#">
Standard (Flexible Rate) from 139 €
</a>
</h2>
<div class="estancias_precios estancias_precios_new">
<table style="width: 285px;">
<tr class="" title="">
<cont>
<td style="width: 25px;">
<input type="radio" name="estancias" id="tarifa602385" elem="tarifa" idelem="602" idreg="385" precio="139" reg="Only%20Bed" nombre="Standard%20%28Flexible%20Rate%29" />
</td>
<td style="width: 155px;">
<label class="descrip" for="tarifa602385" precio="139.00" reg="Only%20Bed" nombre="Standard%20%28Flexible%20Rate%29">
Only Bed
</label>
</td>
<td style="width: 55px;"></td>
<td style="width: 55px;">
<strong class="precios_mos">139.00 €</strong>
</td>
</cont>
</tr>
<tr class="" title="">
<cont>
<td style="width: 25px;">
<input type="radio" name="estancias" id="tarifa602386" elem="tarifa" idelem="602" idreg="386" precio="156.9" reg="Breakfast%20Included" nombre="Standard%20%28Flexible%20Rate%29" />
</td>
<td style="width: 155px;">
<label class="descrip" for="tarifa602386" precio="156.90" reg="Breakfast%20Included" nombre="Standard%20%28Flexible%20Rate%29">
Breakfast Included
</label>
</td>
<td style="width: 55px;"></td>
<td style="width: 55px;">
<strong class="precios_mos">156.90 €</strong>
</td>
</cont>
</tr>
</table>
</div>
<h2>
<a href="#">
Executive (Flexible Rate) from 169 €
</a>
</h2>
<div class="estancias_precios estancias_precios_new">
<table style="width: 285px;">
<tr class="" title="">
<cont>
<td style="width: 25px;">
<input type="radio" name="estancias" id="tarifa666385" elem="tarifa" idelem="666" idreg="385" precio="169" reg="Only%20Bed" nombre="Executive%20%28Flexible%20Rate%29" />
</td>
<td style="width: 155px;">
<label class="descrip" for="tarifa666385" precio="169.00" reg="Only%20Bed" nombre="Executive%20%28Flexible%20Rate%29">
Only Bed
</label>
</td>
<td style="width: 55px;"></td>
<td style="width: 55px;">
<strong class="precios_mos">169.00 €</strong>
</td>
</cont>
</tr>
<tr class="" title="">
<cont>
<td style="width: 25px;">
<input type="radio" name="estancias" id="tarifa666386" elem="tarifa" idelem="666" idreg="386" precio="186.9" reg="Breakfast%20Included" nombre="Executive%20%28Flexible%20Rate%29" />
</td>
<td style="width: 155px;">
<label class="descrip" for="tarifa666386" precio="186.90" reg="Breakfast%20Included" nombre="Executive%20%28Flexible%20Rate%29">
Breakfast Included
</label>
</td>
<td style="width: 55px;"></td>
<td style="width: 55px;">
<strong class="precios_mos">186.90 €</strong>
</td>
</cont>
</tr>
</table>
</div>
</div>
</div>
Here you go with a quick and dirty approach:
class RoomInfo
{
public String Name { get; set; }
public Dictionary<String, Double> Prices { get; set; }
}
private static void HtmlFile()
{
List<RoomInfo> rooms = new List<RoomInfo>();
HtmlDocument document = new HtmlDocument();
document.Load("file.txt");
var h2Nodes = document.DocumentNode.SelectNodes("//h2");
foreach (var h2Node in h2Nodes)
{
RoomInfo roomInfo = new RoomInfo
{
Name = h2Node.InnerText.Trim(),
Prices = new Dictionary<string, double>()
};
var labels = h2Node.NextSibling.NextSibling.SelectNodes(".//label");
foreach (var label in labels)
{
roomInfo.Prices.Add(label.InnerText.Trim(), Convert.ToDouble(label.Attributes["precio"].Value, CultureInfo.InvariantCulture));
}
rooms.Add(roomInfo);
}
}
The rest is up to you! ;-)

How do I scrape information off ASP.NET websites when paging and JavaScript links are being used?

I have been given a staff list which is supposed to be up to date but it doesn't match an intranet People Finder which is written in ASP.NET.
As the information is sensitive I am not able to access the database the People Finder is using so the only way I can get at the information is by scraping the structure starting at the top brass at the top and then going through each tier in turn.
Each person has a Staff number which then forms the URL http://intranet/peoplefinder/index.aspx?srn=ABC1234 and then all the people who report to them are listed underneth in the format <a id="gvEmployees_ctl03_lnkFullName" href="index.aspx?srn=ABC4321" target="_self"> where each URL indicates the Staff number and provides a link to their team.
The trouble arises when the teams are big as paging is implemented in the GridView with an URL such as 2.
How would I scrape this page, capture the SRN and other details along with the people who report to the person on all pages of the GridView then loop through each reportee and do the same process until the whole list is complete?
Example HTML of result
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" >
<head><title>
People Finder: Name Surname
</title><link rel="stylesheet" href="/path/to/style.css" type="text/css" /><link rel="stylesheet" href="/path/to/anotherStyle.css" type="text/css" />
<script type="text/javascript" src="/path/to/peoplefinder.js"></script>
</head>
<body>
<form name="form1" method="post" action="/path/to/index.aspx" id="form1">
<div>
<input type="hidden" name="__EVENTTARGET" id="__EVENTTARGET" value="" />
<input type="hidden" name="__EVENTARGUMENT" id="__EVENTARGUMENT" value="" />
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="### ViewState ###" />
</div>
<script type="text/javascript">
<!--
var theForm = document.forms['form1'];
if (!theForm) {
theForm = document.form1;
}
function __doPostBack(eventTarget, eventArgument) {
if (!theForm.onsubmit || (theForm.onsubmit() != false)) {
theForm.__EVENTTARGET.value = eventTarget;
theForm.__EVENTARGUMENT.value = eventArgument;
theForm.submit();
}
}
// -->
</script>
<script src="/path/to/WebResource.axd?d=AueXWrgAf8xSxMTAt1Q4AA2&t=633311832634916698" type="text/javascript"></script>
<div class="HP3CHeader">
<div id="LWHPBanner">
<h1><span id="lblName">Name Surname</span></h1>
</div>
</div>
<div id='CPMain'>
<div id="mainBox">
<div id="pnlEmployeeDetails">
<div id='basicData'>
<img id="imgPhoto" class="photo" src="/path/to/photo.jpg" style="height:69px;width:69px;border-width:0px;" />
<span id="lblBusinessUnit">Business Unit</span>
<span id="lblCostCentreName">Cost Centre</span>
<span id="lblLocation">Location</span>
<a href='/path/to/checkcontactdetails.htm' target='_blank' onclick='return OpenCheckContactDetails();' >Find out how to change your details/photo.</a>
<div id="manager">
<strong>Reports to: </strong><a id="hlManager" href="/path/to/index.aspx?srn=ABC1234">Name Surname</a>
</div>
</div>
<div id='contactData'>
<div id="pnlSrn">
<strong>Staff number:</strong> <span id="lblSrn">ABC1234</span>
</div>
<div id="pnlEmailAddress">
<strong>Email Address:</strong> <span id="lblEmailAddress">Email</span>
</div>
<div style="clear: both"></div>
</div>
</div>
<div id="pnlGrid">
<h3><span id="lblGridTitle">Name's team</span></h3>
<div>
<table class="subordinates" cellspacing="0" cellpadding="2" rules="cols" border="1" id="gvEmployees" style="border-style:None;border-collapse:collapse;">
<tr style="color:Black;background-color:#EFF3FB;border-style:None;font-weight:bold;">
<th scope="col">SRN</th><th scope="col">Full name</th><th scope="col">RACFID</th>
</tr><tr class="reports" style="background-color:White;border-style:None;">
<td style="width:70px;">ABC1234</td><td>
<a id="gvEmployees_ctl02_lnkFullName" href="index.aspx?srn=1K5932" target="_self">Name Surname</a>
</td><td>ABCD</td>
</tr><tr class="reports" style="background-color:#EFF3FB;border-style:None;">
<td style="width:70px;">ABC1234</td><td>
<a id="gvEmployees_ctl03_lnkFullName" href="/path/to/index.aspx?srn=ABC1234" target="_self">Name Surname</a>
</td><td>ABCD</td>
</tr><tr class="reports" style="background-color:White;border-style:None;">
<td style="width:70px;">ABC1234</td><td>
<a id="gvEmployees_ctl04_lnkFullName" href="/path/to/index.aspx?srn=ABC1234" target="_self">Name Surname</a>
</td><td>ABCD</td>
</tr><tr class="reports" style="background-color:#EFF3FB;border-style:None;">
<td style="width:70px;">ABC1234</td><td>
<a id="gvEmployees_ctl05_lnkFullName" href="/path/to/index.aspx?srn=ABC1234" target="_self">Name Surname</a>
</td><td>ABCD</td>
</tr><tr class="reports" style="background-color:White;border-style:None;">
<td style="width:70px;">ABC1234</td><td>
<a id="gvEmployees_ctl06_lnkFullName" href="/path/to/index.aspx?srn=ABC1234" target="_self">Name Surname</a>
</td><td>ABCD</td>
</tr><tr class="reports" style="background-color:#EFF3FB;border-style:None;">
<td style="width:70px;">ABC1234</td><td>
<a id="gvEmployees_ctl07_lnkFullName" href="/path/to/index.aspx?srn=ABC1234" target="_self">Name Surname</a>
</td><td>ABCD</td>
</tr><tr class="reports" style="background-color:White;border-style:None;">
<td style="width:70px;">ABC1234</td><td>
<a id="gvEmployees_ctl08_lnkFullName" href="/path/to/index.aspx?srn=ABC1234" target="_self">Name Surname</a>
</td><td>ABCD</td>
</tr><tr class="reports" style="background-color:#EFF3FB;border-style:None;">
<td style="width:70px;">ABC1234</td><td>
<a id="gvEmployees_ctl09_lnkFullName" href="/path/to/index.aspx?srn=ABC1234" target="_self">Name Surname</a>
</td><td>ABCD</td>
</tr><tr class="reports" style="background-color:White;border-style:None;">
<td style="width:70px;">ABC1234</td><td>
<a id="gvEmployees_ctl10_lnkFullName" href="/path/to/index.aspx?srn=ABC1234" target="_self">Name Surname</a>
</td><td>ABCD</td>
</tr><tr class="reports" style="background-color:#EFF3FB;border-style:None;">
<td style="width:70px;">ABC1234</td><td>
<a id="gvEmployees_ctl11_lnkFullName" href="/path/to/index.aspx?srn=ABC1234" target="_self">Name Surname</a>
</td><td>ABCD</td>
</tr><tr class="PagerStyle" style="color:#000039;border-style:None;">
<td colspan="3"><table border="0">
<tr>
<td><span>1</span></td><td>2</td>
</tr>
</table></td>
</tr>
</table>
</div>
</div>
</div>
<div id="searchBox">
<strong>Search People Finder:</strong>
<br /><br />
<span>Forename:</span><br/>
<span><input name="txtFirstname" type="text" id="txtFirstname" /></span><br/>
<span>Surname:</span><br/>
<span><input name="txtSurname" type="text" id="txtSurname" /></span><br/>
<span>RACFID:</span><br/>
<span><input name="txtRacfid" type="text" id="txtRacfid" /></span><br/>
<span>Staff number:</span><br/>
<span><input name="txtSrn" type="text" id="txtSrn" /></span><br/>
<div class="searchBoxItem" style="text-align:center;width:100%"><input type="submit" name="btnFind" value="Search" onclick="javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions("btnFind", "", false, "", "index.aspx", false, false))" id="btnFind" title="Search for employees member" class="button" style="border-style:Outset;" /></div><br/>
<div>People Finder searches only UK staff.</div>
<!-- <div><a class="execBoardLink" href="/path/to/index.aspx?srn=ABC1234">Show Executive Board</a></div> -->
<div style="margin-top:5px;">Open Phonebook</div>
</div>
</div>
<div class="contentFooter" style="text-align:center;">
<table width="100%" cellpadding="0" cellspacing="0" border="0" summary="Navigation layout table">
<tr>
<td align="left"><span class="linkArrow"><</span> Back</td>
<td align="center"></td>
<td align="right"><span class="linkArrow">^ </span>Top</td>
</tr>
</table>
</div>
<div>
<input type="hidden" name="__PREVIOUSPAGE" id="__PREVIOUSPAGE" value="vy066Txz34y1E515UsTSTDabHKEmdBRCsq7xM0lpJls1" />
<input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWCgKM3uTTAgLP/83pDwLfwaTTAQKNguzjCAKt98LeCwLZh62pDwKKqdGpBwLd2q7jAwKa+5aMBAL5zb65C42zY4GBEUKujhjtZ/hZ8sLESfiF" />
</div></form>
</body>
</html>
You could post a variable to the HTML page to go through the paging.
string lcUrl = "http://www.mysite.com/page.aspx";
HttpWebRequest loHttp =
(HttpWebRequest) WebRequest.Create(lcUrl);
// *** Send any POST data
string lcPostData =
"gvEmployees=" + HttpUtility.UrlEncode("Page$2");
loHttp.Method="POST";
byte [] lbPostBuffer = System.Text.
Encoding.GetEncoding(1252).GetBytes(lcPostData);
loHttp.ContentLength = lbPostBuffer.Length;
Stream loPostData = loHttp.GetRequestStream();
loPostData.Write(lbPostBuffer,0,lbPostBuffer.Length);
loPostData.Close();
HttpWebResponse loWebResponse = (HttpWebResponse) loHttp.GetResponse();
Encoding enc = System.Text.Encoding.GetEncoding(1252);
StreamReader loResponseStream =
new StreamReader(loWebResponse.GetResponseStream(),enc);
string lcHtml = loResponseStream.ReadToEnd();
loWebResponse.Close();
loResponseStream.Close();
Then parse out the data you need from the string.
--EDIT--
Here is what I would try (something similar) where all of the post data is sent:
string lcPostData =
"__EVENTTARGET" + HttpUtility.UrlEncode("gvEmployees"); &
"__EVENTARGUMENT" + HttpUtility.UrlEncode("Page%242"); &
"__VIEWSTATE" + HttpUtility.UrlEncode("<Value of _Viewstate>");
You open the fiddler and open the second page of asp.net website table.Go to webforms tab in Fiddler for that particular page session and check in the body what are the variables are posting.Concat all variables in same sequence format and post data using HttpWebRequest.
In my case it was:
string PostData = "__EVENTTARGET="
+ HttpUtility.UrlEncode("ctl00$ContentPlaceHolder2$grdDirectory")
+ "&"
+ "__EVENTARGUMENT="+HttpUtility.UrlEncode("Page$2")
+ "&"
+ "__VIEWSTATE="+ HttpUtility.UrlEncode(view_state)
+ "&"
+ "__VIEWSTATEGENERATOR="
+ HttpUtility.UrlEncode(viewstategenerator)
+ "&"
+ "__VIEWSTATEENCRYPTED="
+ HttpUtility.UrlEncode(viewstateencrypted)
+ "&"
+ "__EVENTVALIDATION=" + HttpUtility.UrlEncode(eventvalidation);
Hope It will work.

Categories

Resources