Develop Reference

Webclient.DownloadString() does not retrieve current whole page

I know there is another question with practically identical title here: Webclient.DownloadString does not retrieve the whole page
But the solution doesn't help me, maybe somebody else have the same problem.
I'm trying to get the html code of this URL:
https://cubebrush.co/?freebies=true
To achieve that, I'm using the following code in C#:
WebClient webClient = new WebClient();
string webString = webClient.DownloadString("https://cubebrush.co/?freebies=true");
But the retrieved html lacks some information, for example, all the button tags inside the website. This can be quickly checked using the library HtmlAgilityPack and checking for all the tags inside the website with the following code:
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(webString);
HashSet<string> hs = new HashSet<string>();
foreach (var dec in doc.DocumentNode.Descendants())
{
hs.Add(dec.Name);
}
If we run this, it will show 26 tags, but none of them will be a button tag. This makes sense, since the initial webString also lacks that "button information".
I've tried to copy webString into a file, to check if, as the initial commented post says, was a problem with the visualizer, but it doesn't, visualizer and file looks exactly equal.
Can somebody tells me what I'm doing wrong? Thanks!

How deep is the visible area of HtmlAgilityPack?

I need to grab some posts from a blog. All went well until I've wanted to get the post creation date. The DOM-tree for it is:
div class="stories-feed__container"
-> article
-> div class="story__main"
-> div class="story__footer"
-> div class="story__user user"
-> div class="user__info-item"
-> time datetime="date and time in UTC format".
So I wrote the code:
HtmlWeb web = new HtmlWeb();
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc = web.Load("https://pikabu.ru/#serhiy1994");
string postDate = doc.DocumentNode.SelectSingleNode("//div[contains(#class, 'stories-feed__container')]/article[2]/div[contains(#class, 'story__main')]/div[contains(#class, 'story__footer')]/div[contains(#class, 'story__user user')]/div[contains(#class, 'user__info-item')]/time").GetAttributeValue("datetime", "NULL"); // e.g. for the 2nd article on the page
And it returns the NullReferenceException.
BUT if you stop at the "div class="story__user user"" level (e.g.,
string postDate = doc.DocumentNode.SelectSingleNode("//div[contains(#class, 'stories-feed__container')]/article[2]/div[contains(#class, 'story__main')]/div[contains(#class, 'story__footer')]/div[contains(#class, 'story__user user')]").InnerHtml;
it works properly and return you the inner HTML-code.
So I think there is something like 'maximum visibility level" for HtmlAgilityPack, and you won't able to manipulate with the deeper markdown.
Am I right or I'm coding something wrong?
The original page code is here: https://pastebin.com/jFC0XD9C

HtmlAgility will scrape the entire website, regardless of how deep you want to go. You can use this to get to the item you are looking for since you dont have to provide the entire path.
This will search the entire site and look for the first <div> tag that has the class name user__info-item. You can also change SelectSingleNode to SelectNodes if there are multiple tags then loop through them to get the dates.
HtmlWeb web = new HtmlWeb();
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc = web.Load("https://pikabu.ru/#serhiy1994");
var postDate = doc.DocumentNode.SelectSingleNode("//div[#class='user__info-item']/time");
Console.WriteLine(postDate.InnerText);
Whats wrong with your code?
Reason the code above you have doesnt work is because there is another div that you are missing, '<div class="user__info user__info_left">'.
If you write your code like this, it works.
var nodes = doc.DocumentNode.SelectSingleNode("//div[#class='story__main']/div[#class='story__footer']/div[#class='story__user user']/div[#class='user__info user__info_left']/div[#class='user__info-item']/time");
Console.WriteLine(nodes.InnerText);
Another way
Another way to do it is by searching for a parent div. Once you find the parent tag, search under that tag to find what you are looking for.
var nodes = doc.DocumentNode.SelectNodes("//div[#class='story__user user']");
foreach (HtmlNode node in nodes)
{
// Search within each node using .// notation
var timeNodes = node.SelectSingleNode(".//div[#class='user__info-item']/time");
Console.WriteLine(timeNodes.InnerText);
}
Tested Code here

c# Html agility pack HtmlDocument does not contain all Elements from the website

at the time I´m making a chatbot. The bot should be able to define a word, so I tried getting the span Element from Google (https://www.google.de/webhp?sourceid=chrome-instant&rlz=1C1CHBD_deDE721DE721&ion=1&espv=2&ie=UTF-8#q=define%20test) where the definition is writen in, wich didn't work. It turns out that the htmlDocument does not contain the hole website.
string Url = "https://www.google.de/webhp?sourceid=chrome- instant&rlz=1C1CHBD_deDE721DE721&ion=1&espv=2&ie=UTF-8#q=define%20test";
HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load(Url);
HtmlNode node = doc.DocumentNode.SelectSingleNode("//*[#id='uid_0']/div[1]/div/div[1]/div[2]/div/ol/li[1]/div/div/div[2]/div/div[1]/span");
if (!String.IsNullOrEmpty(node.InnerText))
output += node.InnerText;
node is not set to an Instance of an object
I let me give the InnerHtml of the document and put it in a gist: https://gist.github.com/MarcelBulpr/bb44a527d8202eb7fffb4e21fb8b4fed
it seems that the website does not load the result of the search request.
Does anyone know how to work around this?
Thanks in advance

C# htmlAgility Webscrape html node inside the first Html node

I am trying to access these nodes
on this website.
http://bcres.paragonrels.com/publink/default.aspx?GUID=2033c143-cdf1-46b4-9aac-2e27371de22d&Report=Yes
however they appear to be in a secondary Html document within the initial one.
I am confused how I access the secondary html path and then parse through for the
this is an example of one of the nodes.
<div style="top:219px;left:555px;width:45px;height:14px;" id="" class="mls29">2</div>
I am using htmlAgility pack and I recieve null whenever I try to access Div.
I tried working my way down the nodes but It didn't work.
Any help or a place to look up the necessary information to figure this out would be appreciated
var webGet = new HtmlWeb();
var document = webGet.Load("http://bcres.paragonrels.com/publink/default.aspx?GUID=d27a1d95- 623d-4f6a-9e49-e2e46ede136c&Report=Yes");
var divTags = document.DocumentNode.SelectNodes("/html");
var text = document.DocumentNode.InnerText;
MessageBox.Show(text);

You will be able to scrape the data if you access the following url:
http://bcres.paragonrels.com/publink/Report.aspx?outputtype=HTML&GUID=2033c143-cdf1-46b4-9aac-2e27371de22d&ListingID=262103824:0&Report=Yes&view=29&layout_id=63
HtmlWeb w = new HtmlWeb();
var hd = w.Load("http://bcres.paragonrels.com/publink/Report.aspx?outputtype=HTML&GUID=2033c143-cdf1-46b4-9aac-2e27371de22d&ListingID=262103824:0&Report=Yes&view=29&layout_id=63");
var presentedBy = hd.DocumentNode.CssSelect(".mls23.at-phone-link");
if (presentedBy != null)
{
Console.WriteLine(presentedBy.FirstOrDefault().InnerText);
}
As an example, scraping the Presented By field:
Remarks:
I use ScrapySharp nuget package along with HtmlAgilityPack, so I can scrape using css selectors instead of xpath expressions - something I find easier to do.
The url you are scraping from is your problem. I am scraping from the last get request that is performed after the page is loaded, as you can see in the screenshot below, using Firefox developer tools to analyze the site traffic/network requests/responses:
I could not yet identify who/what triggers this http request in the end (may be by javascript code, may be via one of the frame htmls that are requested in the main document (the frame-enabled one).
If you only have a couple of urls like this to scrape, then even manually extracting the correct url will be an option.

Trouble Scraping .HTM File

I have just begun scraping basic text off web pages, and am currently using the HTMLAgilityPack C# library. I had some success with boxscores off rivals.yahoo.com (sports is my thing so why not scrape something interesting?) but I am stuck on NHL's game summary pages. I think this is kind of an interesting problem so I would post it here.
The page I am testing is:
http://www.nhl.com/scores/htmlreports/20102011/GS020079.HTM
Upon first glance, it seems like basic text with no ajax or stuff to mess up a basic scraper. Then I realize I can't right click due to some javascript, so I work around that. I right click in firefox and get the xpath of the home team using XPather and I get:
/html/body/table[#id='MainTable']/tbody/tr[1]/td/table[#id='StdHeader']/tbody/tr/td/table/tbody/tr/td[3]/table[#id='Home']/tbody/tr[3]/td
When I try to grab that node / inner text, htmlagilitypack won't find it. Does anyone see anything strange in the page's source code that might be stopping me?
I am new to this and still learning how people might stop me from scraping, any tips or tricks are gladly appreciated!
p.s. I observe all site rules regarding bots, etc, but I noticed this strange behavior and saw it as a challenge.

Ok so it appears that my xpaths have tbody's in them. When I remove these tbodys manually from the xpath, HTMLAgilityPack can handle it fine.
I'd still like to know why I am getting invalid xpaths, but for now I have answered my question.

I think unless my xpath knowledge is heaps flawed(probably) the problem is with the /tbody node in your xpath expression.
When I do
string test = string.Empty;
StreamReader sr = new StreamReader(#"C:\gs.htm");
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.Load(sr);
sr.Close();
sr = null;
string xpath = #"//table[#id='Home']/tr[3]/td";
test = doc.DocumentNode.SelectSingleNode(xpath).InnerText;
That works fine.. returns a
"COLUMBUS BLUE JACKETSGame 5 Home Game 3"
which I hope is the string you wanted.
Examining the html I couldn't find a /tbody.