Develop Reference

HtmlDocument get a incomplete page

I am doing a small web scraping project, and I am having a problem with the function that takes the html code. The web that I inspect in the browser is different from the web that downloads the method (for the same URL).
I have tried to improve the coding process, but to no avail. The same thing happens for "i=2".
static void Main(string[] args)
{
string prefixurl = "https://www.aaabbbcccdddeee.de/en/do-business-with-finland/finnish-suppliers/finnish-suppliers-results?query=africa";
for (int i = 1; i < 18; i++)
{
string url = prefixurl;
if (i > 1)
{
url = prefixurl + "&page=" + i;
}
var doc = GetDocument(url);
var links = GetBusinessLinks(url);
List<Empresa> empresas = GetBusiness(links);
Export(empresas);
}
}
static List<string> GetBusinessLinks(string url)
{
var doc = GetDocument(url);
var linkNodes = doc.DocumentNode.SelectNodes("/html/body/section/div/div/div/div[2]/div[2]//a");
// //a[#class=\"btn bf-ghost-button\"]
var baseUri= new Uri(url);
var links = new List<string>();
//The problem its there, in the incomplete page the program haven't found nodes
foreach (var node in linkNodes)
{
var link = node.Attributes["href"].Value;
bool business = link.Contains("companies");
if (business)
{
link = new Uri(baseUri, link).AbsoluteUri;
links.Add(link);
}
}
return links;
}
static HtmlDocument GetDocument(string url)
{
var web = new HtmlWeb();
HtmlDocument doc = new HtmlDocument()
{
OptionDefaultStreamEncoding = Encoding.UTF8
};
doc = web.Load(url);
return doc;
}
´´´

Your suggestion has made me suspect where I should continue looking, thanks.
I have used PupperSharp in non-headless mode.
https://betterprogramming.pub/web-scraping-using-c-and-net-d99a085dace2

Extract email address from a website for each link inside DOM of page

I Want to develope an app I give Url of a specific website to it,and it extract all links from that Web page. For each extracted link I want to get the HTML content. I am based in the concept of deep crawling.
My purpose is to get all email addresses of website. Below is my source code:
static string ExtractEmails(string data)
{
//instantiate with this pattern
Regex emailRegex = new Regex(#"\w+([-+.]\w+)*#\w+([-.]\w+)*\.\w+([-.]\w+)*", RegexOptions.IgnoreCase);
//find items that matches with our pattern
MatchCollection emailMatches = emailRegex.Matches(data);
//StringBuilder sb = new StringBuilder();
string s = "";
foreach (Match emailMatch in emailMatches)
{
//sb.AppendLine(emailMatch.Value);
s += emailMatch.Value + ",";
}
return s;
}
static readonly List<ParsResult> _results = new List<ParsResult>();
static Int32 _maxDepth = 4;
static String Foo(String urlToCheck = null, Int32 depth = 0, ParsResult parent = null)
{
string email = "";
if (depth >= _maxDepth) return email;
String html;
using (var wc = new WebClient())
html = wc.DownloadString(urlToCheck ?? parent.Url);
var doc = new HtmlDocument();
doc.LoadHtml(html);
var aNods = doc.DocumentNode.SelectNodes("//a");
if (aNods == null || !aNods.Any()) return email;
foreach (var aNode in aNods)
{
var url = aNode.Attributes["href"];
if (url == null)
continue;
var wc2 = new WebClient();
String html2 = wc2.DownloadString(url.Value);
email = ExtractEmails(html2);
Console.WriteLine(email);
var result = new ParsResult
{
Depth = depth,
Parent = parent,
Url = url.Value
};
_results.Add(result);
Console.WriteLine("{0} - {1}", depth, result.Url);
Foo(depth: depth + 1, parent: result);
return email;
}
return email;
}
static void Main(string[] args)
{
String res = Foo("http://www.mobileridoda.com", 0);
Console.WriteLine("emails " + res);
}
I want to dispaly in console all emails extracted by all pages of all links that are inside DOM of Main page, But it dispalys no emails in console. How can I solve this issue ?
Thank you

Found a few things wrong but no worries, got the details on why and what to do to fix them.
In your foreach loop, when you go through the first URL, you are using a return statement at the end essentially breaking the loop and terminating. Use return only after you have processed ALL the URLs and accumulated the email addresses.
You are overwriting the email (i see it as a csv) when you go over the loop. Use += to continue adding. email = ExtractEmails(html2);
You are not returning anything when you call Foo within your forEach loop. You need to use email += Foo(xyz). Foo(depth: depth + 1, parent: result);
You are going through a URL that you have already processed... possibly causing an infinite cycle. I added a list of strings that keeps track of URLs you have already visited so as to prevent the infinite loop you might get into.
Here is a complete working solution.
static string ExtractEmails(string data)
{
//instantiate with this pattern
Regex emailRegex = new Regex(#"\w+([-+.]\w+)*#\w+([-.]\w+)*\.\w+([-.]\w+)*", RegexOptions.IgnoreCase);
//find items that matches with our pattern
MatchCollection emailMatches = emailRegex.Matches(data);
//StringBuilder sb = new StringBuilder();
string s = "";
foreach (Match emailMatch in emailMatches)
{
//sb.AppendLine(emailMatch.Value);
s += emailMatch.Value + ",";
}
return s;
}
static readonly List<ParsResult> _results = new List<ParsResult>();
static Int32 _maxDepth = 4;
static List<string> urlsAlreadyVisited = new List<string>();
static String Foo(String urlToCheck = null, Int32 depth = 0, ParsResult parent = null)
{
if (urlsAlreadyVisited.Contains(urlToCheck))
return string.Empty;
else
urlsAlreadyVisited.Add(urlToCheck);
string email = "";
if (depth >= _maxDepth) return email;
String html;
using (var wc = new WebClient())
html = wc.DownloadString(urlToCheck ?? parent.Url);
var doc = new HtmlDocument();
doc.LoadHtml(html);
var aNods = doc.DocumentNode.SelectNodes("//a");
if (aNods == null || !aNods.Any()) return email;
// Get Distinct URLs from all the URls on this page.
List<string> allUrls = aNods.ToList().Select(x => x.Attributes["href"].Value).Where(url => url.StartsWith("http")).Distinct().ToList();
foreach (string url in allUrls)
{
var wc2 = new WebClient();
try
{
email += ExtractEmails(wc2.DownloadString(url));
}
catch { /* Swallow Exception ... URL not found or other errors. */ continue; }
Console.WriteLine(email);
var result = new ParsResult
{
Depth = depth,
Parent = parent,
Url = url
};
_results.Add(result);
Console.WriteLine("{0} - {1}", depth, result.Url);
email += Foo(depth: depth + 1, parent: result);
}
return email;
}
public class ParsResult
{
public int Depth { get; set; }
public ParsResult Parent { get; set; }
public string Url { get; set; }
}
// ========== MAIN CLASS ==========
static void Main(string[] args)
{
String res = Foo("http://www.mobileridoda.com", 0);
Console.WriteLine("emails " + res);
}

HtmlAgilityPack modify html and return updated content

I have not used the HtmlAgilityPack often and I'm stuck on the following issue.
I'm checking to see if the browser supports WebP, if yes I then append a new parameter to the src of the image.
I have that working, but I cannot work out how to return the updated HTML, any help will be appreciated.
public static HtmlString AppendWebPString(HtmlString htmlText)
{
bool browserSupportsWebP = BrowserSupportsWebPHelper.WebPSupported();
if (!browserSupportsWebP) return htmlText;
var h = new HtmlDocument();
h.LoadHtml(htmlText.ToString());
const string webP = "&quality=80&format=webp";
if (h.DocumentNode.SelectNodes("//img[#src]") == null) return htmlText;
string imgOuterHtml = string.Empty;
foreach (HtmlNode image in h.DocumentNode.SelectNodes("//img[#src]"))
{
var src = image.Attributes["src"].Value.Split('&');
image.SetAttributeValue("src", src[1] + string.Format(webP));
imgOuterHtml = image.OuterHtml;
}
//How do I return the updated html here
return new HtmlString(h.ParsedText);
}

Ok, I could not find anything that was built into the agility pack to do what I wanted.
I have managed to achieve what I was after using the code below
public static HtmlString AppendWebPString(HtmlString htmlText)
{
bool browserSupportsWebP = BrowserSupportsWebPHelper.WebPSupported();
if (!browserSupportsWebP) return htmlText;
var h = new HtmlDocument();
h.LoadHtml(htmlText.ToString());
const string webP = "&quality=80&format=webp";
if (h.DocumentNode.SelectNodes("//img[#src]") == null) return htmlText;
string modifiedHtml = htmlText.ToString();
List<ReplaceImageValues> images = new List<ReplaceImageValues>();
foreach (HtmlNode image in h.DocumentNode.SelectNodes("//img[#src]"))
{
var src = image.Attributes["src"].Value.Split('&');
string oldSrcValue = image.OuterHtml;
image.SetAttributeValue("src", src[0] + src[1] + string.Format(webP));
string newSrcValue = image.OuterHtml;
images.Add(new ReplaceImageValues(oldSrcValue,newSrcValue));
}
foreach (var newImages in images)
{
modifiedHtml = modifiedHtml.Replace(newImages.OldVal, newImages.NewVal);
}
return new HtmlString(modifiedHtml);
}

C# Webscraper to grab amount of Google Results given a specific search term

I've been working on a webscraper as a Windows Forms application in C#. The user enter a search term and the term and the program will then split the search string for each individual words and look up the amount of search results through Yahoo and Google.
My issue lies with the orientation of the huge HTML document. I've tried multiple approaches such as
iterating recursively and comparing ids aswell as with lamba and the Where statements. Both results in null. I also manually looked into the html document to make sure the id of the div I want exist in the document.
The id I'm looking for is "resultStats" but it is suuuuuper nested. My code looks like this:
using HtmlAgilityPack;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
namespace WebScraper2._0
{
public class Webscraper
{
private string Google = "http://google.com/#q=";
private string Yahoo = "http://search.yahoo.com/search?p=";
private HtmlWeb web = new HtmlWeb();
private HtmlDocument GoogleDoc = new HtmlDocument();
private HtmlDocument YahooDoc = new HtmlDocument();
public Webscraper()
{
Console.WriteLine("Init");
}
public int WebScrape(string searchterms)
{
//Console.WriteLine(searchterms);
string[] ssize = searchterms.Split(new char[0]);
int YahooMatches = 0;
int GoogleMatches = 0;
foreach (var term in ssize)
{
//Console.WriteLine(term);
var y = web.Load(Yahoo + term);
var g = web.Load(Google + term + "&cad=h");
YahooMatches += YahooFilter(y);
GoogleMatches += GoogleFilter(g);
}
Console.WriteLine("Yahoo found " + YahooMatches.ToString() + " matches");
Console.WriteLine("Google found " + GoogleMatches.ToString() + " matches");
return YahooMatches + GoogleMatches;
}
//Parse to get correct info
public int YahooFilter(HtmlDocument doc)
{
//Look for node with correct ID
IEnumerable<HtmlNode> nodes = doc.DocumentNode.Descendants().Where(n => n.HasClass("mw-jump-link"));
foreach (var item in nodes)
{
// displaying final output
Console.WriteLine(item.InnerText);
}
//TODO: Return search resultamount.
return 0;
}
int testCounter = 0;
string toReturn = "";
bool foundMatch = false;
//Parse to get correct info
public int GoogleFilter(HtmlDocument doc)
{
if (doc == null)
{
Console.WriteLine("Null");
}
foreach (var node in doc.DocumentNode.ChildNodes)
{
toReturn += Looper(node, testCounter, toReturn, foundMatch);
}
Console.WriteLine(toReturn);
/*
var stuff = doc.DocumentNode.Descendants("div")
.Where(node => node.GetAttributeValue("id", "")
.Equals("extabar")).ToList();
IEnumerable<HtmlNode> nodes = doc.DocumentNode.Descendants().Where(n => n.HasClass("appbar"));
*/
return 0;
}
public string Looper(HtmlNode node, int counter, string returnstring, bool foundMatch)
{
Console.WriteLine("Loop started" + counter.ToString());
counter++;
Console.WriteLine(node.Id);
if (node.Id == "resultStats")
{
returnstring += node.InnerText;
}
foreach (HtmlNode n in node.Descendants())
{
Looper(n, counter, returnstring, foundMatch);
}
return returnstring;
}
}
}

I made an google HTML Scraper a few weeks ago, a few things to consider
First: Google don't like when you try to Scrape their Search HTML, while i was running a list of companies trying to get their addresses and phone number, Google block my IP from accessing their website for a little bit (Which cause a hilarious panic in the office)
Second: Google will change the HTML (Id names and etc) of the page so using ID's won't work, on my case i used the combination of HTML Tags and specific information to parse the response and extract the information that i wanted.
Third: It's better to just use their API to grab the information you need, just make sure you respect their free tier query limit and you should be golden.
Here is the Code i used.
public static string getBetween(string strSource, string strStart, string strEnd)
{
int Start, End;
if (strSource.Contains(strStart) && strSource.Contains(strEnd))
{
Start = strSource.IndexOf(strStart, 0) + strStart.Length;
End = strSource.IndexOf(strEnd, Start);
return strSource.Substring(Start, End - Start);
}
else
{
return "";
}
}
public void SearchResult()
{
//Run a Google Search
string uriString = "http://www.google.com/search";
string keywordString = "Search String";
WebClient webClient = new WebClient();
NameValueCollection nameValueCollection = new NameValueCollection();
nameValueCollection.Add("q", keywordString);
webClient.QueryString.Add(nameValueCollection);
string result = webClient.DownloadString(uriString);
string search = getBetween(result, "Address", "Hours");
rtbHtml.Text = getBetween(search, "\">", "<");
}
On my case i used the String Address and Hours to limit what information i wanted to extract.
Edit: Fixed the Logic and added the Code i used.
Edit2: forgot to add the GetBetween Class. (sorry it's my first Answer)

The name 'str' does not exist in the current context

I have declared a class variable in here
void downloader_OpenReadCompleted(object sender, OpenReadCompletedEventArgs e)
{
if (e.Error == null)
{
Stream responseStream = e.Result;
StreamReader responseReader = new StreamReader(responseStream);
string response = responseReader.ReadToEnd();
string[] split1 = Regex.Split(response, "},{");
List<string> pri1 = new List<string>(split1);
pri1.RemoveAt(0);
string last = pri1[pri1.Count() - 1];
pri1.Remove(last);
}
}
and I want to use the class variable str in this method
void AddPrimaryMarkerGraphics(object sender, getPrimaryListCompletedEventArgs e)
{
List<PrimaryClass> primaryList = new List<PrimaryClass>(e.Result);
PrimaryClass sc = new PrimaryClass();
for (int a = 0; a <= e.Result.Count - 1; a++)
{
string schname = e.Result.ElementAt(a).PrimarySchool;
string tophonour = e.Result.ElementAt(a).TopHonour;
string cca = e.Result.ElementAt(a).Cca;
string topstudent = e.Result.ElementAt(a).TopStudent;
string topaggregate = e.Result.ElementAt(a).TopAggregate;
string topimage = e.Result.ElementAt(a).TopImage;
foreach (string item in str)
{
string abc = "[{" + item + "}]";
byte[] buf = System.Text.Encoding.UTF8.GetBytes(abc);
MemoryStream ms = new MemoryStream(buf);
JsonArray users = (JsonArray)JsonArray.Load(ms);
var members = from member in users
//where member["SEARCHVAL"]
select member;
foreach (JsonObject member in members)
{
string schname = member["SEARCHVAL"];
string axisX = member["X"];
string axisY = member["Y"];
// Do something...
string jsonCoordinateString = "{'Coordinates':[{'X':" + axisX + ",'Y':" + axisY + "}]}";
CustomCoordinateList coordinateList = DeserializeJson<CustomCoordinateList>(jsonCoordinateString);
GraphicsLayer graphicsLayer = MyMap.Layers["MyGraphicsLayer_Primary"] as GraphicsLayer;
for (int i = 0; i < coordinateList.Coordinates.Count; i++)
{
Graphic graphic = new Graphic()
{
Geometry = new MapPoint(coordinateList.Coordinates[i].X, coordinateList.Coordinates[i].Y),
Symbol = i > 0 ? PrimarySchoolMarkerSymbol : PrimarySchoolMarkerSymbol
};
graphic.Attributes.Add("PrimarySchool", schname);
graphic.Attributes.Add("xcoord", axisX);
graphic.Attributes.Add("ycoord", axisY);
graphicsLayer.Graphics.Add(graphic);
}
}
}
}
}
That's where the error shows.

You've almost certainly declared the variable in a method (i.e. as a local variable), instead of directly in the class itself (as an instance variable). For example:
// Wrong
class Bad
{
void Method1()
{
List<string> str = new List<string>();
}
void Method2()
{
foreach (string item in str)
{
...
}
}
}
// Right
class Good
{
private List<string> str = new List<string>();
void Method1()
{
str = CreateSomeOtherList();
}
void Method2()
{
foreach (string item in str)
{
...
}
}
}
As a side-note: if you're very new to C#, I would strongly recommend that you stop working on Silverlight temporarily, and write some console apps just to get you going, and to teach you the basics. That way you can focus on C# as a language and the core framework types (text, numbers, collections, I/O for example) and then start coding GUIs later. GUI programming often involves learning a lot more things (threading, XAML, binding etc) and trying to learn everything in one go just makes things harder.

It doesn't work because str is not declared in the other variable. It's sscopong problem. Can you pass str as an input to the other function?