This question already has answers here:
What is the best way to parse html in C#? [closed]
(15 answers)
Closed 9 years ago.
Is there a way to parse HTML string in .Net code behind like DOM parsing...
i.e. GetElementByTagName("abc").GetElementByTagName("tag")
I've this code chunk...
private void LoadProfilePage()
{
string sURL;
sURL = "http://www.abcd1234.com/abcd1234";
WebRequest wrGETURL;
wrGETURL = WebRequest.Create(sURL);
//WebProxy myProxy = new WebProxy("myproxy",80);
//myProxy.BypassProxyOnLocal = true;
//wrGETURL.Proxy = WebProxy.GetDefaultProxy();
Stream objStream;
objStream = wrGETURL.GetResponse().GetResponseStream();
if (objStream != null)
{
StreamReader objReader = new StreamReader(objStream);
string sLine = objReader.ReadToEnd();
if (String.IsNullOrEmpty(sLine) == false)
{
....
}
}
}
You can use the excellent HTML Agility Pack.
This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams).
Take a look at using the Html Agility Pack
Example of its use:
HtmlDocument doc = new HtmlDocument();
doc.Load("file.htm");
foreach(HtmlNode link in doc.DocumentNode.SelectNodes("//a[#href]")
{
HtmlAttribute att = link["href"];
att.Value = FixLink(att);
}
You can use the HTML Agility Pack and a little XPath (it can even download the document for you):
HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("http://www.abcd1234.com/abcd1234");
HtmlNodeCollection tags = doc.DocumentNode.SelectNodes("//abc//tag");
I've used the HTML Agility Pack to do this exact thing and I think it's great. It has been really helpful to me.
maybe this can help: What is the best way to parse html in C#?
Related
I am trying to scrape a website and get a URL from it, I am using htmlagilitypack and the code below:
HtmlWeb hw = new HtmlWeb();
HtmlAgilityPack.HtmlDocument doc = hw.Load("http://putlocker.ist/scorpion-season-1-episode-1/");
foreach (HtmlNode link in doc.DocumentNode.SelectNodes("//[#id='vplayer_media']/video"))
{
string whatever = link.OuterHtml;
textBox1.Text = whatever;
}
I used opera's developer tools to copy the XPath which came out as this:
//*[#id="vplayer_media"]/video
I couldn't use it because of double quotes so I replaced it with
#"//*[#id=""vplayer_media""]/video"
but I get the error:
Object reference not set to an instance of an object
What am I doing wrong?
Escape the double-quotes in your xpath:
"//[#id=\"vplayer_media\"]/video"
Or use double-double-quotes with a 'literal'
#"//[#id=""vplayer_media""]/video"
got a trouble with HtmlAgilityPack. I can't parse Cyrillic text, it's appears as some unknown symbols.
HtmlWeb webGet = new HtmlWeb();
webGet.OverrideEncoding = Encoding.UTF8;
HtmlAgilityPack.HtmlDocument doc = webGet.Load("http://vk.com/glitchhop");
HtmlNode myNode = doc.DocumentNode.SelectSingleNode("//div[#id='page_wall_posts']/*[2]//div[#class='wall_post_text']");
if (myNode != null)
return myNode.InnerText;
else return "Nothing found";
Also attach example of error and how that text should be looks like
This problem is not related to HTMLAgilityPack, it is caused by incorrect encoding you're using.
Webpage you're trying to parse is encoded using windows-1251 encoding.
So changing webGet.OverrideEncoding from Encoding.UTF8 to Encoding.GetEncoding(1251) should help you.
Well my program is reading a web target that somewhere in the body there is the iframe that i want to read.
My html source
<html>
...
<iframe src="http://www.mysite.com" ></iframe>
...
</html>
in my program i have a method that is returning the source as a string
public static string get_url_source(string url)
{
using (WebClient client = new WebClient())
{
return client.DownloadString(url);
}
}
My problem is that i want to get the source of the iframe when it's reading the source, as it would do in normal browsing.
Can i do this only by using WebBrowser Class or there is a way to do it within WebClient or even another class?
The real question:
How can i get the outer html given a url? Any appoach is welcomed.
After getting the source of the site, you can use HtmlAgilityPack to get the url of the iframe
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);
var src = doc.DocumentNode.SelectSingleNode("//iframe")
.Attributes["src"].Value;
then make a second call to get_url_source
Parse your source using HTML Agility Pack and then:
List<String> iframeSource = new List<String>();
HtmlDocument doc = new HtmlDocument();
doc.Load(url);
foreach (HtmlNode node in doc.DocumentElement.SelectNodes("//iframe"))
iframeSource.Add(get_url_source(mainiFrame.Attributes["src"]));
If you are targeting a single iframe, try to identify it using ID attribute or something else so you can only retrieve one source:
String iframeSource;
HtmlDocument doc = new HtmlDocument();
doc.Load(url);
foreach (HtmlNode node in doc.DocumentElement.SelectNodes("//iframe"))
{
// Just an example for check, but you could use different approaches...
if (node.Attributes["id"].Value == 'targetframe')
iframeSource = get_url_source(node.Attributes["src"].Value);
}
Well i found the answer after some search and this is what i wanted
webBrowser1.Url = new Uri("http://www.mysite.com/");
while (webBrowser1.ReadyState != WebBrowserReadyState.Complete) Application.DoEvents();
string InnerSource = webBrowser1.Document.Body.InnerHtml;
//You can use here OuterHtml too.
I am using the Html Agility pack. When the Load method of HtmlDocument class is passed the URL like "http://www.stackoverflow.com" it says the URI is not in correct format.
doc.Load(TextBoxUrl.Text, Encoding.UTF8 );
the url I try is this http://www.stackoverflow.com/questions/846994/how-to-use-html-agility-pack
HAP can not load from url, only from file or from a string. Use WebClient or HttpWebRequest to get the page.
For example:
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
using (var wc = new WebClient())
{
doc.LoadHtml(wc.DownloadString(TextBoxUrl.Text));
}
I'm sure this question has asked before and i've looked before I cant find the answer, or maybe I am just doing something wrong.
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(indivdualfix[0]);
HtmlWeb hwObject = new HtmlWeb();
HtmlAgilityPack.HtmlDocument htmldocObject = hwObject.Load(indivdualfix[0]);
HtmlNode body = htmldocObject.DocumentNode.SelectSingleNode("//body");
body.Attributes.Remove("style");
foreach (var a in body.Attributes.ToArray())
a.Remove();
string bodywork = body.InnerHtml.ToString();
The string body still returns all the html coding. I might be missing something really small here. What needs to be doen to remove all the html coding basically.
Use body.InnerText not body.InnerHtml