I'm trying to get the text within pre tags in C# which uses Xpath. The webpage only consists of the following:
<body>
<pre>
The text I am trying to select
</pre>
</body>
I can't seem to select just that text and I don't exactly know how to put that in a string. This is the code I'm using:
var WebgetME2_ = new HtmlWeb();
var docME2_ = WebgetME2_.Load(webpage); //loading the webpage
HtmlNode NODEME2_ = docME2_.DocumentNode.SelectSingleNode("//*/pre"); //select the node
string innerME_ = NODEME2_.InnerText; //put the node innertext in string
// After getting the text within the <pre> tags I want to select a part of it using RegEx, that's why I need it in a string
string imagineME2_ = Regex.Match(innerME_, "(?=http)(.+?)(?<=.jpg)").ToString();
I've found a way to bypass the problem.
System.Net.WebClient WebclientME_ = new System.Net.WebClient();
byte[] rawME_ = WebclientME_.DownloadData(webpage); //download page
string innerME_ = System.Text.Encoding.UTF8.GetString(rawME_); //write to a string
string imagineME2_ = Regex.Match(innerME_, "(?=http)(.+?)(?<=.jpg)").ToString();
It downloads the page which I'm not fond of because it works slower but it works.
Please try below Xpath :-
/body/pre/text()
text() function will retrieve all text for you from the path you have mentioned in the Xpath
pre should be the child of body.. If it is not your real structure then use double slash // .... // means it will find the pre node from your complete DOM.
You can also give a try with below Xpath:-
/body//pre/text()
Hope it will help you :)
I have a variable in my program that contains HTML data as a string. The variable, htmlText, contains something like the following:
<ul><li><u>Mode selector </u></li><li><u>LAND ALT</u></li>
I'd like to iterate through this HTML, using the HtmlAgilityPack, but every example I see tries to load the HTML as a document. I already have the HTML that I want to parse within the variable htmlText. Can someone show me how to parse this, without loading it as a document?
The example I'm looking at right now looks like this:
static void Main(string[] args)
{
var web = new HtmlWeb();
var doc = web.Load("http://www.stackoverflow.com");
var nodes = doc.DocumentNode.SelectNodes("//a[#href]");
foreach (var node in nodes)
{
Console.WriteLine(node.InnerHtml);
}
}
I want to convert this to use my htmlText and find all underline elements within. I just don't want to load this as a document since I already have the HTML that I want to parse stored in a variable.
You can use the LoadHtml method of HtmlDocument class
Document is simply a name, it's not really a document (or doesn't have to be).
var doc = New HtmlAgilityPack.HtmlDocument;
string myHTML = "<ul><li><u>Mode selector </u></li><li><u>LAND ALT</u></li>";
doc.LoadHtml(myHTML);
foreach (var node in doc.DocumentNode.SelectNodes("//a[#href]")) {
Console.WriteLine(node.InnerHtml);
}
I've used this exact same thing to parse html chunks in variables.
So I have an HTML snippet that I want to modify using C#.
<div>
This is a specialSearchWord that I want to link to
<img src="anImage.jpg" />
A hyperlink
Some more text and that specialSearchWord again.
</div>
and I want to transform it to this:
<div>
This is a <a class="special" href="http://mysite.com/search/specialSearchWord">specialSearchWord</a> that I want to link to
<img src="anImage.jpg" />
A hyperlink
Some more text and that <a class="special" href="http://mysite.com/search/specialSearchWord">specialSearchWord</a> again.
</div>
I'm going to use HTML Agility Pack based on the many recommendations here, but I don't know where I'm going. In particular,
How do I load a partial snippet as a string, instead of a full HTML document?
How do edit?
How do I then return the text string of the edited object?
The same as a full HTML document. It doesn't matter.
The are 2 options: you may edit InnerHtml property directly (or Text on text nodes) or modifying the dom tree by using e.g. AppendChild, PrependChild etc.
You may use HtmlDocument.DocumentNode.OuterHtml property or use HtmlDocument.Save method (personally I prefer the second option).
As to parsing, I select the text nodes which contain the search term inside your div, and then just use string.Replace method to replace it:
var doc = new HtmlDocument();
doc.LoadHtml(html);
var textNodes = doc.DocumentNode.SelectNodes("/div/text()[contains(.,'specialSearchWord')]");
if (textNodes != null)
foreach (HtmlTextNode node in textNodes)
node.Text = node.Text.Replace("specialSearchWord", "<a class='special' href='http://mysite.com/search/specialSearchWord'>specialSearchWord</a>");
And saving the result to a string:
string result = null;
using (StringWriter writer = new StringWriter())
{
doc.Save(writer);
result = writer.ToString();
}
Answers:
There may be a way to do this but I don't know how. I suggest
loading the entire document.
Use a combination of XPath and regular
expressions
See the code below for a contrived example. You may have
other constraints not mentioned but this code sample should get you
started.
Note that your Xpath expression may need to be more complex to find the div that you want.
HtmlDocument doc = new HtmlDocument();
doc.Load(yourHtmlFile);
HtmlNode divNode = doc.DocumentNode.SelectSingleNode("//div[2]");
string newDiv = Regex.Replace(divNode.InnerHtml, #"specialSearchWord",
"<a class='special' href='http://etc'>specialSearchWord</a>");
divNode.InnerHtml = newDiv;
Console.WriteLine(doc.DocumentNode.OuterHtml);
How can I select every paragraph in a div tag for example.
<div id="body_text">
<p>Hi</p>
<p>Help Me Please</P>
<p>Thankyou</P>
I have got Html Agility downloaded and referenced in my program, All I need is the paragraphs. There may be a variable number of paragraphs and there are loads of different div tags but I only need the content within the body_text. Then I assume this can be stored as a string which I then want to write to a .txt file for later reference. Thankyou.
The valid XPATH for your case is //div[#id='body_text']/p
foreach(HtmlNode node in yourHTMLAgilityPackDocument.DocumentNode.SelectNodes("//div[#id='body_text']/p")
{
string text = node.InnerText; //that's the text you are looking for
}
Here's a solution that grabs the paragraphs as an enumeration of HtmlNodes:
HtmlDocument doc = new HtmlDocument();
doc.Load("your.html");
var div = doc.GetElementbyId("body_text");
var paragraphs = div.ChildNodes.Where(item => item.Name == "p");
Without explicit Linq:
var paragraphs = doc.GetElementbyId("body_text").Elements("p");
I need to parse through the aspx file (from disk, and not the one rendered on the browser) and make a list of all the server side asp.net controls present on the page, and then create an xml file from it. which would be the best way to do it? Also, are there any available libraries for this?
For eg, if my aspx file contains
<asp:label ID="lbl1" runat="server" Text="Hi"></asp:label>
my xml file would be
<controls>
<ID>lbl1</ID>
<runat>server</runat>
<Text>Hi</Text>
</controls>
Xml parsers wouldn't understand the ASP directives: <%# <%= etc.
You'll probably best to use regular expressions to do this, likely in 3 stages.
Match any tag elements from the entire page.
For Each tag, match the tag and control type.
For Each tag that matches (2), match any attributes.
So, starting from the top, we can use the following regex:
(?<tag><[^%/](?:.*?)>)
This will match any tags that don't have <% and < / and does so lazily (we don't want greedy expressions, as we won't read the content correctly). The following could be matched:
<asp:Content ID="ph_PageContent" ContentPlaceHolderID="ph_MainContent" runat="server">
<asp:Image runat="server" />
<img src="/test.png" />
For each of those captured tags, we want to then extract the tag and type:
<(?<tag>[a-z][a-z1-9]*):(?<type>[a-z][a-z1-9]*)
Creating named capture groups makes this easier, this will allow us to easily extract the tag and type. This will only match server tags, so standard html tags will be dropped at this point.
<asp:Content ID="ph_PageContent" ContentPlaceHolderID="ph_MainContent" runat="server">
Will yield:
{ tag = "asp", type = "Content" }
With that same tag, we can then match any attributes:
(?<name>\S+)=["']?(?<value>(?:.(?!["']?\s+(?:\S+)=|[>"']))+.)["']?
Which yields:
{ name = "ID", value = "ph_PageContent" },
{ name = "ContentPlaceHolderID", value = "ph_MainContent" },
{ name = "runat", value = "server" }
So putting that all together, we can create a quick function that can create an XmlDocument for us:
public XmlDocument CreateDocumentFromMarkup(string content)
{
if (string.IsNullOrEmpty(content))
throw new ArgumentException("'content' must have a value.", "content");
RegexOptions options = RegexOptions.CultureInvariant | RegexOptions.Compiled | RegexOptions.IgnoreCase;
Regex tagExpr = new Regex("(?<tag><[^%/](?:.*?)>)", options);
Regex serverTagExpr = new Regex("<(?<tag>[a-z][a-z1-9]*):(?<type>[a-z][a-z1-9]*)", options);
Regex attributeExpr = new Regex("(?<name>\\S+)=[\"']?(?<value>(?:.(?![\"']?\\s+(?:\\S+)=|[>\"']))+.)[\"']?", options);
XmlDocument document = new XmlDocument();
XmlElement root = document.CreateElement("controls");
Func<XmlDocument, string, string, XmlElement> creator = (document, name, value) => {
XmlElement element = document.CreateElement(name);
element.InnerText = value;
return element;
};
foreach (Match tagMatch in tagExpr.Matches(content)) {
Match serverTagMatch = serverTagExpr.Match(tagMatch.Value);
if (serverTagMatch.Success) {
XmlElement controlElement = document.CreateElement("control");
controlElement.AppendChild(
creator(document, "tag", serverTagMatch.Groups["tag"].Value));
controlElement.AppendChild(
creator(document, "type", serverTagMatch.Groups["type"].Value));
XmlElement attributeElement = document.CreateElement("attributes");
foreach (Match attributeMatch in attributeExpr.Matches(tagMatch.Value)) {
if (attributeMatch.Success) {
attributeElement.AppendChild(
creator(document, attributeMatch.Groups["name"].Value, attributeMatch.Groups["value"].Value));
}
}
controlElement.AppendChild(attributeElement);
root.AppendChild(controlElement);
}
}
return document;
}
The resultant document could look like this:
<controls>
<control>
<tag>asp</tag>
<type>Content</type>
<attributes>
<ID>ph_PageContent</ID>
<ContentPlaceHolderID>ph_MainContent</ContentPlaceHolderID>
<runat>server</runat>
</attributes>
</control>
</controls>
Hope that helps!
I used the below three regular expressions with the above code and it gives me html tags as well. Also I can obtain the value in between opening and closing tags too.
Regex tagExpr = new Regex("(?<tag><[^%/](?:.*?)>[^/<]*)", options);
Regex serverTagExpr = new Regex("<(?<type>[a-z][a-z1-9:]*)[^>/]*(?:/>|[>/])(?<value>[^</]*)", options);
Regex attributeExpr = new Regex("(?<name>\\S+)=[\"']?(?<value>(?:.(?![\"']?\\s+(?:\\S+)=|[>\"']))+.)[\"']?", options);
Func<XmlDocument, string, string, XmlElement> creator = (document, name, value) => {
XmlElement element = document.CreateElement(name);
element.InnerText = value;
the above generic template will work version 3.5 and above.. so if any one using version below that , create function like :
public XmlElement creator(XmlDocument document, string name, string value)
{
XmlElement element = document.CreateElement(name);
element.InnerText = value;
return element;
}
this will work
ASPX files should be valid XML, so maybe XSLT would be a good solution. The W3 Schools site has a good introduction and reference. You could then call this XSLT from a simple program to pick the required file(s).
Alternatively, you could use Linq to XML to load the ASPX file(s) and iterate over the controls in a Linq-style.
if the code for the tag is written in more than one line, we may have an issue in extracting the tag data. to avoid that I have removed the newline characters as below from the source string that we are passing to the above function (content)
string contentRemovedNewLines = Regex.Replace(content, #"\t|\n|\r", "");
then we can use contentRemovedNewLines instead of content.
Above code works as i wanted. one more thing can be added. you can call the above method as shown below and then save as an xml file so, we can check that the expected result is there or not.
XmlDocument xmlDocWithWebContent = CreateDocumentFromMarkup(sourceToRead);
string xmlfileLocation = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location) + "tempXmlOutputFileOfWebSource.xml";
xmlDocWithWebContent.Save(xmlfileLocation);
to do that, we have to have a root element for the xml file
XmlDocument document = new XmlDocument();
XmlNode xmlNode = document.CreateNode(XmlNodeType.XmlDeclaration, "", "");
XmlElement root = document.CreateElement("controls");
document.AppendChild(root);
i used the above fix for that