Develop Reference

scraping data from website with a C# console application

I'm trying to learn Spanish and making some flash cards (for my personal use) to help me learn the verbs.
Here is an example, page example. So near the top of the page you will see the past participle: bloqueado & gerund: bloqueando. It is these two values that I wish to obtain in my code and use for my flash cards.
If this is possible I will use a C# console application. I am aware that scraping data from a website is not ideal however this is a once off.
Any guidance on how to start something like this and pitfalls to avoid would be very helpful!

I know this isn't an exact answer, but here is the process I would suggest.
https://www.gnu.org/software/wget/ and mirror the website to a
folder. Wget is a web spider and will follow the links on the site until it has downloaded everything. You'll have to run it with a few different parameters until you figure out the correct settings you want.
Use C# to run through each file in the folder and extract the
words from <section class="verb-mood-section"> in each file. It's your choosing of whether you want to output them to the console or store them in a database or flat file.
Should be that easy, in theory.

Use SGMLReader. SGMLReader is a versatile and robust component that will stream HTML to an XMLReader:
XmlDocument FromHtml(TextReader reader) {
// setup SgmlReader
Sgml.SgmlReader sgmlReader = new Sgml.SgmlReader();
sgmlReader.DocType = "HTML";
sgmlReader.WhitespaceHandling = WhitespaceHandling.All;
sgmlReader.CaseFolding = Sgml.CaseFolding.ToLower;
sgmlReader.InputStream = reader;
// create document
XmlDocument doc = new XmlDocument();
doc.PreserveWhitespace = true;
doc.XmlResolver = null;
doc.Load(sgmlReader);
return doc;
}
You can see that you need to create a TextReader first. TThis would in reality be a StreamReader as a TextReader is an abstract class.
Then you create the XMLDocument over that. Once you've got it into the XMLDocument you can use the various methods supported by XMLDocument to isolate and extract the nodes you need. I'll leave you to explore that aspect of it.
You might try using the XDocument class as it's a lot easier to handle than the XMLDocument, especially if you're a newbie. It also supports LINQ.

Read an XML file from http address

I need to read an xml file using c#/.net from a source like so: https://10.1.12.15/xmldata?item=all
That is basically just an xml file.
StreamReader does not like that.
What's the best way to read the contents of that link?
The file looks like so:
- <RIMP>
- <HSI>
<SBSN>CZ325000123</SBSN>
<SPN>ProLiant DL380p Gen8</SPN>
<UUID>BBBBBBGGGGHHHJJJJ</UUID>
<SP>1</SP>
<cUUID>0000-000-222-22222-333333333333</cUUID>
- <VIRTUAL>...

You'll want to use LINQ to XML to process the XML file. The XDocument.Load Method supports loading an XML document from an URI:
var document = XDocument.Load("https://10.1.12.15/xmldata?item=all");

Another way to do this is using the XmlDocument class. A lot of servers around the world are still running .Net Framework < 3.0 so it's good to know that this class still exists alongside XDocumentin case you're developing an application that will be run on a server.
string url = #"https://10.1.12.15/xmldata?item=all";
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(url);

Maybe the correct answer must starting by reading the initial question about how to "Read an XML file from a URL (or in this case from a Http address)".
I think that can be the best for you see the next easy demos:
(In this case XmlTextReader but today you can use XmlReader instead of XmlTextReader)
http://support.microsoft.com/en-us/kb/307643
(Parallel you could read this documentation too).
https://msdn.microsoft.com/en-us/library/system.xml.xmlreader(v=vs.110).aspx
regards

Is there a Java equivalent for XmlDocument.LoadXml() from .NET?

In .NET C#, when trying to load a string into xml, you need to use XmlDocument type from System.Xml and do the following:
e.g:
string xmlStr = "<name>Oscar</name>";
XmlDocument doc = new XmlDocument();
doc.LoadXml(xmlStr);
Console.Write(doc.OuterXml);
This seems simple but how can I do this in Java? Is it possible to load a string into xml using something directly, short and simple like above and avoid implementing other methods for this?
Thanks in advance.

Try this:
DocumentBuilderFactory documentBuildFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder doccumentBuilder = documentBuildFactory.newDocumentBuilder();
Document document =
doccumentBuilder.parse(new ByteArrayInputStream("<name>Oscar</name>".getBytes()));
You can traverse Oscar by:
String nodeText = document.getChildNodes().item(0).getTextContent() ;
System.out.println(nodeText);
To transaform back:
TransformerFactory tFactory = TransformerFactory.newInstance();
Transformer transformer = tFactory.newTransformer();
DOMSource domSource = new DOMSource(document);
//to print the string in sysout, System.out
StreamResult streamResult = new StreamResult(System.out);
transformer.transform(domSource, streamResult );
To get the result in String:
DOMSource source = new DOMSource(document);
ByteArrayOutputStream outStream = new ByteArrayOutputStream();
StreamResult result = new StreamResult(outStream);
transformer.transform(source, result);
String resultString = new String( outStream.toByteArray());
System.out.println(resultString);

You have a choice of tree models in Java - DOM, XOM, JDOM, DOM4J. Many people (including Singh above) use DOM by default because it is included in the JDK, but it's probably the worst of the bunch, largely because it's the oldest (it was invented before namespaces came along), and because it tries to do too much (HTML, event handling etc, as well as XML). I'd suggest using JDOM2. It shouldn't be hard if you look at the Javadoc for you to find the method that builds a JDOM2 document from an input stream.

Java has a ton of libraries for working with XML. In addition to the many classes that work with XML included with the standard Java installation, there are lots of other open source libraries available. Here are a few options. Take a look at the docs and see which one meets your needs:
The XStream Library is very fast and easy to use. I've used it for XML serialization and I'm very happy with it. If you would rather not use an extrenal library, then try
the javax.xml.parsers.DocumentBuilder class that is demonstrated here

If you want to parse a String str to Document in Java, you can do following:
DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
DocumentBuilder docBuilder = dbf.newDocumentBuilder();
InputStream is = new ByteArrayInputStream(str.getBytes("UTF-8"));
return docBuilder.parse(is);

Display xml data into html

I have a xml structure stored in XDocument.
I want to present as html document (or something similar) , main idea that a web browser will be able to present it .
Does XSLT will right technology here ?
Is there some examples for how to do so ?
Thansk for help.

Yes, XSLT is good for this. I recently had to do this using the following code:
var xslt = new XslCompiledTransform(true);
xslt.Load(styleSheetFile, XsltSettings.TrustedXslt, new XmlUrlResolver());
xslt.Transform(xmlFile, outputFile);

You can use XSLT or LinqToXML. Many examples out there but you can start # http://msdn.microsoft.com/en-us/library/bb387098.aspx

XSLT and XML Question

I have this kinda interesting requirement.
Typically you use XSLT to transform an XML document. The transformed HTML is viewable in a web browser, this works just great. I am also guessing the browser handles the transformation in memory, because if you view the page source of an xml document with XSLT, you don't see html, only xml.
What I would like to do is the following.
using c#
grab an xml file from the fileSystem....
Load it into some framework object
attach an XSLT stylesheet
output the rendered HTML back to an html file on the file system.
Is this possible.
I don't expect a full answer on the entire solution. Just a push in the right direction would be great :) thanks in advance.

You can use System.Xml.Xsl to do XSLT in C#.
There's an article here: XML transformation using Xslt in C# that explains how - here's the core of it:
XPathDocument myXPathDoc = new XPathDocument(<xml file path>);
XslTransform myXslTrans = new XslTransform();
myXslTrans.Load(<xsl file path>);
XmlTextWriter myWriter = new XmlTextWriter("result.html", null);
myXslTrans.Transform(myXPathDoc, null, myWriter);
(Edit: Note to #John: that code illustrates the basic idea. It doesn't pretend to be production quality.)

So, I found the answer, and pretty quickly... Its all explained here...
http://www.csharpfriends.com/Articles/getArticle.aspx?articleID=63

what if the html is a invaild format xml?
it looks like we can not use xslt?
Any feedbacks?