Develop Reference

Parse an xml document with "dynamic" nodes

I am parsing XML via an XDocument, how can I retreive all languages, i.e <en> or <de> or <CodeCountry> and their child elements?
<en>
<descriptif>In the historic area, this 16th century Town House on 10,764 sq. ft. features 10 rooms and 3 shower-rooms. Period features include a spiral staircase. 2-room annex house with a vaulted cellar. Period orangery. Ref.: 2913.</descriptif>
<prox>NOGENT-LE-ROTROU.</prox>
<libelle>NOGENT-LE-ROTROU.</libelle>
</en>
<de>
<descriptif>`enter code here`In the historic area, this 16th century Town House on 10,764 sq. ft. features 10 rooms and 3 shower-rooms. Period features include a spiral staircase. 2-room annex house with a vaulted cellar. Period orangery. Ref.: 2913.</descriptif>
<prox>NOGENT-LE-ROTROU.</prox>
</de>
...
<lang>
<descriptif></descriptif>
<prox></prox>
<libelle></libelle>
</lang>

As your xml document is not well formatted, you should first add a root element.
You may do something like that.
var content = File.ReadAllText(#"<path to your xml>");
var test = XDocument.Parse("<Language>" + content + "</Language>");
Then, as you have "dynamic top nodes", you may try to work with their children (which don't seem to be dynamic), assuming all nodes have at least a "descriptif" child. (If it's not "descriptif", it may be "prox" or "libelle") **.
//this will give you all parents, <en>, <de> etc. nodes
var parents = test.Descendants("descriptif").Select(m => m.Parent);
Then you can select the language and childrens.
I used an anonymous type, you can of course project to a custom class.
var allNodes = parents.Select(m => new
{
name = m.Name.LocalName,
Descriptif = m.Element("descriptif") == null ? string.Empty : m.Element("descriptif").Value,
Prox = m.Element("prox") == null ? string.Empty : m.Element("prox").Value ,
Label = m.Element("libelle") == null ? string.Empty : m.Element("libelle").Value
});
This is of course not performant code for a big file, but... that's another problem.
**
Worst case, you may do
var parents = test.Descendants("descriptif").Select(m => m.Parent)
.Union(test.Descendants("prox").Select(m => m.Parent))
.Union(test.Descendants("libelle").Select(m => m.Parent));

Group by the parsed value HTML AgilityPack C#

Group data in C#, I have parsed the html file and get all the data on it, now I want to group them as following:
Those lines which are selected are the parent and contain the following childs, the code that I'm working on is here:
var uricontent = File.ReadAllText("TestHtml/Bew.html");
var doc = new HtmlDocument(); // with HTML Agility pack
doc.LoadHtml(uricontent);
var rooms = doc.DocumentNode.SelectNodes("//table[#class='rates']").SelectMany(
detail =>
{
return doc.DocumentNode.SelectNodes("//td[#class='rate-description'] | //table[#class='rooms']//h2 | //table[#class='rooms']//td[#class='room-price room-price-total']").Select(
r => new
{
RoomType = r.InnerText.CleanInnerText(),
});
}).ToArray();
the RoomType contains the data which is parsed by HTML AgilityPack, how can I group them by the Name like Pay & Save , Best Available Room Only ...
HTML File is here : http://notepad.cc/share/g0zh0TcyaG
Thank you

Instead of doing union of 3 XPath queries, then trying to group them back by "Rate Description" (aka by element : <td class="rate-description">), you can do it another way around.
You can base your LINQ selection by "Rate Description", then in projection part, get all room types and room rates under current "Rate Description" using relative XPath :
var rooms =
doc.DocumentNode
.SelectNodes("//table[#class='rates']//tr[#class='rate']")
.Select(r => new
{
RateType = r.SelectSingleNode("./td[#class='rate-description']")
.InnerText.CleanInnerText,
RoomTypes = r.SelectNodes("./following-sibling::tr[#class='rooms'][1]//table[#class='rooms']//h2")
.Select(s => new
{
RoomType = s.InnerText.CleanInnerText,
Rate = s.SelectSingleNode(".//parent::td/following-sibling::td[#class='room-price room-price-total'][1]")
.InnerText.CleanInnerText
}).ToArray()
}).ToArray();
Notice period at the beginning of some XPath queries above. That tells HtmlAgilityPack that the query is relative to current HtmlNode. The result is about like this :

Create XML based on text tree

I need to go from a list like this:
/home
/home/room1
/home/room1/subroom
/home/room2
/home/room2/miniroom
/home/room2/bigroom
/home/room2/hugeroom
/home/room3
to an xml file. I've tried using LINQ to XML to do this but I just end up getting confused and not sure what to do from there. Any help is much appreciated!
Edit:
I want the XML file to look something like this:
<home>
<room1>
<subroom>This is a subroom</subroom>
</room1>
<room2>
<miniroom>This is a miniroom</miniroom>
<bigroom>This is a bigroom</bigroom>
<hugeroom>This is a hugeroom</hugeroom>
</room2>
<room3></room3>
</home>
The text inside if the tags ("this is a subroom", etc) is optional, but would be really nice to have!

Ok buddy, here's a solution.
Couple of notes and explanation.
Your text structure can be split up into lines and then again by the slashes into the names of the XML nodes. If you think of the text in this way, you get a list of "lines" broken into a list of
names.
/home
First of all, the first line /home is the root of the XML; we can get rid of it and just create and XDocument object with that name as the root element;
var xDoc = new XDocument("home");
Of course we don't want to hard code things but this is just an example. Now, on to the real work:
/home/room1/
/home/room1/bigroom
etc...
as a List<T> then it will look like this
myList = new List<List<string>>();
... [ add the items ]
myList[0][0] = home
myList[0][1] = room1
myList[1][0] = home
myList[1][1] = room1
myList[1][2] = bigroom
So what we can do to get the above structure is use string.Split() multiple times to break your text first into lines, then into parts of each line, and end up with a multidimensional array-style List<T> that contains List<T> objects, in this case, List<List<string>>.
First let's create the container object:
var possibleNodes = new List<List<string>>();
Next, we should split the lines. Let's call the variable that holds the text, "text".
var splitLines = text
.Split(new string[] { Environment.NewLine }, StringSplitOptions.RemoveEmptyEntries)
.ToList();
This gives us a List but our lines are still not broken up. Let's split them again by the slash (/) character. This is where we build our node names. We can do this in a ForEach and just add to our list of possible nodes:
splitLines.ForEach(l =>
possibleNodes.Add(l
.Split(new char[] { '/' }, StringSplitOptions.RemoveEmptyEntries)
.ToList()
)
);
Now, we need to know the DEPTH of the XML. Your text shows that there will be 3 nodes of depth. The node depth is the maximum depth of any one given line of nodes, now stored in the List<List<string>>; we can use the .Max() method to get this:
var nodeDepth = possibleNodes.Max(n => n.Count);
A final setup step: We don't need the first line, because it's just "home" and it will be our root node. We can just create an XDocument object and give it this first line to use as the name of Root:
// Create the root node
XDocument xDoc = new XDocument(new XElement(possibleNodes[0][0]));
// We don't need it anymore
possibleNodes.RemoveAt(0);
Ok, here is where the real work happens, let me explain the rules:
We need to loop through the outer list, and through each inner list.
We can use the list indexes to understand which node to add to or which names to ignore
We need to keep hierarchy proper and not duplicate nodes, and some XLinq helps here
The loops - see the comments for a detailed explanation:
// This gets us looping through the outer nodes
for (var i = 0; i < possibleNodes.Count; i++)
{
// Here we go "sideways" by going through each inner list (each broken down line of the text)
for (var ii = 1; ii < nodeDepth; ii++)
{
// Some lines have more depth than others, so we have to check this here since we are looping on the maximum
if (ii < possibleNodes[i].Count)
{
// Let's see if this node already exists
var existingNode = xDoc.Root.Descendants().FirstOrDefault(d => d.Name.LocalName == (possibleNodes[i][ii]));
// Let's also see if a parent node was created in the previous loop iteration.
// This will tell us whether to add the current node at the root level, or under another node
var parentNode = xDoc.Root.Descendants().FirstOrDefault(d => d.Name.LocalName == (possibleNodes[i][ii - 1]));
// If the current node has already been added, we do nothing (this if statement is not entered into)
// Otherwise, existingNode will be null and that means we need to add the current node
if (null == existingNode)
{
// Now, use parentNode to decide where to add the current node
if (null == parentNode)
{
// The parent node does not exist; therefore, the current node will be added to the root node.
xDoc.Root.Add(new XElement(possibleNodes[i][ii]));
}
else
{
// There IS a parent node for this node!
// Therefore, we must add the current node to the parent node
// (remember, parent node is the previous iteration of the inner for loop on nodeDepth )
var newNode = new XElement(possibleNodes[i][ii]);
parentNode.Add(newNode);
// Add "this is a" text (bonus!) -- only adding this text if the current node is the last one in the list.
if (possibleNodes[i].Count -1 == ii)
{
newNode.Add(new XText("This is a " + newNode.Name.LocalName));
}
}
}
}
}
}
The bonus here is this code will work with any number of nodes and build your XML.
To check it, XDocument has a nifty .ToString() overriden implementation that just spits out all of the XML it is holding, so all you do is this:
Console.Write(xDoc.ToString());
And, you'll get this result:
(Note I added a test node to make sure it works with more than 3 levels)
Below, you will find the entire program with your test text, etc, as a working solution:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Xml.Linq;
namespace XmlFromTextString
{
class Program
{
static void Main(string[] args)
{
// This simulates text from a file; note that it must be flush to the left of the screen or else the extra spaces
// add unneeded nodes to the lists that are generated; for simplicity of code, I chose not to implement clean-up of that and just
// ensure that the string literal is not indented from the left of the Visual Studio screen.
string text =
#"/home
/home/room1
/home/room1/subroom
/home/room2
/home/room2/miniroom
/home/room2/test/thetest
/home/room2/bigroom
/home/room2/hugeroom
/home/room3";
var possibleNodes = new List<List<string>>();
var splitLines = text
.Split(new string[] { Environment.NewLine }, StringSplitOptions.RemoveEmptyEntries)
.ToList();
splitLines.ForEach(l =>
possibleNodes.Add(l
.Split(new char[] { '/' }, StringSplitOptions.RemoveEmptyEntries)
.ToList()
)
);
var nodeDepth = possibleNodes.Max(n => n.Count);
// Create the root node
XDocument xDoc = new XDocument(new XElement(possibleNodes[0][0]));
// We don't need it anymore
possibleNodes.RemoveAt(0);
// This gets us looping through the outer nodes
for (var i = 0; i < possibleNodes.Count; i++)
{
// Here we go "sideways" by going through each inner list (each broken down line of the text)
for (var ii = 1; ii < nodeDepth; ii++)
{
// Some lines have more depth than others, so we have to check this here since we are looping on the maximum
if (ii < possibleNodes[i].Count)
{
// Let's see if this node already exists
var existingNode = xDoc.Root.Descendants().FirstOrDefault(d => d.Name.LocalName == (possibleNodes[i][ii]));
// Let's also see if a parent node was created in the previous loop iteration.
// This will tell us whether to add the current node at the root level, or under another node
var parentNode = xDoc.Root.Descendants().FirstOrDefault(d => d.Name.LocalName == (possibleNodes[i][ii - 1]));
// If the current node has already been added, we do nothing (this if statement is not entered into)
// Otherwise, existingNode will be null and that means we need to add the current node
if (null == existingNode)
{
// Now, use parentNode to decide where to add the current node
if (null == parentNode)
{
// The parent node does not exist; therefore, the current node will be added to the root node.
xDoc.Root.Add(new XElement(possibleNodes[i][ii]));
}
else
{
// There IS a parent node for this node!
// Therefore, we must add the current node to the parent node
// (remember, parent node is the previous iteration of the inner for loop on nodeDepth )
var newNode = new XElement(possibleNodes[i][ii]);
parentNode.Add(newNode);
// Add "this is a" text (bonus!) -- only adding this text if the current node is the last one in the list.
if (possibleNodes[i].Count -1 == ii)
{
newNode.Add(new XText("This is a " + newNode.Name.LocalName));
// For the same default text on all child-less nodes, us this:
// newNode.Add(new XText("This is default text"));
}
}
}
}
}
}
Console.Write(xDoc.ToString());
Console.ReadKey();
}
}
}

Time for LINQ magic?
// load file into string[]
var input = File.ReadAllLines("TextFile1.txt");
// in case you have more than one home in your file
var homes =
new XDocument(
new XElement("root",
from line in input
let items = line.Split(new[] { "/" }, StringSplitOptions.RemoveEmptyEntries)
group items by items[0] into g
select new XElement(g.Key,
from rooms in g.OrderBy(x => x.Length).Skip(1)
group rooms by rooms[1] into g2
select new XElement(g2.Key,
from name in g2.OrderBy(x => x.Length).Skip(1)
select new XElement(name[2], string.Format("This is a {0}", name[2]))))));
// get the right home
var home = new XDocument(homes.Root.Element("home"));

If statement in LINQ (If no value do something...)

I'm trying to make an application which shows the bushours including bus numbers, Aimed hour and Expected hour.
I get my (live) information from a HttpWebRequest. My response from the request is stored in a string variable in XML format.
I can get all the information that I want; like the bus hour, Aimed hour and Expected hour.
The problem is that if there is no Expected hour nothing will be showed. I like to have when there is no Expected hour my code just takes the same value as the Aimed hour:
An example
Bus | Aimed | Execepted
-----------------------
1 | 17:05 | 17:07
2 | 17:05 | <nothing> so take value of aimed -> 17:05
I have already the following code
//XMLResponse put in documentRoot
//responseFromServer is a string variable in XML format with all the information
XElement documentRoot = XDocument.Parse(responseFromServer).Root;
XNamespace ns = "http://www.siri.org.uk/";
var buses = (from tblBuses in documentRoot.Descendants(ns + "PublishedLineName")
select tblBuses.Value).ToList();
var expHours = (from tblHours in documentRoot.Descendants(ns + "ExpectedDepartureTime")
select tblHours.Value).ToList();
foreach (var bus in buses)
{
string output = bus.Substring(bus.IndexOf('T') + 1);
int index = output.IndexOf(".");
if (index > 0)
output = output.Substring(0, index);
listBox1.Items.Add("Bus: " + output);
}
//Show every ExpectedDepartureTime
//If there is no expectedTime take value AimedDepartureTime
foreach (var expH in expHours)
{
string output = expH.Substring(expH.IndexOf('T') + 1);
int index = output.IndexOf(".");
if (index > 0)
output = output.Substring(0, index);
lstHours.Items.Add(output);
}
for being more clear to having an understand of my XML response, below an example of my XML response (One with AimedDeparturetime and Expected and one with without Expected)
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Siri version="1.0" xmlns="http://www.siri.org.uk/">
<ServiceDelivery>
<ResponseTimestamp>2013-03-26T16:09:48.181Z</ResponseTimestamp>
<StopMonitoringDelivery version="1.0">
<ResponseTimestamp>2013-03-26T16:09:48.181Z</ResponseTimestamp>
<RequestMessageRef>12345</RequestMessageRef>
<MonitoredStopVisit>
<RecordedAtTime>2013-03-26T16:09:48.181Z</RecordedAtTime>
<MonitoringRef>020035811</MonitoringRef>
<MonitoredVehicleJourney>
<FramedVehicleJourneyRef>
<DataFrameRef>-</DataFrameRef>
<DatedVehicleJourneyRef>-</DatedVehicleJourneyRef>
</FramedVehicleJourneyRef>
<VehicleMode>bus</VehicleMode>
<PublishedLineName>2</PublishedLineName>
<DirectionName>Elstow P+R</DirectionName>
<OperatorRef>STB</OperatorRef>
<MonitoredCall>
<AimedDepartureTime>2013-03-26T16:11:00.000Z</AimedDepartureTime>
<ExpectedDepartureTime>2013-03-26T16:11:28.000Z</ExpectedDepartureTime>
</MonitoredCall>
</MonitoredVehicleJourney>
</MonitoredStopVisit>
---------------------------------------------------
<MonitoredStopVisit>
<RecordedAtTime>2013-03-26T16:09:48.181Z</RecordedAtTime>
<MonitoringRef>020035811</MonitoringRef>
<MonitoredVehicleJourney>
<FramedVehicleJourneyRef>
<DataFrameRef>-</DataFrameRef>
<DatedVehicleJourneyRef>-</DatedVehicleJourneyRef>
</FramedVehicleJourneyRef>
<VehicleMode>bus</VehicleMode>
<PublishedLineName>53</PublishedLineName>
<DirectionName>Wootton</DirectionName>
<OperatorRef>STB</OperatorRef>
<MonitoredCall>
<AimedDepartureTime>2013-03-26T16:19:00.000Z</AimedDepartureTime>
</MonitoredCall>
</MonitoredVehicleJourney>
</MonitoredStopVisit>
</StopMonitoringDelivery>
</ServiceDelivery>
</Siri>
So for this moment my application doesn't show every departure time of a bus.
How can I solve this?
Thanks!

Apologies upfront, because this is not the greatest XML parsing ever, but I would adjust my LINQ query:
var buses = from tblBuses in documentRoot.Descendants(ns + "MonitoredVehicleJourney")
select new
{
LineName = tblBuses.Descendants(ns + "PublishedLineName").Single().Value,
AimedHours = tblBuses.Descendants(ns + "AimedDepartureTime").Single().Value,
ExpectedHours = tblBuses.Descendants(ns + "ExpectedDepartureTime").Select(el => el.Value).SingleOrDefault()
};
This will create an IEnumerable of some anonymous type which allows you to access the bus data more easily in subsequent code:
foreach (var bus in buses)
{
// Take ExpectedHours, or AimedHours if the first is null
string expH = bus.ExpectedHours ?? bus.AimedHours
// Same code as before here
string output = expH.Substring(expH.IndexOf('T') + 1);
int index = output.IndexOf(".");
if (index > 0)
output = output.Substring(0, index);
lstHours.Items.Add(output);
}
In your original code, buses that did not have an <ExpectedDepartureTime> were never iterated over because they never show up in your expHours List. In contrast, this LINQ query will contain all buses. It assumes that they all have a single <AimedDepartureTime> and an optional <ExpectedDepartureTime>.
For the expected departure time, I used a Select to get the element value for each of the descendants. Using SingleOrDefault().Value cannot be used, because the query might yield no elements and get_Value() would be called on a null reference.
One last comment about my query: for production code I would refrain from using Descendants and do more strict querying of the XML structure.

You can do everything inside one linq query and use an ?: conditional operator to select the correct output:
var buses =
(from tblBuses in documentRoot.Descendants(ns + "PublishedLineName")
let bus = tblBuses.Value
let output = bus.Substring(bus.IndexOf('T') + 1)
let index = output.IndexOf(".")
select (index > 0) ? output.Substring(0, index) : output);
foreach (var bus in buses)
{
listBox1.Items.Add("Bus: " + bus);
}
or even
var buses =
(from ...
select "Bus: " + ((index > 0) ? output.Substring(0, index) : output));
.ToArray();
listBox1.Items.AddRange(buses);
The same pattern can be applied to expHours.

Sort by most recent date and cluster (group) similar titles

Looking for LINQ needed to sort on a date field but also have similar titles grouped and sorted. Consider something like the following desired ordering:
Title Date
"Some Title 1/3" 2009/1/3 "note1: even this is old title 3/3 causes this group to be 1st"
"Some Title 2/3" 2011/1/31 "note2: dates may not be in sequence with titles"
"Some Title 3/3" 2011/1/1 "note3: this date is most recent between "groups" of titles
"Title XYZ 1of2" 2010/2/1
"Title XYz 2of2" 2010/2/21
I've shown titles varying by some suffix. What if a poster used something like the following for titles?
"1 LINQ Tutorial"
"2 LINQ Tutorial"
"3 LINQ Tutorial"
How would the query recognize these are similar titles?
You don't have to solve everything, a solution for the 1st example is much appreciated.
Thank you.
Addendum #1 20110605
#svick also Title authors typically are not thoughtful to use say 2 digits when their numbering scheme goes beyond 9. for example 01,02...10,11 etc..
Typical patterns I've seen tend to be either prefix or suffix or even buried in such as
1/10 1-10 ...
(1/10) (2/10) ...
1 of 10 2 of 10
Part 1 Part 2 ...
You pointed out a valid pattern as well:
xxxx Tutorial : first session, xxxx Tutorial : second session, ....
If I have a Levenshtein function StringDistance( s1, s2 ) how would I fit into the LINQ query :)

Normal grouping in LINQ (and in SQL, but that's not relevant here) works by selecting some key for every element in the collection. You don't have such key, so I wouldn't use LINQ, but two nested foreaches:
var groups = new List<List<Book>>();
foreach (var book in books)
{
bool found = false;
foreach (var g in groups)
{
if (sameGroup(book.Title, g[0].Title))
{
found = true;
g.Add(book);
break;
}
}
if (!found)
groups.Add(new List<Book> { book });
}
var result = groups.Select(g => g.OrderBy(b => b.Date).ToArray()).ToArray();
This gradually creates a list of groups. Each book is compared with the first one in each group. If it matches, it is added to the group. If no group matched, the book creates a new group. In the end, we sort the results using LINQ with dot notation.
It would be more correct if books were compared with each book in a group, not just the first. But you're may not get completely correct results anyway, so I think this optimization is worth it.
This has time complexity O(N²), so it's probably not the best solution if you had millions of books.
EDIT: To sort the groups, use something like
groups.OrderBy(g => g.Max(b => b.Date))

For ordering by date you should use the OrderBy operator.
Example:
//Assuming your table is called Table in datacontext ctx
var data = from t in ctx.Table
order by t.Date
select t;
For grouping strings after similarity you should consider something like the Hamming distance or the Metaphone algorithm. (Although I do not know any direct implementations of these in .Net).
EDIT: As suggested in the comment by svick, the Levenstein distance may also be considered, as a better alternative to the Hamming distance.

Assuming that your Title and Date fields are contained in class called model consider the following class definition
public class Model
{
public DateTime Date{get;set;}
public string Title{get;set;}
public string Prefix
{get
{
return Title.Substring(0,Title.LastIndexOf(' '));
}
}
}
Alongside Date and Title properties i have created a prefix property with no setter and it is returning us the common prefix using substring. you can use any method of your choice in getter of this property. Rest of job is simple. Consider this Linqpad program
void Main()
{
var model = new List<Model>{new Model{Date = new DateTime(2011,1,3), Title = "Some Title 1/3"},
new Model{Date = new DateTime(2011,1,1), Title = "Some Title 2/3"},
new Model{Date = new DateTime(2011,1,1), Title = "Some Title 3/3"},
new Model{Date = new DateTime(2011,1,31), Title = "Title XYZ 1of2"},
new Model{Date = new DateTime(2011,1,31), Title = "Title XYZ 2of2"}};
var result = model.OrderBy(x => x.Date).GroupBy(x => x.Prefix);
Console.WriteLine(result);
}
Edits >>>
If we put the prefix aside the query itself is not returning what I was after which is: 1) Sort the groups by their most recent date 2) sort by title within clusters. Try the following
var model = new List<Model>{
new Model{Date = new DateTime(2009,1,3), Title = "BTitle 1/3"},
new Model{Date = new DateTime(2011,1,31), Title = "BTitle 2/3"},
new Model{Date = new DateTime(2011,1,1), Title = "BTitle 3/3"},
new Model{Date = new DateTime(2011,1,31), Title = "ATitle XYZ 2of2"},
new Model{Date = new DateTime(2011,1,31), Title = "ATitle XYZ 1of2"}
};
var result = model.OrderBy(x => x.Date).GroupBy(x => x.Prefix);
Console.WriteLine(result);