Develop Reference

Loading large amounts of data into a List collection of business objects from Oracle

I need a way to optimize the data retrieval of millions of records from an Oracle database into a List of custom business object. The data that is being returned from Oracle is in XML format and I need a way to serialize it into a List of business objects.
The code I wrote is working properly, however, it is taking a long time to execute during the loading of the XML into memory, specifically when the code hits the line:
var xDoc = XDocument.Load(xmlReader);
Code :
//Custom Business object
public class AccountType
{
public int AccountTypeID { get; set; }
public string AccountCode { get; set; }
public string BookType { get; set; }
public int Status { get; set; }
}
//Code that retrieves data from Oracle DB
using (OracleConnection objOracleConnection = new OracleConnection(strConnectionString))
{
using (OracleCommand orclCmd = objOracleConnection.CreateCommand())
{
try
{
orclCmd.CommandText = strXMLSQL;
orclCmd.BindByName = true;
orclCmd.XmlCommandType = OracleXmlCommandType.Query;
orclCmd.XmlQueryProperties.RootTag = "AccountData";
orclCmd.XmlQueryProperties.RowTag = "ROW";
objOracleConnection.Open();
XmlReader xmlReader = orclCmd.ExecuteXmlReader();
var xDoc = XDocument.Load(xmlReader);
List<AccountType> accountTypes = (from data in xDoc.Root.Elements("ROW")
select new AccountType
{
AccountTypeID = data.GetIntXMLElementValue("ACCOUNTTYPEID"),
AccountCode = data.GetStringXMLElementValue("ACCOUNTCODE"),
BookType = data.GetStringXMLElementValue("BOOKTYPE"),
Status = data.GetIntXMLElementValue("STATUS")
}).ToList();
}
catch (OracleException oracleEx)
{
throw oracleEx;
}
catch (Exception generalEx)
{
throw generalEx;
}
finally
{
objOracleConnection.Close();
}
}
Any help would be greatly appreciated.
Thanks!

Paging is an option. Also threading using PLINQ right before you call the ToList() because that is where you are taking the hit. I would also add that removing the XDocument loading could help. Try deserializing the xml. Assuming your XML looks like this
<ROWSET>
<ROW>
<AccountData>
<ACCOUNTTYPEID>1</ACCOUNTTYPEID>
<ACCOUNTCODE>ABC</ACCOUNTCODE>
<BOOKTYPE>FOO</BOOKTYPE>
<STATUS>10</STATUS>
</AccountData>
</ROW>
<ROW>
<AccountData>
<ACCOUNTTYPEID>2</ACCOUNTTYPEID>
<ACCOUNTCODE>XYZ</ACCOUNTCODE>
<BOOKTYPE>BAR</BOOKTYPE>
<STATUS>20</STATUS>
</AccountData>
</ROW>
</ROWSET>
You can set up the following contracts:
[DataContract(Namespace = "")]
public class AccountData
{
[DataMember(Name = "ACCOUNTTYPEID")]
public int Id { get; set; }
}
[DataContract(Name = "ROW", Namespace = "")]
public class Row
{
[DataMember(Name = "AccountData")]
public AccountData Data { get; set; }
}
[CollectionDataContract(Name="ROWSET", Namespace = "")]
public class RowSet
: List<Row>
{
}
and de-serialize like this
var s = new DataContractSerializer(typeof(RowSet));
var o = s.ReadObject(xmlreader) as RowSet;
This avoides the XDocument loading and LINQ-XML overhead.

Do you need the millions of records all at once?
Is the data stored in the database as XML? If not you can use ExexcutePageReader instead of ExecuteXmlReader
If it has to be XML then you could implement your own pager by calling for records where id is between lastId plus 1 and lastId plus pageLength

Adding to XML file

I am making a WPF that searches through an XML file pulling out restaurant information. The XML is in this format:
<FoodPhoneNumbers>
<Restaurant Name="Pizza Place">
<Type>Pizza</Type>
<PhoneNumber>(123)-456-7890</PhoneNumber>
<Hours>
<Open>11:00am</Open>
<Close>11:00pm</Close>
</Hours>
</Restaurant>
</FoodPhoneNumbers>
I want to be able to add a new restaurant to the XML file. I have a textbox for the restaurant name, and type. Then three textboxes for the phone number. 4 comboboxes for the open hour, open minute, close hour, and close minute. I also have 2 listboxes for selecting AM or PM for the open and close times.
I assume I use XmlTextWriter, but I could not figure out how to add the text to a pre-existing XML file.

The simplest way isn't to use XmlTextWriter - it's just to load the whole into an in-memory representation, add the new element, then save. Obviously that's not terribly efficient for large files, but it's really simple if you can get away with it. For example, using XDocument:
XDocument doc = XDocument.Load("test.xml");
XElement restaurant = new XElement("Restaurant",
new XAttribute("Name", "Frenchies"),
new XElement("Type", "French"),
new XElement("PhoneNumber", "555-12345678"),
new XElement("Hours",
new XElement("Open", "1:00pm"),
new XElement("Close", "2:00pm")));
doc.Root.Add(restaurant);
doc.Save("test.xml");
Or, better:
XDocument doc = XDocument.Load("test.xml");
Restaurant restaurant = ...; // Populate a Restaurant object
// The Restaurant class could know how to serialize itself to an XElement
XElement element = restaurant.ToXElement();
doc.Root.Add(element);

The instance of XmlSerializer class can be used to achieve this.
[XmlRoot("FoodPhoneNumbers")]
public class FoodPhoneNumbers
{
[XmlElement(ElementName = "Restaurant")]
public List<Restaurant> Restaurants { get; set; }
}
public class Restaurant
{
[XmlAttribute]
public string Name { get; set; }
[XmlElement]
public string Type { get; set; }
[XmlElement]
public string PhoneNumber { get; set; }
[XmlElement(ElementName = "Hours")]
public List<Hours> Hours { get; set; }
}
public class Hours
{
[XmlElement]
public string Open { get; set; }
[XmlElement]
public string Close { get; set; }
}
Serialization/deserialization code:
// Deserialize.
FoodPhoneNumbers food;
using (Stream inputStream = File.OpenRead(inputFilePath))
food = (FoodPhoneNumbers) xmlSerializer.Deserialize(inputStream);
// Add a new one.
Restaurant restaurant = new Restaurant
{
Name = "Coffee restraurant",
PhoneNumber = "0xFF",
Type = "Coffee shop"
};
food.Restaurants.Add(restaurant);
// Serialize.
using (Stream outputStream = File.OpenWrite(outputFilePath))
xmlSerializer.Serialize(outputStream, food);

Extracting Data From XML File Using C#

I have some XML file sitting in my /bin folder (using vs2010). I would like to extract some data (attributes, element values) from this xml. What is the best way to do this using C#?
do I need to use XMLTextReader or would it be better to create an xmlDocument...I'm confused...

You could use LINQ to XML ?
http://msdn.microsoft.com/en-us/library/bb387098.aspx

Either System.Xml og System.Linq.Xml
I would recommend Linq2Xml: http://msdn.microsoft.com/en-us/library/bb387098.aspx
This is a good guide: http://www.hookedonlinq.com/LINQtoXML5MinuteOverview.ashx

Here is a simple example of querying an XML File
public class Job
{
public int Id { get; set; }
public int CompanyId { get; set; }
public string Title { get; set; }
public string Description { get; set; }
public int HoursPerWeek { get; set; }
}
Snippet of XML File:
<?xml version="1.0" encoding="utf-8"?>
<Jobs>
<Job>
<Id>1</Id>
<CompanyId>2</CompanyId>
<Description>Must be willing to relocate to Nebraska.</Description>
<HoursPerWeek>90</HoursPerWeek>
<JobStatus>1</JobStatus>
<JobType>3</JobType>
<Location>NY</Location>
<Title>Application Engineer for Gooooogle Plus</Title>
</Job>
<Job>
Class which populates Job objects from XML File
public class XMLJobsDAL
{
XDocument JobDoc
{
get { return XDocument.Load(System.Web.HttpContext.Current.Server.MapPath("~/App_Data/Job.xml")); }
}
public List<Job> GetJobs()
{
var results = from job in JobDoc.Descendants("Job")
select new Job
{
Id = (int)job.Element("Id"),
CompanyId = (int)job.Element("CompanyId"),
Description = (string)job.Element("Description"),
HoursPerWeek = (int)job.Element("HoursPerWeek"),
Title = (string) job.Element("Title")
}
return results.ToList();
}
public Job GetJob(int id)
{
var result = from job in JobDoc.Descendants("Job")
where (int)job.Element("Id") == id
select new Job
{
Id = (int)job.Element("Id"),
CompanyId = (int)job.Element("CompanyId"),
Description = (string)job.Element("Description"),
HoursPerWeek = (int)job.Element("HoursPerWeek"),
Title = (string) job.Element("Title")
}
return result.SingleOrDefault();
}
}

I would recommend that you check Serialization out. Here is a little example of how to Serialize/Deserialize XML to/from an object using ek_ny XML Document:
using System.IO;
using System.Xml.Serialization;
[Serializable]
public class Job
{
public Job() { }
public int ID { get; set; }
public int CompanyID { get; set; }
public string Description { get; set; }
public int HoursPerWeek { get; set; }
public int JobStatus { get; set; }
public int JobType { get; set; }
public string Location { get; set; }
public string Title { get; set; }
public void SerializeToXML(string path)
{
//creates a file
FileStream fs = new FileStream(path, FileMode.Create);
//now we create the serializer
XmlSerializer xs = new XmlSerializer(typeof(Job));
//serialize it to the file
xs.Serialize(fs, this);
//close the file
fs.Close();
}
public static Job DeserializeToJob(string path) {
//open file
FileStream fs = new FileStream(path, FileMode.Open);
//create deserializer
XmlSerializer xs = new XmlSerializer(typeof(Job));
//Deserialize
Job job = (Job)xs.Deserialize(fs);
//close the file
fs.Close();
//return the deserialized job
return job;
}
}
Implementation:
class Program
{
static void Main(string[] args)
{
Job j = new Job();
j.CompanyID = 2;
j.ID = 1;
j.Description = "Must be willing to relocate to Nebraska.";
j.HoursPerWeek = 90;
j.JobStatus = 1;
j.JobType = 3;
j.Location = "NY";
j.Title = "Application Engineer for Gooooogle Plus";
//Saving the object serialized.
j.SerializeToXML("MyJob.xml");
//deserialize the saved job into a new instance
Job j2 = Job.DeserializeToJob("MyJob.xml");
}
}
With this you will get the object back and forth from and to XML. This is what has worked the best for me. The only cons I can see here is that if your XML document has a lot of properties you will have to make a very big class.
http://support.microsoft.com/kb/815813
http://www.switchonthecode.com/tutorials/csharp-tutorial-xml-serialization
Good luck!

There's one good way to do this. You can use
DataTable.ReadXml(string fileName)
it takes care of everything easily. Depending on the problem, it could be useful or not.
Note this methods cannot get the schema from xml file, you should add columns to the table yourself.

Element was not expected While Deserializing an Array with XML Serialization

OK. I'm trying to work on communicating with the Pivotal Tracker API, which only returns data in an XML format. I have the following XML that I'm trying to deserialize into my domain model.
<?xml version="1.0" encoding="UTF-8"?>
<stories type="array" count="2" total="2">
<story>
<id type="integer">2909137</id>
<project_id type="integer">68153</project_id>
<story_type>bug</story_type>
<url>http://www.pivotaltracker.com/story/show/2909137</url>
<current_state>unscheduled</current_state>
<description></description>
<name>Test #2</name>
<requested_by>Anthony Shaw</requested_by>
<created_at type="datetime">2010/03/23 20:05:58 EDT</created_at>
<updated_at type="datetime">2010/03/23 20:05:58 EDT</updated_at>
</story>
<story>
<id type="integer">2909135</id>
<project_id type="integer">68153</project_id>
<story_type>feature</story_type>
<url>http://www.pivotaltracker.com/story/show/2909135</url>
<estimate type="integer">-1</estimate>
<current_state>unscheduled</current_state>
<description></description>
<name>Test #1</name>
<requested_by>Anthony Shaw</requested_by>
<created_at type="datetime">2010/03/23 20:05:53 EDT</created_at>
<updated_at type="datetime">2010/03/23 20:05:53 EDT</updated_at>
</story>
</stories>
My 'story' object is created as follows:
public class story
{
public int id { get; set; }
public int estimate { get; set; }
public int project_id { get; set; }
public string story_type { get; set; }
public string url { get; set; }
public string current_state { get; set; }
public string description { get; set; }
public string name { get; set; }
public string requested_by { get; set; }
public string labels { get; set; }
public string lighthouse_id { get; set; }
public string lighthouse_url { get; set; }
public string owned_by { get; set; }
public string accepted_at { get; set; }
public string created_at { get; set; }
public attachment[] attachments { get; set; }
public note[] notes { get; set; }
}
When I execute my deserialization code, I receive the following exception:
Exception:
There is an error in XML document (2, 2).
Inner Exception:
<stories xmlns=''> was not expected.
I can deserialize the individual stories just fine, I just cannot deserialize this xml into an array of 'story' objects
And my deserialization code (value is a string of the xml)
var byteArray = Encoding.ASCII.GetBytes(value);
var stream = new MemoryStream(byteArray);
var deserializedObject = new XmlSerializer(typeof (story[])).Deserialize(stream)
Does anybody have any ideas?

Let me offer a more concise solution. Set your deserialization up to look like this:
var deserializedObject = new XmlSerializer(typeof(story[]), new XmlRootAttribute("stories")).Deserialize(stream);
By specifying that second parameter in the XmlSerializer, you can avoid having to stub out that class. It lets the serializer know what the root element's name is.
For this to work, the name of the class that represents the array-element type must exactly match the XML name, e.g. class story {}, <story>. You can get around this (and I'd recommend it as a best practice anyway) by specifying the XmlType:
[XmlType("story")]
public class Story
{
...
}
I prefer to do this as it frees me from being stuck with the XML type name.

The problem is that you have no property named "stories". The XML Serializer has no idea what to do with the stories element when it sees it.
One thing you could try is to create a "stories" class:
public class stories : List<story> {}
and use
var byteArray = Encoding.ASCII.GetBytes(value);
stories deserializedObject = null;
using (var stream = new MemoryStream(byteArray))
{
var storiesSerializer = new XmlSerializer(typeof (stories));
deserializedObject = (stories)storiesSerializer .Deserialize(stream);
}

Try something like
public class stories
{
[XmlElement("story")]
public story[] storyarray { get; set; }
}
...
var byteArray = Encoding.ASCII.GetBytes(value);
XmlSerializer serializer = new XmlSerializer(typeof(stories));
stories myStories = null;
using (var stream = new MemoryStream(byteArray))
{
myStories = (stories)serializer.Deserialize(stream);
}
foreach (story stor in myStories.storyarray)
Console.WriteLine(stor.story_type);
Edit: Updated code sample to use using statement based on feedback.

XMSerializer expects an XML Namespace with which to understand your XML from.
xmlns="http://schemas.microsoft.com"
... ought to do. See the XML sample at the bottom of this page.

I would recommend that you generate an XSD from some sample XML you get from the web service. Then, with that XSD, you can generate the classes that have all the proper serialization attributes affixed to them.
To generate a schema (unless you prefer to write your own), open the sample XML file in Visual Studio, and select the XML -> Create Schema menu option. Save that XSD.
To generate the classes, run the XSD command from the VS command prompt. If you run it without parameters, it'll show you the command-line parameters you can use.
Now you can create a type-safe XML serializer.

What is the best way to manually parse an XElement into custom objects?

I have an XElement variable named content which consists of the following XML:
<content>
<title>Contact Data</title>
<p>This is a paragraph this will be displayed in front of the first form.</p>
<form idCode="contactData" query="limit 10; category=internal"/>
<form idCode="contactDataDetail" query="limit 10; category=internal">
<title>Contact Data Detail</title>
<description>This is the detail information</description>
</form>
</content>
I now want to simply run through each of the level-1 nodes and parse them into objects. Back in C# 2.0 I use to do this with XmlReader, check the type of node, and parse it accordingly.
But what is the best way to parse the XML nodes with LINQ, I would expect something like this:
var contentItems = from contentItem in pageItem.content.DescendantNodes()
select new ContentItem
{
Type = contentItem.Element()
};
foreach (var contentItem in contentItems)
{
switch (contentItem.Type)
{
case "title":
...(parse title)...
case "p":
...(parse p)...
case "form":
...(parse form)...
}
}
where:
public class ContentItem
{
public string Type { get; set; }
public string IdCode { get; set; }
public XElement Content { get; set; }
}

Does it have to be XElement? I would (either manually, or via xsd.exe) just create classes that map to the element/attribute names - and use XmlSerializer - in particular via StringReader:
Content content;
using(StringReader sr = new StringReader(xml))
using(XmlReader xr = XmlReader.Create(sr)) {
XmlSerializer ser = new XmlSerializer(typeof(Content));
content = (Content)ser.Deserialize(xr);
}
(edit)
With entity classes:
[XmlRoot("content")]
public class Content {
[XmlElement("title")]
public string Title { get; set; }
[XmlElement("p")]
public string Description { get; set; }
[XmlElement("form")]
public List<ContentForm> Forms { get; set; }
}
public class ContentForm {
[XmlAttribute("idCode")]
public string Id { get; set; }
[XmlAttribute("query")]
public string Query { get; set; }
[XmlElement("title")]
public string Title { get; set; }
[XmlElement("description")]
public string Description { get; set; }
}

I'd suggest inheriting XElement, and implement properties for the stuff you want in it.
These properties shouldn't use backing fields, but rather work directly with the underlying XML element. That way, you'll keep object in sync with XML.

With XML to LINQ one pulls specific data items out of the XML, rather than iterating through the XML looking for what is found.
var results = from node in XmlDocument.SomeContext(...)
select new MyType {
Prop1 = node.Element("One").Value,
Prop2 = node.Element("One").Attributes().Where(
a => A.Value.Contains("Foo").Value
};
If conditions are needed, then (extension) methods and arbitrary expressions can be used (null-coalescing operator, ??, can be very helpful to default).