I have a database result (as a list of items) that is huge. 450K items are returned.
I then use Linq-To-Xml to convert this collection to an XDocument. It works - but it's a huge memory hit.
Is it possible to convert the collection items to XML but not all in memory, at once .. but streaming to a file as the conversation happens?
The XML data finally saves to disk at over 1Gig. So it's fair to assume that the memory consumption will at least be this.
So - is there a way to stream the XML segments to disk, as we iterate over each item in the list instead of converting the entire result-set to an InMemory XDocument and then saving this to disk?
NOTE: Please do not make suggestions about breaking up the list into smaller parts, etc. I understand that, but I've ruled that out.
Here's some sample code I'm doing (to help give you an idea).
// Create the xml doc.
var elements = from user in userResults
select new XElement("user",
new XElement("id", user.Id),
.....<snip>...... );
return new XDocument(new XDeclaration("1.0", "utf-8", "yes"),
new XElement("users", elements));
// Save the doc to the filesystem.
using (var writer = _fileSystemWrapper.CreateText(destinationXmlFileName))
{
xmldDocument.Save(writer);
}
Update
Maybe there's some other tricks like using Linq-To-Xml to create element segments in batches of 10 .. and for each 10, append that to the end of a file?
You can use XStreamingElement. It would require some small refactoring though, mainly not creating XDocument and using different Save method.
Here is the corresponding sample to yours just to give you an idea:
var elements = from user in userResults
select new XElement("user",
new XElement("id", user.Id),
.....<snip>...... );
var content = new XStreamingElement("users", elements);
using (var output = _fileSystemWrapper.CreateText(destinationXmlFileName))
using (var writer = XmlWriter.Create(output, new XmlWriterSettings { Encoding = Encoding.UTF8, Indent = true })
{
// Use the next line if you don't require standalone="yes" attribute
// content.Save(writer);
writer.WriteStartDocument(true);
content.WriteTo(writer);
writer.WriteEndDocument();
}
Related
Is it possible to add literal XML data within a C# code file? I'm currently using a multiline string literal but it gets messy as you can see. Any better way of doing this?
string XML = #"<?xml version=""1.0"" encoding=""utf-8""?>
<customUI xmlns=""http://schemas.example.com/customui"">
<toolbar id=""save"">
</toolbar>
</customUI>";
XML literals are a feature of VB.NET, not C#.
What you have posted is as close as you can get in C#.
You may want to consider replacing the embedded double quotes with single quotes though (as both types are valid XML).
For larger amounts of XML you may want to consider the answer from Marc - using an XML file (loaded once and stored in memory), so you can take advantage of the XML editor.
If the XML is big enough to get in the way, consider using a flat .xml file instead, either loaded from disk, or embedded as a resource. As long as you only load it once (perhaps in a static constructor) this will make no difference to performance. It will be considerably easier to maintain, as it will use the IDE's XML file editor. And it won't get in the way of your code.
With reference to my comment, I couldn't recall where I saw this, but I finally found the XmlBuilder link.
In retrospect, it seems Linq to XML would be your best bet. It's cleaner, faster and more maintainable than concatenating XML strings:
XNamespace ns = "http://schemas.example.com/customui";
XDocument doc = new XDocument(
new XDeclaration("1.0", "utf-8", "yes"),
new XElement(ns + "customUI",
new XElement(ns + "taskbar",
new XAttribute("id", "save"))
)
);
var stringWriter = new StringWriter();
doc.Save(stringWriter); //Write to StringWriter, preserving the declaration (<?xml version="1.0" encoding="utf-16" standalone="yes"?>)
var xmlString = stringWriter.ToString(); //Save as string
doc.Save(#"d:\out.xml"); //Save to file
As a peculiar, and very case-specific solution, if you happen to be working in an ASP.NET environment using the Razor engine, in a CSHTML file you can:
Func<MyType, HelperResult> xml = #<root>
<item>#(item.PropertyA)</item>
<item>#(item.PropertyB)</item>
<item>#(item.PropertyC)</item>
</root>;
With the addition of an extension method:
public static XDocument ToXDocument<T>(this Func<T, HelperResult> source, T item)
{
return XDocument.Parse(source(item).ToHtmlString());
}
You can then:
XDocument document = xml.ToXDocument(new MyType() {
PropertyA = "foo",
PropertyB = "bar",
PropertyC = "qux",
});
Again, peculiar? Yes. Case-specific? Yes. But it works, and gives great Intellisense. (mind you, it also will give a bunch of validity warnings, depending on the document validation version)
The closest we could have in C# would be through LINQ, something like that:
var xml = XDocument.Load(
new StringReader(#"<Books>
<Book author='Dan Brown'>The Da Vinci Code</Book>
<Book author='Dan Brown'>The Lost Symbol</Book>
</Books>"));
var query = from book in xml.Elements("Books").Elements("Book")
where book.Attribute("author").Value == "Dan Brown"
select book.Value;
foreach (var item in query) Console.WriteLine(item);
I am trying to figure out how to modify custom XML parts previusly saved in Excel. All the web resources I have found so far explain how to add custom XML parts in Excel. This I already know. But I want to modify existing parts.
The API seems to have only Add method. If Add method is called again it adds additional XML parts.
I use the following code to save my custom XML
XNamespace NS = "http://schema.blabla.com";
var xDoc = new XDocument(
new XDeclaration("1.0", "utf-8", "no"),
new XComment("Custom XML Parts demo"),
new XElement(NS + "demo",
new XElement(NS + "config",
new XElement(NS + "property",
new XAttribute("value", "myVlaue",
new XAttribute("key", "myKey"))))));
Office.CustomXMLPart customXMLPart = workbook.CustomXMLParts.Add(xDoc.ToString(), System.Type.Missing);
I use the following code to retrieve my custom XML
var retrievedXMLParts = workbook.CustomXMLParts.SelectByNamespace(NS.NamespaceName);
//FirstOrDefault always returns first saved data, LastOrDefault needs to be called to get the latest
//var customXMLPart = retrievedXMLParts.Cast<CustomXMLPart>().FirstOrDefault();
var customXMLPart = retrievedXMLParts.Cast<CustomXMLPart>().LastOrDefault();
var propertiesXML = customXMLPart != null ? customXMLPart.XML : String.Empty;
What I would like to achieve is to check if a custom XML exists update its content instead off adding it as duplicate
I think I have found a solution but it involves iterating through all the custom XML parts, deleting the one you want to update and then add again:
IEnumerator e = workbook.CustomXMLParts.GetEnumerator();
CustomXMLPart p;
while (e.MoveNext())
{
p = (CustomXMLPart) e.Current;
//p.BuiltIn will be true for internal buildin excel parts
if (p != null && !p.BuiltIn && p.NamespaceURI == NS.NamespaceName)
p.Delete();
}
I'm using this c# code to write data to xml file:
Employee[] employees = new Employee[2];
employees[0] = new Employee(1, "David", "Smith", 10000);
employees[1] = new Employee(12, "Cecil", "Walker", 120000);
using (XmlWriter writer = XmlWriter.Create("employees.xml"))
{
writer.WriteStartDocument();
writer.WriteStartElement("Employees");
foreach (Employee employee in employees)
{
writer.WriteStartElement("Employee");
writer.WriteElementString("ID", employee.Id.ToString());
writer.WriteElementString("FirstName", employee.FirstName);
writer.WriteElementString("LastName", employee.LastName);
writer.WriteElementString("Salary", employee.Salary.ToString());
writer.WriteEndElement();
}
writer.WriteEndElement();
writer.WriteEndDocument();
}
Now suppose I restart my application and I want to add new data to the xml file without losing the existed data, using the same way will overwrite the data on my xml file, I tried to figure out how to do that and I searched for a similar example but I couldn't come to anything , any ideas ??
Perhaps you should look at some examples using datasets and xml:
http://www.codeproject.com/Articles/13854/Using-XML-as-Database-with-Dataset
or use System.Xml.Serialization.XmlSerializer, when you dont't have amount of records.
Example using XmlDocument
XmlDocument xd = new XmlDocument();
xd.Load("employees.xml");
XmlNode nl = xd.SelectSingleNode("//Employees");
XmlDocument xd2 = new XmlDocument();
xd2.LoadXml("<Employee><ID>20</ID><FirstName>Clair</FirstName><LastName>Doner</LastName><Salary>13000</Salary></Employee>");
XmlNode n = xd.ImportNode(xd2.FirstChild,true);
nl.AppendChild(n);
xd.Save(Console.Out);
Using an xml writer for small amounts of data is awkward. You would be better of using an XDocument that you either initialize from scratch for the first run, or read from an existing file in subsequent runs.
Using XDocument you can manipulate the XML with XElement and XAttribute instances and then write the entire thing out to a file when you want to persist it.
Is it possible to add literal XML data within a C# code file? I'm currently using a multiline string literal but it gets messy as you can see. Any better way of doing this?
string XML = #"<?xml version=""1.0"" encoding=""utf-8""?>
<customUI xmlns=""http://schemas.example.com/customui"">
<toolbar id=""save"">
</toolbar>
</customUI>";
XML literals are a feature of VB.NET, not C#.
What you have posted is as close as you can get in C#.
You may want to consider replacing the embedded double quotes with single quotes though (as both types are valid XML).
For larger amounts of XML you may want to consider the answer from Marc - using an XML file (loaded once and stored in memory), so you can take advantage of the XML editor.
If the XML is big enough to get in the way, consider using a flat .xml file instead, either loaded from disk, or embedded as a resource. As long as you only load it once (perhaps in a static constructor) this will make no difference to performance. It will be considerably easier to maintain, as it will use the IDE's XML file editor. And it won't get in the way of your code.
With reference to my comment, I couldn't recall where I saw this, but I finally found the XmlBuilder link.
In retrospect, it seems Linq to XML would be your best bet. It's cleaner, faster and more maintainable than concatenating XML strings:
XNamespace ns = "http://schemas.example.com/customui";
XDocument doc = new XDocument(
new XDeclaration("1.0", "utf-8", "yes"),
new XElement(ns + "customUI",
new XElement(ns + "taskbar",
new XAttribute("id", "save"))
)
);
var stringWriter = new StringWriter();
doc.Save(stringWriter); //Write to StringWriter, preserving the declaration (<?xml version="1.0" encoding="utf-16" standalone="yes"?>)
var xmlString = stringWriter.ToString(); //Save as string
doc.Save(#"d:\out.xml"); //Save to file
As a peculiar, and very case-specific solution, if you happen to be working in an ASP.NET environment using the Razor engine, in a CSHTML file you can:
Func<MyType, HelperResult> xml = #<root>
<item>#(item.PropertyA)</item>
<item>#(item.PropertyB)</item>
<item>#(item.PropertyC)</item>
</root>;
With the addition of an extension method:
public static XDocument ToXDocument<T>(this Func<T, HelperResult> source, T item)
{
return XDocument.Parse(source(item).ToHtmlString());
}
You can then:
XDocument document = xml.ToXDocument(new MyType() {
PropertyA = "foo",
PropertyB = "bar",
PropertyC = "qux",
});
Again, peculiar? Yes. Case-specific? Yes. But it works, and gives great Intellisense. (mind you, it also will give a bunch of validity warnings, depending on the document validation version)
The closest we could have in C# would be through LINQ, something like that:
var xml = XDocument.Load(
new StringReader(#"<Books>
<Book author='Dan Brown'>The Da Vinci Code</Book>
<Book author='Dan Brown'>The Lost Symbol</Book>
</Books>"));
var query = from book in xml.Elements("Books").Elements("Book")
where book.Attribute("author").Value == "Dan Brown"
select book.Value;
foreach (var item in query) Console.WriteLine(item);
I need to create xml file that have N Level.
for ex. in my below example I have 'AlbumDetails' is root element and 'PrintPackage' is another child root and 'UpgradePackage' is another child root.
Can any one let me know how can i make N Level/Multi Level XML in c#.
<AlbumDetails>
<Album Id="203">
<Institute>Oxford</Institute>
<Venue>Wallingford School</Venue>
<PrintPackage>
<SizeName>Combination Pack</SizeName>
<Price>1.00</Price>
<Weight>60.00</Weight>
<UpgradePackage>
<SizeName>Upgrade 1</SizeName>
<Price>1.00</Price>
<Weight>60.00</Weight>
</UpgradePackage>
<SizeName>Standard Pack</SizeName>
<Price>90.0000</Price>
<Weight>600.0000</Weight>
</PrintPackage>
</Album>
</AlbumDetails>
You are looking for the XmlWriter class.
Update: In case you want to create a document similar to the one above:
var builder = new StringBuilder();
using (var writer = XmlWriter.Create(builder))
{
writer.WriteStartElement("AlbumDetails");
writer.WriteStartElement("Album");
writer.WriteAttributeString("Id", "203");
writer.WriteElementString("Venue", "Wallingford School");
writer.WriteStartElement("PrintPackage");
.... etc.
writer.WriteEndElement(); // close PrintPackage
writer.WriteEndElement(); // close Album
writer.WriteEndElement(); // close AlbumDetails
}
Console.WriteLine(builder.ToString());
Use XDocument and XElement from System.Xml.Linq ( Linq2Xml )
XDocument doc = new XDocument(new XDeclaration("1.0","utf-8","true"),
new XElement("AlbumDetails",
new XElement("Album",new XAttribute("Id","203"),
new XElement("Institute","Oxford"),
new XElement("Venue","Wallingford School")
...
)
)
);
If you are just looking for XElement only, you can build it up in a similar way. You can have a processingElement and create the XElement based on your logic and do
doc.Add(processingElement);
or
ele.Add(processingElement);
They're not really "child roots" - they're just elements which have other child elements.
Personally I'd use LINQ to XML. It's by far the simplest XML API I've used. For example:
var element = new XElement("AlbumDetails",
new XElement("Album",
new XAttribute("ID", 203"),
new XElement("Institute", "Oxford"),
new XElement("Venue", "Wallingford School"),
new XElement("PrintPackage",
new XElement("SizeName", "Combination Pack"),
// etc
new XElement("UpgradePackage",
new XElement("SizeName", "Upgrade 1"),
// etc
)
)
);
Of course, you don't have to build up everything in a single statement - you can add child nodes separately, potentially constructing them entirely separately. Indeed, you may want a separate method to create each "container" element.