Develop Reference

Update XLSX file changes whilst reading the file with XmlReader

We had a code which was loading the Excel XLSX document into the memory, doing some modifications with it and saving it back.
XmlDocument doc = new XmlDocument();
doc.Load(pp.GetStream());
XmlNode rootNode = doc.DocumentElement;
if (rootNode == null) return;
ProcessNode(rootNode);
if (this.fileModified)
{
doc.Save(pp.GetStream(FileMode.Create, FileAccess.Write));
}
This was working good with small files, but throwing OutOfMemory exceptions with some large Excel files. So we decided to change the approach and use XmlReader class to not load the file into the memory at once.
PackagePartCollection ppc = this.Package.GetParts();
foreach (PackagePart pp in ppc)
{
if (!this.xmlContentTypesXlsx.Contains(pp.ContentType)) continue;
using (XmlReader reader = XmlReader.Create(pp.GetStream()))
{
reader.MoveToContent();
while (reader.EOF == false)
{
XmlDocument doc;
XmlNode rootNode;
if (reader.NodeType == XmlNodeType.Element && reader.Name == "hyperlinks")
{
doc = new XmlDocument();
rootNode = doc.ReadNode(reader);
if (rootNode != null)
{
doc.AppendChild(rootNode);
ProcessNode(rootNode); // how can I save updated changes back to the file?
}
}
else if (reader.NodeType == XmlNodeType.Element && reader.Name == "row")
{
doc = new XmlDocument();
rootNode = doc.ReadNode(reader);
if (rootNode != null)
{
doc.AppendChild(rootNode);
ProcessNode(rootNode); // how can I save updated changes back to the file?
}
}
else
{
reader.Read();
}
}
}
}
This reads the file node by node and processes nodes we need (and changes some values there). However, I'm not sure how we can update those values back to the original Excel file.
I tried to use XmlWriter together with the XmlReader, but was not able to make it work. Any ideas?
UPDATE:
I tried to use #dbc's suggestions from the comments section, but it seems too slow to me. It probably will not throw OutOfMemory exceptions for huge files, but processing will take forever.
PackagePartCollection ppc = this.Package.GetParts();
foreach (PackagePart pp in ppc)
{
if (!this.xmlContentTypesXlsx.Contains(pp.ContentType)) continue;
StringBuilder strBuilder = new StringBuilder();
using (XmlReader reader = XmlReader.Create(pp.GetStream()))
{
using (XmlWriter writer = this.Package.FileOpenAccess == FileAccess.ReadWrite ? XmlWriter.Create(strBuilder) : null)
{
reader.MoveToContent();
while (reader.EOF == false)
{
XmlDocument doc;
XmlNode rootNode;
if (reader.NodeType == XmlNodeType.Element && reader.Name == "hyperlinks")
{
doc = new XmlDocument();
rootNode = doc.ReadNode(reader);
if (rootNode != null)
{
doc.AppendChild(rootNode);
ProcessNode(rootNode);
writer?.WriteRaw(rootNode.OuterXml);
}
}
else if (reader.NodeType == XmlNodeType.Element && reader.Name == "row")
{
doc = new XmlDocument();
rootNode = doc.ReadNode(reader);
if (rootNode != null)
{
doc.AppendChild(rootNode);
ProcessNode(rootNode);
writer?.WriteRaw(rootNode.OuterXml);
}
}
else
{
WriteShallowNode(writer, reader); // Used from the #dbc's suggested stackoverflow answers
reader.Read();
}
}
writer?.Flush();
}
}
}
NOTE 1: I'm using StringBuilder for the test, but was planning to switch to a temp file in the end.
NOTE 2: I tried flushing the XmlWriter after every 100 elements, but it's still slow.
Any ideas?

Try following. I've been use for a long time with huge xml files that give Out of Memory
using (XmlReader reader = XmlReader.Create("File Stream", readerSettings))
{
while (!reader.EOF)
{
if (reader.Name != "row")
{
reader.ReadToFollowing("row");
}
if (!reader.EOF)
{
XElement row = (XElement)XElement.ReadFrom(reader);
}
}
}
}

I did some more modifications with #dbc's help and now it works as I wanted.
PackagePartCollection ppc = this.Package.GetParts();
foreach (PackagePart pp in ppc)
{
try
{
if (!this.xmlContentTypesXlsx.Contains(pp.ContentType)) continue;
string tempFilePath = GetTempFilePath();
using (XmlReader reader = XmlReader.Create(pp.GetStream()))
{
using (XmlWriter writer = this.Package.FileOpenAccess == FileAccess.ReadWrite ? XmlWriter.Create(tempFilePath) : null)
{
while (reader.EOF == false)
{
if (reader.NodeType == XmlNodeType.Element && reader.Name == "hyperlinks")
{
XmlDocument doc = new XmlDocument();
XmlNode rootNode = doc.ReadNode(reader);
if (rootNode != null)
{
ProcessNode(rootNode);
if (writer != null)
{
rootNode.WriteTo(writer);
}
}
}
else if (reader.NodeType == XmlNodeType.Element && reader.Name == "row")
{
XmlDocument doc = new XmlDocument();
XmlNode rootNode = doc.ReadNode(reader);
if (rootNode != null)
{
ProcessNode(rootNode);
if (writer != null)
{
rootNode.WriteTo(writer);
}
}
}
else
{
WriteShallowNode(writer, reader); // Used from the #dbc's suggested StackOverflow answers
reader.Read();
}
}
}
}
if (this.packageChanged) // is being set in ProcessNode method
{
this.packageChanged = false;
using (var tempFile = File.OpenRead(tempFilePath))
{
tempFile.CopyTo(pp.GetStream(FileMode.Create, FileAccess.Write));
}
}
}
catch (OutOfMemoryException)
{
throw;
}
catch (Exception ex)
{
Log.Exception(ex, #"Failed to process a file."); // our inner log method
}
finally
{
if (!string.IsNullOrWhiteSpace(tempFilePath))
{
// Delete temp file
}
}
}

XMLReader Invalid XML Character Exception

I am parsing a big XML file ~500MB, and it contains some invalid XML character 0x07 , so you can imagine what's happening, the XMLReader is throwing an Invalid XML character exception, to handle this we streamed the Stream into StreamReader and used Regex.Replace and wrote the result to memory using StreamWriter and stream the clean version back to XMLReader, now I would like to avoid this and skip this filthy tag from the XMLReader directly, my question is if there's anyway to achieve that, below is the code snippet where I try to do this but it's throwing the exception at this line
var node = (XElement)XNode.ReadFrom(xr);
protected override IEnumerable<XElement> StreamReader(Stream stream, string elementName)
{
var arrTag = elementName.Split('|').ToList();
using (var xr = XmlReader.Create(stream, new XmlReaderSettings { CheckCharacters = false }))
{
while (xr.Read())
{
if (xr.NodeType == XmlNodeType.Element && arrTag.Contains(xr.Name))
{
var node = (XElement)XNode.ReadFrom(xr);
node.ReplaceWith(node.Elements().Where(e => e.Name != "DaylightSaveInfo"));
yield return node;
}
}
xr.Close();
}
}
XML SAMPLE, the invalid attribute DaylightSaveInfo
<?xml version="1.0" encoding="ISO-8859-1"?>
<LATree>
<LA className="BTT00NE" fdn="NE=9739">
<attr name="fdn">NE=9739</attr>
<attr name="IP">10.157.144.100</attr>
<attr name="realLatitude">0D0&apos;0"S</attr>
<attr name="realLongitude">0D0&apos;0"W</attr>
<attr name="DaylightSaveInfo">NO</attr>
</LA>
</LATree>

I just saw that Jon Skeet wrote something about this, so I cannot take credit really, but since his stature on SO is way above mine, I could perhaps gain a point or two for writing it. :)
First I wrote a class that overloads the TextReader class.
(Some reference material in the links.)
https://www.w3.org/TR/xml/#NT-Char
https://github.com/Microsoft/referencesource/blob/master/mscorlib/system/io/textreader.cs
class FilterInvalidXmlReader : System.IO.TextReader
{
private System.IO.StreamReader _streamReader;
public System.IO.Stream BaseStream => _streamReader.BaseStream;
public FilterInvalidXmlReader(System.IO.Stream stream) => _streamReader = new System.IO.StreamReader(stream);
public override void Close() => _streamReader.Close();
protected override void Dispose(bool disposing) => _streamReader.Dispose();
public override int Peek()
{
var peek = _streamReader.Peek();
while (IsInvalid(peek, true))
{
_streamReader.Read();
peek = _streamReader.Peek();
}
return peek;
}
public override int Read()
{
var read = _streamReader.Read();
while (IsInvalid(read, true))
{
read = _streamReader.Read();
}
return read;
}
public static bool IsInvalid(int c, bool invalidateCompatibilityCharacters)
{
if (c == -1)
{
return false;
}
if (invalidateCompatibilityCharacters && ((c >= 0x7F && c <= 0x84) || (c >= 0x86 && c <= 0x9F) || (c >= 0xFDD0 && c <= 0xFDEF)))
{
return true;
}
if (c == 0x9 || c == 0xA || c == 0xD || (c >= 0x20 && c <= 0xD7FF) || (c >= 0xE000 && c <= 0xFFFD))
{
return false;
}
return true;
}
}
Then I created a console application and in the main I put:
using (var memoryStream = new System.IO.MemoryStream(System.Text.Encoding.UTF8.GetBytes("<Test><GoodAttribute>a\u0009b</GoodAttribute><BadAttribute>c\u0007d</BadAttribute></Test>")))
{
using (var xmlFilteredTextReader = new FilterInvalidXmlReader(memoryStream))
{
using (var xr = System.Xml.XmlReader.Create(xmlFilteredTextReader))
{
while (xr.Read())
{
if (xr.NodeType == System.Xml.XmlNodeType.Element)
{
var xe = System.Xml.Linq.XElement.ReadFrom(xr);
System.Console.WriteLine(xe.ToString());
}
}
}
}
}
Hopefully this could help, or at least provide some starter point.

Following xml linq code runs without errors. I used in xml file following "NO" :
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.Xml.Linq;
using System.IO;
namespace ConsoleApplication108
{
class Program
{
const string FILENAME = #"c:\temp\test.xml";
static void Main(string[] args)
{
XmlReaderSettings settings = new XmlReaderSettings();
settings.CheckCharacters = false;
XmlReader reader = XmlReader.Create(FILENAME, settings);
XDocument doc = XDocument.Load(reader);
Dictionary<string, string> dict = doc.Descendants("attr")
.GroupBy(x => (string)x.Attribute("name"), y => (string)y)
.ToDictionary(x => x.Key, y => y.FirstOrDefault());
}
}
}

ReadOuterXml is throwing OutOfMemoryException reading part of large (1 GB) XML file

I am working on a large XML file and while running the application, XmlTextReader.ReadOuterXml() method is throwing memory exception.
Lines of codes are like,
XmlTextReader xr = null;
try
{
xr = new XmlTextReader(fileName);
while (xr.Read() && success)
{
if (xr.NodeType != XmlNodeType.Element)
continue;
switch (xr.Name)
{
case "A":
var xml = xr.ReadOuterXml();
var n = GetDetails(xml);
break;
}
}
}
catch (Exception ex)
{
//Do stuff
}
Using:
private int GetDetails (string xml)
{
var rootNode = XDocument.Parse(xml);
var xnodes = rootNode.XPathSelectElements("//A/B").ToList();
//Then working on list of nodes
}
Now while loading the XML files, the application throwing exception on the xr.ReadOuterXml() line. What can be done to avoid this? The size of XML is almost 1 GB.

The most likely reason you are getting a OutOfMemoryException in ReadOuterXml() is that you are trying to read in a substantial portion of the 1 GB XML document into a string, and are hitting the Maximum string length in .Net.
So, don't do that. Instead load directly from the XmlReader using XDocument.Load() with XmlReader.ReadSubtree():
using (var xr = XmlReader.Create(fileName))
{
while (xr.Read() && success)
{
if (xr.NodeType != XmlNodeType.Element)
continue;
switch (xr.Name)
{
case "A":
{
// ReadSubtree() positions the reader at the EndElement of the element read, so the
// next call to Read() moves to the next node.
using (var subReader = xr.ReadSubtree())
{
var doc = XDocument.Load(subReader);
GetDetails(doc);
}
}
break;
}
}
}
And then in GetDetails() do:
private int GetDetails(XDocument rootDocument)
{
var xnodes = rootDocument.XPathSelectElements("//A/B").ToList();
//Then working on list of nodes
return xnodes.Count;
}
Not only will this use less memory, it will also be more performant. ReadOuterXml() uses a temporary XmlWriter to copy the XML in the input stream to an output StringWriter (which you then parse a second time). This version of the algorithm completely skips this extra work. It also avoids creating strings large enough to go on the large object heap which can cause additional performance issues.
If this is still using too much memory you will need to implement SAX-like parsing for your XML where you only load one element <B> at a time. First, introduce the following extension method:
public static partial class XmlReaderExtensions
{
public static IEnumerable<XElement> WalkXmlElements(this XmlReader xmlReader, Predicate<Stack<XName>> filter)
{
Stack<XName> names = new Stack<XName>();
while (xmlReader.Read())
{
if (xmlReader.NodeType == XmlNodeType.Element)
{
names.Push(XName.Get(xmlReader.LocalName, xmlReader.NamespaceURI));
if (filter(names))
{
using (var subReader = xmlReader.ReadSubtree())
{
yield return XElement.Load(subReader);
}
}
}
if ((xmlReader.NodeType == XmlNodeType.Element && xmlReader.IsEmptyElement)
|| xmlReader.NodeType == XmlNodeType.EndElement)
{
names.Pop();
}
}
}
}
Then, use it as follows:
using (var xr = XmlReader.Create(fileName))
{
Predicate<Stack<XName>> filter =
(stack) => stack.Peek().LocalName == "B" && stack.Count > 1 && stack.ElementAt(1).LocalName == "A";
foreach (var element in xr.WalkXmlElements(filter))
{
//Then working on the specific node.
}
}

using (var reader = XmlReader.Create(fileName))
{
XmlDocument oXml = new XmlDocument();
while (reader.Read())
{
oXml.Load(reader);
}
}
For me above code resolved the issue when we return it to XmlDocument through XmlDocument Load method

Build XPath for node from XmlReader

I am writing an application which parses dynamic xml from various sources and traverses the XML and returns all the unique elements.
Given the sometimes very large size of the Xml files I am using a XmlReader to parse the Xml structure due to memory constraints.
public IDictionary<string, int> Discover(string filePath)
{
Dictionary<string, string> nodeTable = new Dictionary<string, string>();
using (XmlReader reader = XmlReader.Create(filePath))
{
while (!reader.EOF)
{
if (reader.NodeType == XmlNodeType.Element)
{
if (!nodeTable.ContainsKey(reader.LocalName))
{
nodeTable.Add(reader.LocalName, reader.Depth);
}
}
reader.Read();
}
}
Debug.WriteLine("The node table has {0} items.", nodeTable.Count);
return nodeTable;
}
This works a treat and is nice and performant, however the final piece of the puzzle eludes me, I am trying to generate the XPath for each element.
Now, this at first seemed straight forward using something like this.
var elements = new Stack<string>();
while (reader.Read())
{
switch (reader.NodeType)
{
case XmlNodeType.Element:
elements.Push(reader.LocalName);
break;
case XmlNodeType.EndElement:
elements.Pop();
break;
case XmlNodeType.Text:
path = string.Join("/", elements.Reverse());
break;
}
}
But this only really gives me one part of the solution. Given that I wish to return the XPath for every node in the tree which contains data and also detect if a given node tree contains nested collections of data.
i.e.
<customers>
<customer id=2>
<name>ted smith</name>
<addresses>
<address1>
<line1></line1>
</address1>
<address2>
<line1></line1>
<line2></line2>
</address2>
</addresses>
</customer>
<customer id=322>
<name>smith mcsmith</name>
<addresses>
<address1>
<line1></line1>
<line2></line2>
</address1>
<address2>
<line1></line1>
<line2></line2>
</address2>
</addresses>
</customer>
</customers>
Keeping in mind the data is completely dynamic and the schema is unknown.
So the output should include
/customer/name
/customer/address1/line1
/customer/address1/line2
/customer/address2/line1
/customer/address2/line2

I like using recursive method rather than push/pop. See code below
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.IO;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
string input =
"<customers>" +
"<customer id=\"2\">" +
"<name>ted smith</name>" +
"<addresses>" +
"<address1>" +
"<line1></line1>" +
"</address1>" +
"<address2>" +
"<line1></line1>" +
"<line2></line2>" +
"</address2>" +
"</addresses>" +
"</customer>" +
"<customer id=\"322\">" +
"<name>smith mcsmith</name>" +
"<addresses>" +
"<address1>" +
"<line1></line1>" +
"<line2></line2>" +
"</address1>" +
"<address2>" +
"<line1></line1>" +
"<line2></line2>" +
"</address2>" +
"</addresses>" +
"</customer>" +
"</customers>";
StringReader sReader = new StringReader(input);
XmlReader reader = XmlReader.Create(sReader);
Node root = new Node();
ReadNode(reader, root);
}
static bool ReadNode(XmlReader reader, Node node)
{
Boolean done = false;
Boolean endElement = false;
while(done = reader.Read())
{
switch (reader.NodeType)
{
case XmlNodeType.Element:
if (node.name.Length == 0)
{
node.name = reader.Name;
GetAttrubutes(reader, node);
}
else
{
Node newNode = new Node();
newNode.name = reader.Name;
if (node.children == null)
{
node.children = new List<Node>();
}
node.children.Add(newNode);
GetAttrubutes(reader, newNode);
done = ReadNode(reader, newNode);
}
break;
case XmlNodeType.EndElement:
endElement = true;
break;
case XmlNodeType.Text:
node.text = reader.Value;
break;
case XmlNodeType.Attribute:
if (node.attributes == null)
{
node.attributes = new Dictionary<string, string>();
}
node.attributes.Add(reader.Name, reader.Value);
break;
}
if (endElement)
break;
}
return done;
}
static void GetAttrubutes(XmlReader reader, Node node)
{
for (int i = 0; i < reader.AttributeCount; i++)
{
if (i == 0) node.attributes = new Dictionary<string, string>();
reader.MoveToNextAttribute();
node.attributes.Add(reader.Name, reader.Value);
}
}
}
public class Node
{
public string name = string.Empty;
public string text = string.Empty;
public Dictionary<string, string> attributes = null;
public List<Node> children = null;
}
}
​

how to read xml files in C# ?

I have a code which reads an xml file. There are some parts I dont understand.
From my understanding , the code will create an xml file with 2 elements,
"Product" and "OtherDetails" . How come we only have to use writer.WriteEndElement();
once when we used writer.WriteStartElement twice ? , shouldn't we close each
writer.WriteStartElement statement with a writer.WriteEndElement() statement ?
using System.Xml;
public class Program
{
public static void Main()
{
XmlWriterSettings settings = new XmlWriterSettings();
settings.Indent = true;
XmlWriter writer = XmlWriter.Create("Products.xml", settings);
writer.WriteStartDocument();
writer.WriteComment("This file is generated by the program.");
writer.WriteStartElement("Product"); // first s
writer.WriteAttributeString("ID", "001");
writer.WriteAttributeString("Name", "Soap");
writer.WriteElementString("Price", "10.00")
// Second Element
writer.WriteStartElement("OtherDetails");
writer.WriteElementString("BrandName", "X Soap");
writer.WriteElementString("Manufacturer", "X Company");
writer.WriteEndElement();
writer.WriteEndDocument();
writer.Flush();
writer.Close();
}
}
using System;
using System.Xml;
public class Program
{
public static void Main()
{
XmlReader reader = XmlReader.Create("Products.xml");
while (reader.Read())
{
if (reader.NodeType == XmlNodeType.Element
&& reader.Name == "Product")
{
Console.WriteLine("ID = " + reader.GetAttribute(0));
Console.WriteLine("Name = " + reader.GetAttribute(1));
while (reader.NodeType != XmlNodeType.EndElement)
{
reader.Read();
if (reader.Name == "Price")
{
while (reader.NodeType != XmlNodeType.EndElement)
{
reader.Read();
if (reader.NodeType == XmlNodeType.Text)
{
Console.WriteLine("Price = {0:C}", Double.Parse(reader.Value));
}
}
reader.Read();
} //end if
if (reader.Name == "OtherDetails")
{
while (reader.NodeType != XmlNodeType.EndElement)
{
reader.Read();
if (reader.Name == "BrandName")
{
while (reader.NodeType != XmlNodeType.EndElement)
{
reader.Read();
if (reader.NodeType == XmlNodeType.Text)
{
Console.WriteLine("Brand Name = " + reader.Value);
}
}
reader.Read();
} //end if
if (reader.Name == "Manufacturer")
{
while (reader.NodeType != XmlNodeType.EndElement)
{
reader.Read();
if (reader.NodeType == XmlNodeType.Text)
{
Console.WriteLine("Manufacturer = " + reader.Value);
}
}
} //end if
}
} //end if
} //end while
} //end if
} //end while
}
}
I don't get this part:
if (reader.Name == "OtherDetails")
{
while (reader.NodeType != XmlNodeType.EndElement)
{
reader.Read();
if (reader.Name == "BrandName")
{
while (reader.NodeType != XmlNodeType.EndElement)
{
reader.Read();
if (reader.NodeType == XmlNodeType.Text)
{
Console.WriteLine("Brand Name = " + reader.Value);
}
}
notice how the condition while (reader.NodeType != XmlNodeType.EndElement) has been used twice ?
why is that we don't have to specify
if (reader.NodeType == XmlNodeType.Element for OtherDetails) as we did for Product,
like this
if (reader.NodeType == XmlNodeType.Element
&& reader.Name == "OtherDetails")
{}

To answer your first question:
As the MSDN documentation for XmlWriter.WriteEndDocument() says:
Closes any open elements or attributes and puts the writer back in the Start state.
So it will automatically close any open elements for you. In fact, you can remove the call to WriteEndElement() altogether and it will still work ok.
And as people are saying in the comments above, you should perhaps consider using Linq-to-XML.
It can make things much easier. For example, to create the XML structure from your program using Linq-to-XML you can do this:
var doc = new XDocument(
new XElement("Product",
new XAttribute("ID", "001"), new XAttribute("Name", "Soap"),
new XElement("Price", 10.01),
new XElement("OtherDetails",
new XElement("BrandName", "X Soap"),
new XElement("Manufacturer", "X Company"))));
File.WriteAllText("Products.xml", doc.ToString());
If you were reading data from the XML, you can use var doc = XDocument.Load("Filename.xml") to load the XML from a file, and then getting the data out is as simple as:
double price = double.Parse(doc.Descendants("Price").Single().Value);
string brandName = doc.Descendants("BrandName").Single().Value;
Or alternatively (casting):
double price = (double) doc.Descendants("Price").Single();
string brandName = (string) doc.Descendants("BrandName").Single();
(In case you're wondering how on earth we can cast an object of type XElement like that: It's because a load of explict conversion operators are defined for XElement.)

If you need anything strait forward (no reading or research), here is what I did:
I recently wrote a custom XML parsing method for my MenuStrip for WinForms (it had hundreds of items and XML was my best bet).
// load the document
// I loaded mine from my C# resource file called TempResources
XDocument doc = XDocument.Load(new MemoryStream(Encoding.UTF8.GetBytes(TempResources.Menu)));
// get the root element
// (var is an auto token, it becomes what ever you assign it)
var elements = doc.Root.Elements();
// iterate through the child elements
foreach (XElement node in elements)
{
// if you know the name of the attribute, you can call it
// mine was 'name'
// (if you don't know, you can call node.Attributes() - this has the name and value)
Console.WriteLine("Loading list: {0}", node.Attribute("name").Value);
// in my case, every child had additional children, and them the same
// *.Cast<XElement>() would give me the array in a datatype I can work with
// menu_recurse(...) is just a resursive helper method of mine
menu_recurse(node.Elements().Cast<XElement>().ToArray()));
}
(My answer can also be found here: Reading an XML File With Linq - though it unfortunately is not Linq)

Suppose if you want to read an xml file we need to use a dataset,because xml file internally converts into datatables using a dataset.Use the following line of code to access the file and to bind the dataset with the xml data.
DataSet ds=new DataSet();
ds.ReadXml(HttpContext.Current.Server.MapPath("~/Labels.xml");
DataSet comprises of many datatables,count of those datatables depends on number of Parent Child Tags in an xml file