I have been using the traditional way of serializing content with the following code
private void SaveToXml(IdentifiableEntity IE)
{
try
{
XmlSerializer serializer = new XmlSerializer(IE.GetType());
TextWriter textWriter = new StreamWriter(IE.FilePath);
serializer.Serialize(textWriter, IE);
textWriter.Close();
}
catch (Exception e )
{
Console.WriteLine("erreur : "+ e);
}
}
private T LoadFromXml<T>(string path)
{
XmlSerializer deserializer = new XmlSerializer(typeof(T));
TextReader textReader = new StreamReader(path);
T entity = (T)deserializer.Deserialize(textReader);
textReader.Close();
return entity;
}
Though this approach does the trick, i find it a bit annoying that all my properties have to be public, that i need to tag the properties sometimes [XmlAttribute|XmlElement| XmlIgnore] and that it doesn't deal with dictionaries.
My question is : Is there a better way of serializing objects in c#, a way that with less hassle, more modern and easy to use?
First of all, I would suggest to use "using" blocks in your code.(Sample code)
If my understanding is OK, you are looking for a fast way to build your model classes that you will use during your deserialize/serialize operations.
Every Xml file is different and I don't know any generic way to serialize / deserialize them. At one moment you have to know if there will be an attribute, or elements or if any element can be null etc.
Assuming that you already have a sample XML file with a few lines which gives you general view of how it will look like
I would suggest to use xsd (miracle tool)
xsd yourXMLFileName.xml
xsd yourXMLFileName.xsd \classes
This tool will generate you every time model classes for the XML file you want to work it.
Than you serialize and deserialize easily
To deserialize (assuming that you'll get a class named XXXX representing root node in your xml)
XmlSerializer ser = new XmlSerializer(typeof(XXXX));
XXXX yourVariable;
using (XmlReader reader = XmlReader.Create(#"C:\yyyyyy\yyyyyy\YourXmlFile.xml"))
{
yourVariable= (XXXX) ser.Deserialize(reader);
}
To serialize
var serializer = new XmlSerializer(typeof(XXXX));
using(var writer = new StreamWriter(#"C:\yyyyyy\yyyyyy\YourXmlFile.xml"))
{
serializer.Serialize(writer, yourVariable);
}
Related
I need to deserialize an XML response from an external service containing more than 100.000 rows, but I have a problem unescapeing numeric character references in various places. Since the DOM is complex, I need to have a global solution that applies to the whole document, and not it's specific elements. I have the following situations:
<text>First & Second</text>
I use the following XmlSerializer implementation:
public T DeserializeXmlReader<T>(string path) where T : class
{
XmlSerializer serializer = new XmlSerializer(typeof(T));
using (FileStream fileStream = new FileStream(path, FileMode.Open))
{
using (XmlReader xmlReader = XmlReader.Create(fileStream))
{
return (T)serializer.Deserialize(xmlReader);
}
}
}
After deserialization, I will get the following result: "First & Second", instead of "First & Second". I am not sure if there is an additional step I need to undertake to get the "&" deserialized correctly?
Note: After doing some research, I believe the problem might be similar to this one, but I'm not sure if it's applicable since this concerns php: How to deserialize a xml string along with NCR unescaping?
I'm trying to convert from a huge JSON file(2GB) to xml file. I have some troubles reading the huge JSON file.
I've been researching about how i can read huge JSON files.
I found this:
Out of memory exception while loading large json file from disk
How to parse huge JSON file as stream in Json.NET?
Parsing large json file in .NET
It seems that i'm duplicating my question but i have some troubles which aren't solved in these posts.
So, i need to load the huge JSON File and the community propose something like this:
MyObject o;
using (StreamReader sr = new StreamReader("foo.json"))
using (JsonTextReader reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
reader.SupportMultipleContent = true;
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject)
{
// Deserialize each object from the stream individually and process it
var o = serializer.Deserialize<MyObject>(reader);
//Do something with the object
}
}
}
So, We can read by parts and deserialize objects one by one.
I'll show you my code
JsonSerializer serializer = new JsonSerializer();
string hugeJson = "hugJSON.json";
using (FileStream s = File.Open(hugeJson , FileMode.Open))
{
using (StreamReader sr = new StreamReader(s))
{
using (JsonReader reader = new JsonTextReader(sr))
{
reader.SupportMultipleContent = true;
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject)
{
var jsonObject = serializer.Deserialize(reader);
string xmlString = "";
XmlDocument doc = JsonConvert.DeserializeXmlNode(jsonObject.ToString(), "json");
using (var stringWriter = new StringWriter())
{
using (var xmlTextWriter = XmlWriter.Create(stringWriter))
{
doc.WriteTo(xmlTextWriter);
xmlTextWriter.Flush();
xmlString = stringWriter.GetStringBuilder().ToString();
}
}
}
}
}
}
}
But when i try doc.WriteTo(xmlTextWriter), i get Exception of type System.OutOfMemoryException was thrown.
I've been trying with BufferedStream. This class allows me manage big files but i have another problem.
I'm reading in byte[] format. When i convert to string, the json is splitted and i can't parse to xml file because there are missing characters
for example:
{ foo:[{
foo:something,
foo1:something,
foo2:something
},
{
foo:something,
foo:som
it is cutted.
Is any way to read a huge JSON and convert to XML without load the JSON by parts? or i could load a convert by parts but i don't know how to do this.
Any ideas?
UPDATE:
I have been trying with this code:
static void Main(string[] args)
{
string json = "";
string pathJson = "foo.json";
//Read file
string temp = "";
using (FileStream fs = new FileStream(pathJson, FileMode.Open))
{
using (BufferedStream bf = new BufferedStream(fs))
{
byte[] array = new byte[70000];
while (bf.Read(array, 0, 70000) != 0)
{
json = Encoding.UTF8.GetString(array);
temp = String.Concat(temp, json);
}
}
}
XmlDocument doc = new XmlDocument();
doc = JsonConvert.DeserializeXmlNode(temp, "json");
using (var stringWriter = new StringWriter())
using (var xmlTextWriter = XmlWriter.Create(stringWriter))
{
doc.WriteTo(xmlTextWriter);
xmlTextWriter.Flush();
xmlString = stringWriter.GetStringBuilder().ToString();
}
File.WriteAllText("outputPath", xmlString);
}
This code convert from json file to xml file. but when i try to convert a big json file (2GB), i can't. The process cost a lot of time and the string doesn't have capacity to store all the json. How i can store it? Is any way to do this conversion without use the datatype string?
UPDATE:
The json format is:
[{
'key':[some things],
'data': [some things],
'data1':[A LOT OF ENTRIES],
'data2':[A LOT OF ENTRIES],
'data3':[some things],
'data4':[some things]
}]
Out-of-memory exceptions in .Net can be caused by several problems including:
Allocating too much total memory.
If this might be happening, check whether you are running in 64-bit mode as described here. If not, rebuild in 64-bit mode as described here and re-test.
Allocating too many objects on the large object heap causing memory fragmentation.
Allocating a single object that is larger than the .Net object size limit.
Failing to dispose of unmanaged memory (not applicable here).
In your case, you may be trying to allocate too much total memory but are definitely allocating three very large objects: the in-memory temp JSON string, the in-memory xmlString XML string and the in-memory stringWriter.
You can substantially reduce your memory footprint and completely eliminate these objects by constructing an XDocument or XmlDocument directly via a streaming translation from the JSON file. Then afterward, write the document directly to the XML file using XDocument.Save() or XmlDocument.Save().
To do this, you will need to allocate your own XmlNodeConverter, then construct a JsonSerializer using it and deserialize as shown in Deserialize JSON from a file. The following method(s) do the trick:
public static partial class JsonExtensions
{
public static XDocument LoadXNode(string pathJson, string deserializeRootElementName)
{
using (var stream = File.OpenRead(pathJson))
return LoadXNode(stream, deserializeRootElementName);
}
public static XDocument LoadXNode(Stream stream, string deserializeRootElementName)
{
// Let caller dispose the underlying streams.
using (var textReader = new StreamReader(stream, Encoding.UTF8, true, 1024, true))
return LoadXNode(textReader, deserializeRootElementName);
}
public static XDocument LoadXNode(TextReader textReader, string deserializeRootElementName)
{
var settings = new JsonSerializerSettings
{
Converters = { new XmlNodeConverter { DeserializeRootElementName = deserializeRootElementName } },
};
using (var jsonReader = new JsonTextReader(textReader) { CloseInput = false })
return JsonSerializer.CreateDefault(settings).Deserialize<XDocument>(jsonReader);
}
public static void StreamJsonToXml(string pathJson, string pathXml, string deserializeRootElementName, SaveOptions saveOptions = SaveOptions.None)
{
var doc = LoadXNode(pathJson, deserializeRootElementName);
doc.Save(pathXml, saveOptions);
}
}
Then use them as follows:
JsonExtensions.StreamJsonToXml(pathJson, outputPath, "json");
Here I am using XDocument instead of XmlDocument because I believe (but have not checked personally) that it uses less memory, e.g. as reported in Some hard numbers about XmlDocument, XDocument and XmlReader (x86 versus x64) by Ken Lassesen.
This approach eliminates the three large objects mentioned previously and substantially reduces the chance of running out of memory due to problems #2 or #3.
Demo fiddle here.
If you are still running out of memory even after ensuring you are running in 64-bit mode and streaming directly from and to your file(s) using the methods above, then it may simply be that your XML is too large to fit in your computer's virtual memory space using XDocument or XmlDocument. If that is so, you will need to adopt a pure streaming solution that transforms from JSON to XML on the fly as it streams. Unfortunately, Json.NET does not provide this functionality out of the box, so you will need a more complex solution.
So, what are your options?
You could fork your own version of XmlNodeConverter.cs and rewrite ReadElement(JsonReader reader, IXmlDocument document, IXmlNode currentNode, string propertyName, XmlNamespaceManager manager) to write directly to an XmlWriter instead of an IXmlDocument.
While probably doable with a couple days effort, the difficulty would seem to exceed that of a single stackoverflow answer.
You could use the reader returned by JsonReaderWriterFactory to translate JSON to XML on the fly, and pass that reader directly to XmlWriter.WriteNode(XmlReader). The readers and writers returned by this factory are used internally by DataContractJsonSerializer but can be used directly as well.
If your JSON has a fixed schema (which is unclear from your question) you have many more straightforward options. Incrementally deserializing to some c# data model as shown in Parsing large json file in .NET and re-serializing that model to XML is likely to use much less memory than loading into some generic DOM such as XDocument.
Option #2 can be implemented very simply, as follows:
using (var stream = File.OpenRead(pathJson))
using (var jsonReader = JsonReaderWriterFactory.CreateJsonReader(stream, XmlDictionaryReaderQuotas.Max))
{
using (var xmlWriter = XmlWriter.Create(outputPath))
{
xmlWriter.WriteNode(jsonReader, true);
}
}
However, the XML thereby produced is much less pretty than the XML generated by XmlNodeConverter. For instance, given the simple input JSON
{"Root":[{
"key":["a"],
"data": [1, 2]
}]}
XmlNodeConverter will create the following XML:
<json>
<Root>
<key>a</key>
<data>1</data>
<data>2</data>
</Root>
</json>
While JsonReaderWriterFactory will create the following (indented for clarity):
<root type="object">
<Root type="array">
<item type="object">
<key type="array">
<item type="string">a</item>
</key>
<data type="array">
<item type="number">1</item>
<item type="number">2</item>
</data>
</item>
</Root>
</root>
The exact format of the XML generated can be found in
Mapping Between JSON and XML.
Still, once you have valid XML, there are streaming XML-to-XML transformation solutions that will allow you to transform the generated XML to your final, desired format, including:
C# XSLT Transforming Large XML Files Quickly.
How to: Perform Streaming Transform of Large XML Documents (C#).
Combining the XmlReader and XmlWriter classes for simple streaming transformations.
Is it possible to do the other way?
Unfortunately
JsonReaderWriterFactory.CreateJsonWriter().WriteNode(xmlReader, true);
isn't really suited for conversion of arbitrary XML to JSON as it only allows for conversion of XML with the precise schema specified by Mapping Between JSON and XML.
Furthermore, when converting from arbitrary XML to JSON the problem of array recognition exists: JSON has arrays, XML doesn't, it only has repeating elements. To recognize repeating elements (or tuples of elements where identically named elements may not be adjacent) and convert them to JSON array(s) requires buffering either the XML input or the JSON output (or a complex two-pass algorithm). Mapping Between JSON and XML avoids the problem by requiring type="object" or type="array" attributes.
I am trying to deserialize an XML file, and it works fine except for nodes that contain HTML tags. Here is a snippet from the XML file:
<article mdate="2011-12-29" key="tr/trier/MI99-02" publtype="informal publication">
<author>Friedemann Leibfritz</author>
<title>A LMI-Based Algorithm for Designing Suboptimal Static H<sub>2</sub>/H<sub>infinity</sub> Output Feedback Controllers</title>
<journal>Universität Trier, Mathematik/Informatik, Forschungsbericht</journal>
<volume>99-02</volume>
<year>1999</year>
</article>
Then, I am getting the error:
{"Unexpected node type Element. ReadElementString method can only be called on elements with simple or empty content. Line 1148, position 64."}
The error occurs at:
A LMI-Based Algorithm for Designing Suboptimal Static H2/Hinfinity Output Feedback Controllers
where HTML tags sub and /sub exist.
Is there a way to deserialize the title node as a whole, ignoring the HTML tags? Below is a portion of my code:
XmlReaderSettings readerSettings = new XmlReaderSettings
{
DtdProcessing = DtdProcessing.Parse,
XmlResolver = new LocalXhtmlXmlResolver()
};
XmlRootAttribute xRoot = new XmlRootAttribute();
xRoot.ElementName = "dblp";
xRoot.IsNullable = true;
XmlSerializer deserializer;
XmlReader textReader;
deserializer = new XmlSerializer(typeof(List<Entity.Article>), xRoot);
textReader = XmlReader.Create(xmlPath, readerSettings);
List<Entity.Article> articleList;
articleList = (List<Entity.Article>)deserializer.Deserialize(textReader);
textReader.Close();
Any help would be very much appreciated - Thanks!
Your XML is not properly escaped. The parser has no way of knowing that those tags aren't meant to be part of the XML document, and when they are treated as such, your XML is invalid, because an element is nested within the value of another element.
That XML snippet, correctly escaped would be
<article mdate="2011-12-29" key="tr/trier/MI99-02" publtype="informal publication">
<author>Friedemann Leibfritz</author>
<title>A LMI-Based Algorithm for Designing Suboptimal Static H<sub>2</sub>/H<sub>infinity</sub> Output Feedback Controllers</title>
<journal>Universität Trier, Mathematik/Informatik, Forschungsbericht</journal>
<volume>99-02</volume>
<year>1999</year>
</article>
As the comments to the previous answer point out- as developers we don't always have the luxury of formatting the XML prior to deserialization. There is a much more elegant solution, in my opinion, that satisfies the original question.
Serializer
public static T ParseXml<T>(this string #this) where T : class
{
var serializer = new XmlSerializer(typeof(T));
serializer.UnknownElement += Serializer_UnknownElement;
return serializer.Deserialize(new StringReader(#this)) as T;
}
Handle Problematic Fields
private static void Serializer_UnknownElement(object sender, XmlElementEventArgs e)
{
if (e.ObjectBeingDeserialized is Article article)
{
if (e.Element.Name == "title")
{
article.Title_Custom = e.Element.InnerXml;
return;
}
}
}
Modifications to Article Class
public class Article{
// include your other fields that are not problematic
public string Title_Custom { get; set; }
}
Usage
var myArticles = articlesXmlString.Parse<List<Article>>();
Console.Out(myArticles[0].Title_Custom); // "A LMI-Based Algorithm for Designing Suboptimal Static H<sub>2</sub>/H<sub>infinity</sub> Output Feedback Controllers"
Because the name of the property is now Title_Custom it will naturally be skipped as part of the deserialization process. The Serializer_UnknownElement method will then read in the <title> field as an unknown field. Then, you simply pull the entire contents of the inner XML.
The inclusion of <sup> will trip the Serializer_UnknownElement as well, but as you have no condition for it, it will skip over it.
The net result is that Title_Custom will now contain the full HTML snippet as expected.
How is the LINQ to XML X-DOM from the System.Xml.Linq namespace internally implemented? (XNode, XElement, etc.)
Is it utilizing standard high-performing one-way XmlReader/XmlWriter from the other XML namespaces or something else?
The reason I'm asking is that I'm trying to figure out in which circumstances could or should be used as performance is always a concern.
Using Reflector (or, when that's no longer free, ILSpy :); no I'm not an employee - just spreading the word surreptitiously!) it appears all the load/save methods channel through to XmlReader and XmlWriter.
For example - XElement's implementation of Load(Stream, LoadOptions) does this:
public static XElement Load(Stream stream, LoadOptions options)
{
XmlReaderSettings xmlReaderSettings = XNode.GetXmlReaderSettings(options);
using (XmlReader reader = XmlReader.Create(stream, xmlReaderSettings))
{
return Load(reader, options);
}
}
And it's a similar story for all the other static methods - including Parse.
But then there is the XStreamingElement constructor - however I can't find any real usage of it outside of the XElement class itself. Looks like this could be an optimised type for loading that, as yet, isn't used by much.
Equally, the Save and WriteTo methods ultimately use an XmlWriter instance - e.g:
public void Save(string fileName, SaveOptions options)
{
XmlWriterSettings xmlWriterSettings = XNode.GetXmlWriterSettings(options);
using (XmlWriter writer = XmlWriter.Create(fileName, xmlWriterSettings))
{
this.Save(writer);
}
}
So at least from a performance point of view they started with the right types :)
How can you do a streaming read on a large XML file that contains a xs:sequence just below root element, without loading the whole file into a XDocument instance in memory?
Going with a SAX-style element parser and the XmlTextReader class created with XmlReader.Create would be a good idea, yes. Here's a slightly-modified code example from CodeGuru:
void ParseURL(string strUrl)
{
try
{
using (var reader = XmlReader.Create(strUrl))
{
while (reader.Read())
{
switch (reader.NodeType)
{
case XmlNodeType.Element:
var attributes = new Hashtable();
var strURI = reader.NamespaceURI;
var strName = reader.Name;
if (reader.HasAttributes)
{
for (int i = 0; i < reader.AttributeCount; i++)
{
reader.MoveToAttribute(i);
attributes.Add(reader.Name,reader.Value);
}
}
StartElement(strURI,strName,strName,attributes);
break;
//
//you can handle other cases here
//
//case XmlNodeType.EndElement:
// Todo
//case XmlNodeType.Text:
// Todo
default:
break;
}
}
}
catch (XmlException e)
{
Console.WriteLine("error occured: " + e.Message);
}
}
}
}
I can't add a comment, since I just signed up but the code sample posted by Hirvox and currently selected as the answer has a bug in it. It should not have the new statement when using the static Create method.
Current:
using (var reader = new XmlReader.Create(strUrl))
Fixed:
using (var reader = XmlReader.Create(strUrl))
I think it's not possible if you want to use object model (i.e. XElement\XDocument) to query XML. Obviously, you can't build XML objects tree without reading enough data. However you can use XmlReader class.
The XmlReader class reads XML data
from a stream or file. It provides
non-cached, forward-only, read-only
access to XML data.
Heres is a howto: http://support.microsoft.com/kb/301228/en-us Just remember that you should not use XmlTextReader but instead XmlReader in conjunction with XmlReader.Create
I'm confused by the mention of the "xs:sequence" - this is a XML Schema element.
Are you trying to open a large XML Schema file? Are you open a large XML file that is based on that schema? Or are you trying to open a large XML file and validate it at the same time?
None of these situations should provide you with a problem using the standard XmlReader (or XmlValidatingReader).
Reading XML with XMLReader: http://msdn.microsoft.com/en-us/library/9d83k261(VS.80).aspx
That code sample tries to turn XmlReader style code into SAX style code - if you're writing code from scratch I'd just use XmlReader as it was intended - Pull not Push.