Develop Reference

Although I'm ignoring namespaces when deserializing xml it does not ignore namespace of root node

I'm making a XML deserializer for a certain type of XML. I already have one question regarding it :-) Back then I was trying to conform the namespaces, I have in the sample file. But the truth is - I get XMLs from many different sources. Although they're basically the same XML, some of them follow different namespaces, or add redundant namespaces of their own. That's why I removed namespaces from code (inside XmlRoot Attributes) and used the following code to ignore namespaces:
private Jpk DeserializeInputFile(string path)
{
Jpk obj;
var fileContent = File.ReadAllText(path);
using (TextReader textReader = new StringReader(fileContent))
{
using (XmlTextReader reader = new XmlTextReader(textReader))
{
reader.Namespaces = false;
XmlSerializer serializer = new XmlSerializer(typeof(Jpk));
obj = (Jpk)serializer.Deserialize(reader);
}
}
return obj;
}
Now I encountered a different situation. I received an XML file that starts with the following root:
<tns:JPK xmlns:etd="http://crd.gov.pl/xml/schematy/dziedzinowe/mf/2018/08/24/eD/DefinicjeTypy/" xmlns:tns="http://jpk.mf.gov.pl/wzor/2019/09/27/09271/">
When trying to deserialize it I get the following exception:
System.InvalidOperationException: There is an error in XML document (1, 57).
---> System.InvalidOperationException: <tns:JPK xmlns=''> was not expected.
Previously there was no namespace in the root node. No tns:JPK, just JPK.
How can I ignore this namespace in root node?

How to convert from huge JSON file to xml file in C#

I'm trying to convert from a huge JSON file(2GB) to xml file. I have some troubles reading the huge JSON file.
I've been researching about how i can read huge JSON files.
I found this:
Out of memory exception while loading large json file from disk
How to parse huge JSON file as stream in Json.NET?
Parsing large json file in .NET
It seems that i'm duplicating my question but i have some troubles which aren't solved in these posts.
So, i need to load the huge JSON File and the community propose something like this:
MyObject o;
using (StreamReader sr = new StreamReader("foo.json"))
using (JsonTextReader reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
reader.SupportMultipleContent = true;
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject)
{
// Deserialize each object from the stream individually and process it
var o = serializer.Deserialize<MyObject>(reader);
//Do something with the object
}
}
}
So, We can read by parts and deserialize objects one by one.
I'll show you my code
JsonSerializer serializer = new JsonSerializer();
string hugeJson = "hugJSON.json";
using (FileStream s = File.Open(hugeJson , FileMode.Open))
{
using (StreamReader sr = new StreamReader(s))
{
using (JsonReader reader = new JsonTextReader(sr))
{
reader.SupportMultipleContent = true;
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject)
{
var jsonObject = serializer.Deserialize(reader);
string xmlString = "";
XmlDocument doc = JsonConvert.DeserializeXmlNode(jsonObject.ToString(), "json");
using (var stringWriter = new StringWriter())
{
using (var xmlTextWriter = XmlWriter.Create(stringWriter))
{
doc.WriteTo(xmlTextWriter);
xmlTextWriter.Flush();
xmlString = stringWriter.GetStringBuilder().ToString();
}
}
}
}
}
}
}
But when i try doc.WriteTo(xmlTextWriter), i get Exception of type System.OutOfMemoryException was thrown.
I've been trying with BufferedStream. This class allows me manage big files but i have another problem.
I'm reading in byte[] format. When i convert to string, the json is splitted and i can't parse to xml file because there are missing characters
for example:
{ foo:[{
foo:something,
foo1:something,
foo2:something
},
{
foo:something,
foo:som
it is cutted.
Is any way to read a huge JSON and convert to XML without load the JSON by parts? or i could load a convert by parts but i don't know how to do this.
Any ideas?
UPDATE:
I have been trying with this code:
static void Main(string[] args)
{
string json = "";
string pathJson = "foo.json";
//Read file
string temp = "";
using (FileStream fs = new FileStream(pathJson, FileMode.Open))
{
using (BufferedStream bf = new BufferedStream(fs))
{
byte[] array = new byte[70000];
while (bf.Read(array, 0, 70000) != 0)
{
json = Encoding.UTF8.GetString(array);
temp = String.Concat(temp, json);
}
}
}
XmlDocument doc = new XmlDocument();
doc = JsonConvert.DeserializeXmlNode(temp, "json");
using (var stringWriter = new StringWriter())
using (var xmlTextWriter = XmlWriter.Create(stringWriter))
{
doc.WriteTo(xmlTextWriter);
xmlTextWriter.Flush();
xmlString = stringWriter.GetStringBuilder().ToString();
}
File.WriteAllText("outputPath", xmlString);
}
This code convert from json file to xml file. but when i try to convert a big json file (2GB), i can't. The process cost a lot of time and the string doesn't have capacity to store all the json. How i can store it? Is any way to do this conversion without use the datatype string?
UPDATE:
The json format is:
[{
'key':[some things],
'data': [some things],
'data1':[A LOT OF ENTRIES],
'data2':[A LOT OF ENTRIES],
'data3':[some things],
'data4':[some things]
}]

Out-of-memory exceptions in .Net can be caused by several problems including:
Allocating too much total memory.
If this might be happening, check whether you are running in 64-bit mode as described here. If not, rebuild in 64-bit mode as described here and re-test.
Allocating too many objects on the large object heap causing memory fragmentation.
Allocating a single object that is larger than the .Net object size limit.
Failing to dispose of unmanaged memory (not applicable here).
In your case, you may be trying to allocate too much total memory but are definitely allocating three very large objects: the in-memory temp JSON string, the in-memory xmlString XML string and the in-memory stringWriter.
You can substantially reduce your memory footprint and completely eliminate these objects by constructing an XDocument or XmlDocument directly via a streaming translation from the JSON file. Then afterward, write the document directly to the XML file using XDocument.Save() or XmlDocument.Save().
To do this, you will need to allocate your own XmlNodeConverter, then construct a JsonSerializer using it and deserialize as shown in Deserialize JSON from a file. The following method(s) do the trick:
public static partial class JsonExtensions
{
public static XDocument LoadXNode(string pathJson, string deserializeRootElementName)
{
using (var stream = File.OpenRead(pathJson))
return LoadXNode(stream, deserializeRootElementName);
}
public static XDocument LoadXNode(Stream stream, string deserializeRootElementName)
{
// Let caller dispose the underlying streams.
using (var textReader = new StreamReader(stream, Encoding.UTF8, true, 1024, true))
return LoadXNode(textReader, deserializeRootElementName);
}
public static XDocument LoadXNode(TextReader textReader, string deserializeRootElementName)
{
var settings = new JsonSerializerSettings
{
Converters = { new XmlNodeConverter { DeserializeRootElementName = deserializeRootElementName } },
};
using (var jsonReader = new JsonTextReader(textReader) { CloseInput = false })
return JsonSerializer.CreateDefault(settings).Deserialize<XDocument>(jsonReader);
}
public static void StreamJsonToXml(string pathJson, string pathXml, string deserializeRootElementName, SaveOptions saveOptions = SaveOptions.None)
{
var doc = LoadXNode(pathJson, deserializeRootElementName);
doc.Save(pathXml, saveOptions);
}
}
Then use them as follows:
JsonExtensions.StreamJsonToXml(pathJson, outputPath, "json");
Here I am using XDocument instead of XmlDocument because I believe (but have not checked personally) that it uses less memory, e.g. as reported in Some hard numbers about XmlDocument, XDocument and XmlReader (x86 versus x64) by Ken Lassesen.
This approach eliminates the three large objects mentioned previously and substantially reduces the chance of running out of memory due to problems #2 or #3.
Demo fiddle here.
If you are still running out of memory even after ensuring you are running in 64-bit mode and streaming directly from and to your file(s) using the methods above, then it may simply be that your XML is too large to fit in your computer's virtual memory space using XDocument or XmlDocument. If that is so, you will need to adopt a pure streaming solution that transforms from JSON to XML on the fly as it streams. Unfortunately, Json.NET does not provide this functionality out of the box, so you will need a more complex solution.
So, what are your options?
You could fork your own version of XmlNodeConverter.cs and rewrite ReadElement(JsonReader reader, IXmlDocument document, IXmlNode currentNode, string propertyName, XmlNamespaceManager manager) to write directly to an XmlWriter instead of an IXmlDocument.
While probably doable with a couple days effort, the difficulty would seem to exceed that of a single stackoverflow answer.
You could use the reader returned by JsonReaderWriterFactory to translate JSON to XML on the fly, and pass that reader directly to XmlWriter.WriteNode(XmlReader). The readers and writers returned by this factory are used internally by DataContractJsonSerializer but can be used directly as well.
If your JSON has a fixed schema (which is unclear from your question) you have many more straightforward options. Incrementally deserializing to some c# data model as shown in Parsing large json file in .NET and re-serializing that model to XML is likely to use much less memory than loading into some generic DOM such as XDocument.
Option #2 can be implemented very simply, as follows:
using (var stream = File.OpenRead(pathJson))
using (var jsonReader = JsonReaderWriterFactory.CreateJsonReader(stream, XmlDictionaryReaderQuotas.Max))
{
using (var xmlWriter = XmlWriter.Create(outputPath))
{
xmlWriter.WriteNode(jsonReader, true);
}
}
However, the XML thereby produced is much less pretty than the XML generated by XmlNodeConverter. For instance, given the simple input JSON
{"Root":[{
"key":["a"],
"data": [1, 2]
}]}
XmlNodeConverter will create the following XML:
<json>
<Root>
<key>a</key>
<data>1</data>
<data>2</data>
</Root>
</json>
While JsonReaderWriterFactory will create the following (indented for clarity):
<root type="object">
<Root type="array">
<item type="object">
<key type="array">
<item type="string">a</item>
</key>
<data type="array">
<item type="number">1</item>
<item type="number">2</item>
</data>
</item>
</Root>
</root>
The exact format of the XML generated can be found in
Mapping Between JSON and XML.
Still, once you have valid XML, there are streaming XML-to-XML transformation solutions that will allow you to transform the generated XML to your final, desired format, including:
C# XSLT Transforming Large XML Files Quickly.
How to: Perform Streaming Transform of Large XML Documents (C#).
Combining the XmlReader and XmlWriter classes for simple streaming transformations.
Is it possible to do the other way?
Unfortunately
JsonReaderWriterFactory.CreateJsonWriter().WriteNode(xmlReader, true);
isn't really suited for conversion of arbitrary XML to JSON as it only allows for conversion of XML with the precise schema specified by Mapping Between JSON and XML.
Furthermore, when converting from arbitrary XML to JSON the problem of array recognition exists: JSON has arrays, XML doesn't, it only has repeating elements. To recognize repeating elements (or tuples of elements where identically named elements may not be adjacent) and convert them to JSON array(s) requires buffering either the XML input or the JSON output (or a complex two-pass algorithm). Mapping Between JSON and XML avoids the problem by requiring type="object" or type="array" attributes.

Deserializing XML with HTML tags

I am trying to deserialize an XML file, and it works fine except for nodes that contain HTML tags. Here is a snippet from the XML file:
<article mdate="2011-12-29" key="tr/trier/MI99-02" publtype="informal publication">
<author>Friedemann Leibfritz</author>
<title>A LMI-Based Algorithm for Designing Suboptimal Static H<sub>2</sub>/H<sub>infinity</sub> Output Feedback Controllers</title>
<journal>Universität Trier, Mathematik/Informatik, Forschungsbericht</journal>
<volume>99-02</volume>
<year>1999</year>
</article>
Then, I am getting the error:
{"Unexpected node type Element. ReadElementString method can only be called on elements with simple or empty content. Line 1148, position 64."}
The error occurs at:
A LMI-Based Algorithm for Designing Suboptimal Static H2/Hinfinity Output Feedback Controllers
where HTML tags sub and /sub exist.
Is there a way to deserialize the title node as a whole, ignoring the HTML tags? Below is a portion of my code:
XmlReaderSettings readerSettings = new XmlReaderSettings
{
DtdProcessing = DtdProcessing.Parse,
XmlResolver = new LocalXhtmlXmlResolver()
};
XmlRootAttribute xRoot = new XmlRootAttribute();
xRoot.ElementName = "dblp";
xRoot.IsNullable = true;
XmlSerializer deserializer;
XmlReader textReader;
deserializer = new XmlSerializer(typeof(List<Entity.Article>), xRoot);
textReader = XmlReader.Create(xmlPath, readerSettings);
List<Entity.Article> articleList;
articleList = (List<Entity.Article>)deserializer.Deserialize(textReader);
textReader.Close();
Any help would be very much appreciated - Thanks!

Your XML is not properly escaped. The parser has no way of knowing that those tags aren't meant to be part of the XML document, and when they are treated as such, your XML is invalid, because an element is nested within the value of another element.
That XML snippet, correctly escaped would be
<article mdate="2011-12-29" key="tr/trier/MI99-02" publtype="informal publication">
<author>Friedemann Leibfritz</author>
<title>A LMI-Based Algorithm for Designing Suboptimal Static H<sub>2</sub>/H<sub>infinity</sub> Output Feedback Controllers</title>
<journal>Universität Trier, Mathematik/Informatik, Forschungsbericht</journal>
<volume>99-02</volume>
<year>1999</year>
</article>

As the comments to the previous answer point out- as developers we don't always have the luxury of formatting the XML prior to deserialization. There is a much more elegant solution, in my opinion, that satisfies the original question.
Serializer
public static T ParseXml<T>(this string #this) where T : class
{
var serializer = new XmlSerializer(typeof(T));
serializer.UnknownElement += Serializer_UnknownElement;
return serializer.Deserialize(new StringReader(#this)) as T;
}
Handle Problematic Fields
private static void Serializer_UnknownElement(object sender, XmlElementEventArgs e)
{
if (e.ObjectBeingDeserialized is Article article)
{
if (e.Element.Name == "title")
{
article.Title_Custom = e.Element.InnerXml;
return;
}
}
}
Modifications to Article Class
public class Article{
// include your other fields that are not problematic
public string Title_Custom { get; set; }
}
Usage
var myArticles = articlesXmlString.Parse<List<Article>>();
Console.Out(myArticles[0].Title_Custom); // "A LMI-Based Algorithm for Designing Suboptimal Static H<sub>2</sub>/H<sub>infinity</sub> Output Feedback Controllers"
Because the name of the property is now Title_Custom it will naturally be skipped as part of the deserialization process. The Serializer_UnknownElement method will then read in the <title> field as an unknown field. Then, you simply pull the entire contents of the inner XML.
The inclusion of <sup> will trip the Serializer_UnknownElement as well, but as you have no condition for it, it will skip over it.
The net result is that Title_Custom will now contain the full HTML snippet as expected.

xml serialisation best practices

I have been using the traditional way of serializing content with the following code
private void SaveToXml(IdentifiableEntity IE)
{
try
{
XmlSerializer serializer = new XmlSerializer(IE.GetType());
TextWriter textWriter = new StreamWriter(IE.FilePath);
serializer.Serialize(textWriter, IE);
textWriter.Close();
}
catch (Exception e )
{
Console.WriteLine("erreur : "+ e);
}
}
private T LoadFromXml<T>(string path)
{
XmlSerializer deserializer = new XmlSerializer(typeof(T));
TextReader textReader = new StreamReader(path);
T entity = (T)deserializer.Deserialize(textReader);
textReader.Close();
return entity;
}
Though this approach does the trick, i find it a bit annoying that all my properties have to be public, that i need to tag the properties sometimes [XmlAttribute|XmlElement| XmlIgnore] and that it doesn't deal with dictionaries.
My question is : Is there a better way of serializing objects in c#, a way that with less hassle, more modern and easy to use?

First of all, I would suggest to use "using" blocks in your code.(Sample code)
If my understanding is OK, you are looking for a fast way to build your model classes that you will use during your deserialize/serialize operations.
Every Xml file is different and I don't know any generic way to serialize / deserialize them. At one moment you have to know if there will be an attribute, or elements or if any element can be null etc.
Assuming that you already have a sample XML file with a few lines which gives you general view of how it will look like
I would suggest to use xsd (miracle tool)
xsd yourXMLFileName.xml
xsd yourXMLFileName.xsd \classes
This tool will generate you every time model classes for the XML file you want to work it.
Than you serialize and deserialize easily
To deserialize (assuming that you'll get a class named XXXX representing root node in your xml)
XmlSerializer ser = new XmlSerializer(typeof(XXXX));
XXXX yourVariable;
using (XmlReader reader = XmlReader.Create(#"C:\yyyyyy\yyyyyy\YourXmlFile.xml"))
{
yourVariable= (XXXX) ser.Deserialize(reader);
}
To serialize
var serializer = new XmlSerializer(typeof(XXXX));
using(var writer = new StreamWriter(#"C:\yyyyyy\yyyyyy\YourXmlFile.xml"))
{
serializer.Serialize(writer, yourVariable);
}

How to remove xmlns:xsi and xmlns:xsd from Web API XML serializer

For some hours, I have battled with removing the default namespaces from the XML returned from serializing my independent objects (not MVC model) in ASP.NET Web Api. Sample codes for the application is:
Class Definition:
public class PaymentNotificationResponse
{
[XmlArray("Payments")]
[XmlArrayItem("Payment", typeof(PaymentResponse))]
public PaymentResponse[] Payments { get; set; }
}
I then created a Web Api Controler that creates an object of the PaymentNotificationResponse based on some input, and then serialize the object to the requesting party. The Controller is listed below:
public class PaymentController : ApiController
{
public PaymentNotificationResponse Post()
{
//Read request content (only available async), run the task to completion and pick the stream.
var strmTask = Request.Content.ReadAsStreamAsync();
while (strmTask.Status != TaskStatus.RanToCompletion) { }
Stream strm = strmTask.Result;
//Go back to the beginning of the stream, so that the data can be retrieved. Web Api reads it to the end.
if (strm.CanSeek)
strm.Seek(0, SeekOrigin.Begin);
//Read stream content and convert to string.
byte[] arr = new byte[strm.Length];
strm.Read(arr, 0, arr.Length);
String str = Encoding.UTF8.GetString(arr);
//Change the default serializer to XmlSerializer from DataContractSerializer, so that I don't get funny namespaces in properties.
//Then set a new XmlSerializer for the object
Configuration.Formatters.XmlFormatter.UseXmlSerializer = true;
Configuration.Formatters.XmlFormatter.SetSerializer<PaymentNotificationResponse>(new XmlSerializer(typeof(PaymentNotificationResponse)));
//Now call a function that would convert the string to the required object, which would then be serialized when the Web Api is invoked.
return CreatePaymentNotificationFromString(str);
}
}
Problem is, when I invoke the Api with valid string parameter, it returns an XML of this format (valid XML, but xmlns is not needed):
<PaymentNotificationResponse xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Payments>
<Payment>
<PaymentLogId>8325</PaymentLogId>
<Status>0</Status>
</Payment>
</Payments>
</PaymentNotificationResponse>
The system I'm sending this to doesn't need the xmlns:xsi and the xmlns:xsd. In fact, it returns an exception when it sees the namespaces.
I tried returning a string with XML tags, it just wrapped the response in a <string></string> and encoded all the < and >. So that was not an option.
I saw this post and this one. While the former is very detailed, it didn't solve my problem. It only just introduced one extra xmlns to the generated XML. I think it's because I didn't explicitly call .Serialize() function in the XmlSerializer.
I figured out a solution, and I thought I should share. So I would state it in the answers.

To fix this, I added a method to serialize the PaymentNotificationResponseand return an XML string, thus (I included this in the definition of PaymentNotificationResponse):
//I added this method after I tried serialize directly to no avail
public String SerializeToXml()
{
MemoryStream ms = new MemoryStream();
XmlSerializerNamespaces ns = new XmlSerializerNamespaces();
ns.Add("", "");
new XmlSerializer(typeof(PaymentNotificationResponse)).Serialize(ms, this, ns);
XmlTextWriter textWriter = new XmlTextWriter(ms, Encoding.UTF8);
ms = (System.IO.MemoryStream)textWriter.BaseStream;
return new UTF8Encoding().GetString(ms.ToArray());
}
I parsed the string to create an XDocument, then return the root element. This made me change the return type of the Post() method to XElement. The Post() listing would then be:
public XElement Post()
{
//...Same as it was in the question, just the last line that changed.
var pnr = CreatePaymentNotificationFromString(str);
return XDocument.Parse(pnr.SerializeToXml()).Root;
}
That would make my response XML this:
<PaymentNotificationResponse>
<Payments>
<Payment>
<PaymentLogId>8325</PaymentLogId>
<Status>0</Status>
</Payment>
</Payments>
</PaymentNotificationResponse>
I hope this helps someone.