I'm trying to convert from a huge JSON file(2GB) to xml file. I have some troubles reading the huge JSON file.
I've been researching about how i can read huge JSON files.
I found this:
Out of memory exception while loading large json file from disk
How to parse huge JSON file as stream in Json.NET?
Parsing large json file in .NET
It seems that i'm duplicating my question but i have some troubles which aren't solved in these posts.
So, i need to load the huge JSON File and the community propose something like this:
MyObject o;
using (StreamReader sr = new StreamReader("foo.json"))
using (JsonTextReader reader = new JsonTextReader(sr))
{
var serializer = new JsonSerializer();
reader.SupportMultipleContent = true;
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject)
{
// Deserialize each object from the stream individually and process it
var o = serializer.Deserialize<MyObject>(reader);
//Do something with the object
}
}
}
So, We can read by parts and deserialize objects one by one.
I'll show you my code
JsonSerializer serializer = new JsonSerializer();
string hugeJson = "hugJSON.json";
using (FileStream s = File.Open(hugeJson , FileMode.Open))
{
using (StreamReader sr = new StreamReader(s))
{
using (JsonReader reader = new JsonTextReader(sr))
{
reader.SupportMultipleContent = true;
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject)
{
var jsonObject = serializer.Deserialize(reader);
string xmlString = "";
XmlDocument doc = JsonConvert.DeserializeXmlNode(jsonObject.ToString(), "json");
using (var stringWriter = new StringWriter())
{
using (var xmlTextWriter = XmlWriter.Create(stringWriter))
{
doc.WriteTo(xmlTextWriter);
xmlTextWriter.Flush();
xmlString = stringWriter.GetStringBuilder().ToString();
}
}
}
}
}
}
}
But when i try doc.WriteTo(xmlTextWriter), i get Exception of type System.OutOfMemoryException was thrown.
I've been trying with BufferedStream. This class allows me manage big files but i have another problem.
I'm reading in byte[] format. When i convert to string, the json is splitted and i can't parse to xml file because there are missing characters
for example:
{ foo:[{
foo:something,
foo1:something,
foo2:something
},
{
foo:something,
foo:som
it is cutted.
Is any way to read a huge JSON and convert to XML without load the JSON by parts? or i could load a convert by parts but i don't know how to do this.
Any ideas?
UPDATE:
I have been trying with this code:
static void Main(string[] args)
{
string json = "";
string pathJson = "foo.json";
//Read file
string temp = "";
using (FileStream fs = new FileStream(pathJson, FileMode.Open))
{
using (BufferedStream bf = new BufferedStream(fs))
{
byte[] array = new byte[70000];
while (bf.Read(array, 0, 70000) != 0)
{
json = Encoding.UTF8.GetString(array);
temp = String.Concat(temp, json);
}
}
}
XmlDocument doc = new XmlDocument();
doc = JsonConvert.DeserializeXmlNode(temp, "json");
using (var stringWriter = new StringWriter())
using (var xmlTextWriter = XmlWriter.Create(stringWriter))
{
doc.WriteTo(xmlTextWriter);
xmlTextWriter.Flush();
xmlString = stringWriter.GetStringBuilder().ToString();
}
File.WriteAllText("outputPath", xmlString);
}
This code convert from json file to xml file. but when i try to convert a big json file (2GB), i can't. The process cost a lot of time and the string doesn't have capacity to store all the json. How i can store it? Is any way to do this conversion without use the datatype string?
UPDATE:
The json format is:
[{
'key':[some things],
'data': [some things],
'data1':[A LOT OF ENTRIES],
'data2':[A LOT OF ENTRIES],
'data3':[some things],
'data4':[some things]
}]
Out-of-memory exceptions in .Net can be caused by several problems including:
Allocating too much total memory.
If this might be happening, check whether you are running in 64-bit mode as described here. If not, rebuild in 64-bit mode as described here and re-test.
Allocating too many objects on the large object heap causing memory fragmentation.
Allocating a single object that is larger than the .Net object size limit.
Failing to dispose of unmanaged memory (not applicable here).
In your case, you may be trying to allocate too much total memory but are definitely allocating three very large objects: the in-memory temp JSON string, the in-memory xmlString XML string and the in-memory stringWriter.
You can substantially reduce your memory footprint and completely eliminate these objects by constructing an XDocument or XmlDocument directly via a streaming translation from the JSON file. Then afterward, write the document directly to the XML file using XDocument.Save() or XmlDocument.Save().
To do this, you will need to allocate your own XmlNodeConverter, then construct a JsonSerializer using it and deserialize as shown in Deserialize JSON from a file. The following method(s) do the trick:
public static partial class JsonExtensions
{
public static XDocument LoadXNode(string pathJson, string deserializeRootElementName)
{
using (var stream = File.OpenRead(pathJson))
return LoadXNode(stream, deserializeRootElementName);
}
public static XDocument LoadXNode(Stream stream, string deserializeRootElementName)
{
// Let caller dispose the underlying streams.
using (var textReader = new StreamReader(stream, Encoding.UTF8, true, 1024, true))
return LoadXNode(textReader, deserializeRootElementName);
}
public static XDocument LoadXNode(TextReader textReader, string deserializeRootElementName)
{
var settings = new JsonSerializerSettings
{
Converters = { new XmlNodeConverter { DeserializeRootElementName = deserializeRootElementName } },
};
using (var jsonReader = new JsonTextReader(textReader) { CloseInput = false })
return JsonSerializer.CreateDefault(settings).Deserialize<XDocument>(jsonReader);
}
public static void StreamJsonToXml(string pathJson, string pathXml, string deserializeRootElementName, SaveOptions saveOptions = SaveOptions.None)
{
var doc = LoadXNode(pathJson, deserializeRootElementName);
doc.Save(pathXml, saveOptions);
}
}
Then use them as follows:
JsonExtensions.StreamJsonToXml(pathJson, outputPath, "json");
Here I am using XDocument instead of XmlDocument because I believe (but have not checked personally) that it uses less memory, e.g. as reported in Some hard numbers about XmlDocument, XDocument and XmlReader (x86 versus x64) by Ken Lassesen.
This approach eliminates the three large objects mentioned previously and substantially reduces the chance of running out of memory due to problems #2 or #3.
Demo fiddle here.
If you are still running out of memory even after ensuring you are running in 64-bit mode and streaming directly from and to your file(s) using the methods above, then it may simply be that your XML is too large to fit in your computer's virtual memory space using XDocument or XmlDocument. If that is so, you will need to adopt a pure streaming solution that transforms from JSON to XML on the fly as it streams. Unfortunately, Json.NET does not provide this functionality out of the box, so you will need a more complex solution.
So, what are your options?
You could fork your own version of XmlNodeConverter.cs and rewrite ReadElement(JsonReader reader, IXmlDocument document, IXmlNode currentNode, string propertyName, XmlNamespaceManager manager) to write directly to an XmlWriter instead of an IXmlDocument.
While probably doable with a couple days effort, the difficulty would seem to exceed that of a single stackoverflow answer.
You could use the reader returned by JsonReaderWriterFactory to translate JSON to XML on the fly, and pass that reader directly to XmlWriter.WriteNode(XmlReader). The readers and writers returned by this factory are used internally by DataContractJsonSerializer but can be used directly as well.
If your JSON has a fixed schema (which is unclear from your question) you have many more straightforward options. Incrementally deserializing to some c# data model as shown in Parsing large json file in .NET and re-serializing that model to XML is likely to use much less memory than loading into some generic DOM such as XDocument.
Option #2 can be implemented very simply, as follows:
using (var stream = File.OpenRead(pathJson))
using (var jsonReader = JsonReaderWriterFactory.CreateJsonReader(stream, XmlDictionaryReaderQuotas.Max))
{
using (var xmlWriter = XmlWriter.Create(outputPath))
{
xmlWriter.WriteNode(jsonReader, true);
}
}
However, the XML thereby produced is much less pretty than the XML generated by XmlNodeConverter. For instance, given the simple input JSON
{"Root":[{
"key":["a"],
"data": [1, 2]
}]}
XmlNodeConverter will create the following XML:
<json>
<Root>
<key>a</key>
<data>1</data>
<data>2</data>
</Root>
</json>
While JsonReaderWriterFactory will create the following (indented for clarity):
<root type="object">
<Root type="array">
<item type="object">
<key type="array">
<item type="string">a</item>
</key>
<data type="array">
<item type="number">1</item>
<item type="number">2</item>
</data>
</item>
</Root>
</root>
The exact format of the XML generated can be found in
Mapping Between JSON and XML.
Still, once you have valid XML, there are streaming XML-to-XML transformation solutions that will allow you to transform the generated XML to your final, desired format, including:
C# XSLT Transforming Large XML Files Quickly.
How to: Perform Streaming Transform of Large XML Documents (C#).
Combining the XmlReader and XmlWriter classes for simple streaming transformations.
Is it possible to do the other way?
Unfortunately
JsonReaderWriterFactory.CreateJsonWriter().WriteNode(xmlReader, true);
isn't really suited for conversion of arbitrary XML to JSON as it only allows for conversion of XML with the precise schema specified by Mapping Between JSON and XML.
Furthermore, when converting from arbitrary XML to JSON the problem of array recognition exists: JSON has arrays, XML doesn't, it only has repeating elements. To recognize repeating elements (or tuples of elements where identically named elements may not be adjacent) and convert them to JSON array(s) requires buffering either the XML input or the JSON output (or a complex two-pass algorithm). Mapping Between JSON and XML avoids the problem by requiring type="object" or type="array" attributes.
Related
I need to deserialize an XML response from an external service containing more than 100.000 rows, but I have a problem unescapeing numeric character references in various places. Since the DOM is complex, I need to have a global solution that applies to the whole document, and not it's specific elements. I have the following situations:
<text>First & Second</text>
I use the following XmlSerializer implementation:
public T DeserializeXmlReader<T>(string path) where T : class
{
XmlSerializer serializer = new XmlSerializer(typeof(T));
using (FileStream fileStream = new FileStream(path, FileMode.Open))
{
using (XmlReader xmlReader = XmlReader.Create(fileStream))
{
return (T)serializer.Deserialize(xmlReader);
}
}
}
After deserialization, I will get the following result: "First & Second", instead of "First & Second". I am not sure if there is an additional step I need to undertake to get the "&" deserialized correctly?
Note: After doing some research, I believe the problem might be similar to this one, but I'm not sure if it's applicable since this concerns php: How to deserialize a xml string along with NCR unescaping?
I'm currently serializing HighScoreData from a C# application to an XML file using the XmlSerializer namespace. This is producing an inconsistent result regarding the outputted XML file.
The object I'm serializing is the following struct:
namespace GameProjectTomNijs.GameComponents
{
[Serializable]
public struct HighScoreData
{
public string[] PlayerName;
public int[] Score;
public int Count;
public readonly string HighScoresFilename;
public HighScoreData(int count)
{
PlayerName = new string[count];
Score = new int[count];
Count = count;
HighScoresFilename = "highscores.lst";
}
}
}
Questionable variable accessibility levels aside, it contains an array of string, an array of integers and an integer containing the total objects. This is the data that is being serialized. The output of this usually is :
<?xml version="1.0"?>
<HighScoreData xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<PlayerName>
<string />
<string>MISTERT</string>
<string>TESTER</string>
<string>PAULA</string>
<string>JENS</string>
</PlayerName>
<Score>
<int>554</int>
<int>362</int>
<int>332</int>
<int>324</int>
<int>218</int>
</Score>
<Count>5</Count>
</HighScoreData>
However, about 20-30% of the time it is writing to the XML file in a peculiar manner, the ending root tag would look as follows: </HighScoreData>oreData>
It seems the method writing to the XML file is appending the values rather than overwriting I guess?
The following code is the method actually writing to the XML file:
public static void SaveHighScores(HighScoreData data, string fullpath)
{
// Open the file, creating it if necessary
FileStream stream = File.Open(fullpath, FileMode.OpenOrCreate);
try
{
// Convert the object to XML data and put it in the stream
XmlSerializer serializer = new XmlSerializer(typeof(HighScoreData));
serializer.Serialize(stream, data);
}
finally
{
// Close the file
stream.Close();
}
}`
Is there anything I'm currently overlooking? I've used this method in a large number of projects to great success.
Any help would be greatly appreciated !
This is the problem:
FileStream stream = File.Open(fullpath, FileMode.OpenOrCreate);
That doesn't truncate the file if it already exists - it just overwrites the data that it writes, but if the original file is longer than the data written by the serializer, the "old" data is left at the end.
Just use
using (var stream = File.Create(fullPath))
{
...
}
(Then you don't need try/finally either - always use a using statement for resources...)
I have been using the traditional way of serializing content with the following code
private void SaveToXml(IdentifiableEntity IE)
{
try
{
XmlSerializer serializer = new XmlSerializer(IE.GetType());
TextWriter textWriter = new StreamWriter(IE.FilePath);
serializer.Serialize(textWriter, IE);
textWriter.Close();
}
catch (Exception e )
{
Console.WriteLine("erreur : "+ e);
}
}
private T LoadFromXml<T>(string path)
{
XmlSerializer deserializer = new XmlSerializer(typeof(T));
TextReader textReader = new StreamReader(path);
T entity = (T)deserializer.Deserialize(textReader);
textReader.Close();
return entity;
}
Though this approach does the trick, i find it a bit annoying that all my properties have to be public, that i need to tag the properties sometimes [XmlAttribute|XmlElement| XmlIgnore] and that it doesn't deal with dictionaries.
My question is : Is there a better way of serializing objects in c#, a way that with less hassle, more modern and easy to use?
First of all, I would suggest to use "using" blocks in your code.(Sample code)
If my understanding is OK, you are looking for a fast way to build your model classes that you will use during your deserialize/serialize operations.
Every Xml file is different and I don't know any generic way to serialize / deserialize them. At one moment you have to know if there will be an attribute, or elements or if any element can be null etc.
Assuming that you already have a sample XML file with a few lines which gives you general view of how it will look like
I would suggest to use xsd (miracle tool)
xsd yourXMLFileName.xml
xsd yourXMLFileName.xsd \classes
This tool will generate you every time model classes for the XML file you want to work it.
Than you serialize and deserialize easily
To deserialize (assuming that you'll get a class named XXXX representing root node in your xml)
XmlSerializer ser = new XmlSerializer(typeof(XXXX));
XXXX yourVariable;
using (XmlReader reader = XmlReader.Create(#"C:\yyyyyy\yyyyyy\YourXmlFile.xml"))
{
yourVariable= (XXXX) ser.Deserialize(reader);
}
To serialize
var serializer = new XmlSerializer(typeof(XXXX));
using(var writer = new StreamWriter(#"C:\yyyyyy\yyyyyy\YourXmlFile.xml"))
{
serializer.Serialize(writer, yourVariable);
}
For some hours, I have battled with removing the default namespaces from the XML returned from serializing my independent objects (not MVC model) in ASP.NET Web Api. Sample codes for the application is:
Class Definition:
public class PaymentNotificationResponse
{
[XmlArray("Payments")]
[XmlArrayItem("Payment", typeof(PaymentResponse))]
public PaymentResponse[] Payments { get; set; }
}
I then created a Web Api Controler that creates an object of the PaymentNotificationResponse based on some input, and then serialize the object to the requesting party. The Controller is listed below:
public class PaymentController : ApiController
{
public PaymentNotificationResponse Post()
{
//Read request content (only available async), run the task to completion and pick the stream.
var strmTask = Request.Content.ReadAsStreamAsync();
while (strmTask.Status != TaskStatus.RanToCompletion) { }
Stream strm = strmTask.Result;
//Go back to the beginning of the stream, so that the data can be retrieved. Web Api reads it to the end.
if (strm.CanSeek)
strm.Seek(0, SeekOrigin.Begin);
//Read stream content and convert to string.
byte[] arr = new byte[strm.Length];
strm.Read(arr, 0, arr.Length);
String str = Encoding.UTF8.GetString(arr);
//Change the default serializer to XmlSerializer from DataContractSerializer, so that I don't get funny namespaces in properties.
//Then set a new XmlSerializer for the object
Configuration.Formatters.XmlFormatter.UseXmlSerializer = true;
Configuration.Formatters.XmlFormatter.SetSerializer<PaymentNotificationResponse>(new XmlSerializer(typeof(PaymentNotificationResponse)));
//Now call a function that would convert the string to the required object, which would then be serialized when the Web Api is invoked.
return CreatePaymentNotificationFromString(str);
}
}
Problem is, when I invoke the Api with valid string parameter, it returns an XML of this format (valid XML, but xmlns is not needed):
<PaymentNotificationResponse xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Payments>
<Payment>
<PaymentLogId>8325</PaymentLogId>
<Status>0</Status>
</Payment>
</Payments>
</PaymentNotificationResponse>
The system I'm sending this to doesn't need the xmlns:xsi and the xmlns:xsd. In fact, it returns an exception when it sees the namespaces.
I tried returning a string with XML tags, it just wrapped the response in a <string></string> and encoded all the < and >. So that was not an option.
I saw this post and this one. While the former is very detailed, it didn't solve my problem. It only just introduced one extra xmlns to the generated XML. I think it's because I didn't explicitly call .Serialize() function in the XmlSerializer.
I figured out a solution, and I thought I should share. So I would state it in the answers.
To fix this, I added a method to serialize the PaymentNotificationResponseand return an XML string, thus (I included this in the definition of PaymentNotificationResponse):
//I added this method after I tried serialize directly to no avail
public String SerializeToXml()
{
MemoryStream ms = new MemoryStream();
XmlSerializerNamespaces ns = new XmlSerializerNamespaces();
ns.Add("", "");
new XmlSerializer(typeof(PaymentNotificationResponse)).Serialize(ms, this, ns);
XmlTextWriter textWriter = new XmlTextWriter(ms, Encoding.UTF8);
ms = (System.IO.MemoryStream)textWriter.BaseStream;
return new UTF8Encoding().GetString(ms.ToArray());
}
I parsed the string to create an XDocument, then return the root element. This made me change the return type of the Post() method to XElement. The Post() listing would then be:
public XElement Post()
{
//...Same as it was in the question, just the last line that changed.
var pnr = CreatePaymentNotificationFromString(str);
return XDocument.Parse(pnr.SerializeToXml()).Root;
}
That would make my response XML this:
<PaymentNotificationResponse>
<Payments>
<Payment>
<PaymentLogId>8325</PaymentLogId>
<Status>0</Status>
</Payment>
</Payments>
</PaymentNotificationResponse>
I hope this helps someone.
I'm reading a file like from the web:
<?xml version='1.0' encoding='UTF-8'?>
<eveapi version="2">
<currentTime>2011-07-30 16:08:53</currentTime>
<result>
<rowset name="characters" key="characterID" columns="name,characterID,corporationName,corporationID">
<row name="Conqrad Echerie" characterID="91048359" corporationName="Federal Navy Academy" corporationID="1000168" />
</rowset>
</result>
<cachedUntil>2011-07-30 17:05:48</cachedUntil>
</eveapi>
im still new to XML and i see there are many ways to read XML data, is there a certain way im going to want to do this? what i want to do is load all the data into a StreamReader? and then use get; set; to pull the data later?
If you want object-based access, put the example xml in a file and run
xsd.exe my.xml
xsd.exe my.xsd /classes
this will create my.cs which is an object model similar to the xml that you can use with XmlSerializer:
var ser = new XmlSerializer(typeof(eveapi));
var obj = (eveapi)ser.Deserialize(source);
Use XmlReader Class or XmlTextReader Class
http://msdn.microsoft.com/en-us/library/aa720470(v=vs.71).aspx
http://msdn.microsoft.com/en-us/library/system.xml.xmltextreader(v=vs.71).aspx
If you need to use the data in the easy way, especially when you're new to XML, use XmlDocument.
To load the document:
using System.Xml;
using System.IO;
public class someclass {
void somemethod () {
//Initiate the XmlDocument object
XmlDocument xdoc;
//To load from file
xdoc.Load("SomeFolder\\SomeFile.xml");
//Or to load from XmlTextReader, from a file for example
FileStream fs = FileStream("SomeFolder\\SomeFile.xml", FileMode.Open, FileAccess.Read);
XmlTextReader reader = new XmlTextReader(fs);
xdoc.Load(reader);
//In fact, you can load the stream directly
xdoc.Load(fs);
//Or, you can load from a string
xdoc.LoadXml(#"<rootElement>
<element1>value1</element1>
<element2>value2</element2>
</rootElement>");
}
}
I personally find XmlDocument far easier to use for navigating an Xml file.
To use it efficiently, you need to learn XPath. For example, to get the name of the first row:
string name = xdoc.SelectSingleNode("/eveapi/result/rowset/row").Attribute["name"].InnerText;
or even more XPath:
string name = xdoc.SelectSingleNode("/eveapi/result/rowset/row/#name").InnerText;
you can even filter:
XmlNodeList elems = xdoc.SelectNodes("//*[#name=\"characters\"]")
gives you the rowset element.
But that's off topic.