Develop Reference

Deserializing JSON with numbers as field using JsonSerializer

I need to deserialize this weird JSON (image below). I've seen some deserialization hints using Dictionary<>, etc. but the problem is that "parameters" contains different data, then previous keys.
Can I somehow get it to work using JsonSerializer deserializator without doing foreach loops and other suspicious implementations? I do need data from "data" in my application.
Here's some of my code:
using var client = new WebClient();
var json = client.DownloadString(GetJsonString());
var invoicesData = JsonSerializer.Deserialize<JsonMyData>(json, options);
If using Newtonsoft is necessary I might start using it.

With Newtonsoft you can parse and access arbitrary JSON documents, even ones that can't reasonably be deserialized into a .NET object. So something like:
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System;
namespace ConsoleApp35
{
class Program
{
static void Main(string[] args)
{
var json = #"
{
""myData"" :
{
""0"" : { ""data"": { ""A"":1,""B"":2} },
""1"" : { ""data"": { ""A"":1,""B"":2} },
""2"" : { ""data"": { ""A"":1,""B"":2} },
""3"" : { ""data"": { ""A"":1,""B"":2} },
""parameters"" : { ""p"":""a""}
},
""status"":{ }
}";
var foo = JObject.Parse(json);
var a = foo["myData"]["1"]["data"];
Console.WriteLine(a);
Console.WriteLine("Hit any key to continue");
Console.ReadKey();
}
}
}

I think you should really consider using Newtonsoft.Json instead of default JsonDeserializer, it is much easier to use in such situations.

If you are interested in processing this without foreach loops and wanting to access the data in a list format, I would suggest using Dictionary for this. When you use dictionary, you can use Objects as values that would compensate for differences in numbers (0, 1, 2, ..) and words (parameters).
// Classes to Deserialize data we need.
public class MyObject
{
[JsonProperty("data")]
public Data Data { get; set; }
}
public class Data
{
public int A { get; set; }
public int B { get; set; }
}
Usage in Main
// Read in the JSON
var myData = JsonConvert.DeserializeObject<dynamic>(jsonString)["myData"];
// Convert To Dictionary
Dictionary<string, dynamic> dataAsObjects = myData.ToObject<Dictionary<string, dynamic>>();
string searchFor = "3";
dataAsObjects.TryGetValue(searchFor, out dynamic obj);
if (obj != null)
{
// Conversion to int and matching against searchFor is to ensure its a number.
int.TryParse(searchFor, out int result);
if (result == 0 && result.ToString().Equals(searchFor))
{
MyObject myObject = obj.ToObject<MyObject>();
Console.WriteLine($"A:{myObject.Data.A} - B:{myObject.Data.B}");
}
else if (result == 8 && result.ToString().Equals(searchFor))
{
// I am not clear on whats your parameters class look like.
MyParameters myParams = obj.ToObject<MyParameters>();
}
}
Output
A:1 - B:2
With this method you can either access the numbers or the parameters element.

Failing to read XML content using XmlSerializer

I got this in my class:
namespace MSAToolsLibrary.PublisherEntry
{
[XmlRoot(ElementName = "PublisherDatabase", Namespace = "http://www.publictalksoftware.co.uk/msa")]
public class PublisherData
{
public PublisherData()
{
//_Publishers = new List<Publisher>();
_PublishersDictionary = new Dictionary<string, Publisher>();
}
public List<Publisher> Publishers
{
get { return _PublishersDictionary.Select(x => x.Value).ToList(); }
set { _PublishersDictionary = value.ToDictionary(x => x.Name, x => x); }
}
private Dictionary<string, Publisher> _PublishersDictionary;
[XmlIgnore]
public Dictionary<string, Publisher> PublisherDictionary
{
get { return _PublishersDictionary; }
}
public void AddPublisher(String strName, String strNotes, Gender eGender, Appointed eAppointedAs, Serving eServingAs, bool bUseForDemonstrations, bool bAvailableMidweek, bool bAvailableWeekend, DateTime[] listDatesNotAvailable)
{
Publisher _Publisher = new Publisher()
{
Name = strName,
Notes = strNotes,
Gender = eGender,
AppointedAs = eAppointedAs,
ServingAs = eServingAs,
};
_Publisher.Assignments.UseForDemonstrations = bUseForDemonstrations;
_Publisher.Availability.Midweek = bAvailableMidweek;
_Publisher.Availability.Weekend = bAvailableWeekend;
_Publisher.Availability.DatesNotAvailable = new List<DateTime>(listDatesNotAvailable);
//_Publishers.Add(_Publisher);
_PublishersDictionary.Add(strName, _Publisher);
}
}
}
Now, when I save my data to XML it all works good.
But when I read in:
public void ReadPublisherData(out Int64 iResult)
{
_PublisherData.Publishers.Clear(); // Reset
iResult = MakeResult(true);
try
{
XmlSerializer x = new XmlSerializer(_PublisherData.GetType());
using (StreamReader reader = new StreamReader(_strPathXML))
{
_PublisherData = (PublisherData)x.Deserialize(reader);
iResult = _PublisherData.PublisherDictionary.Count;
}
}
catch
{
iResult = MakeResult(false);
}
}
Doesn't work. I have zero publishers in the list or the dictionary.
What am I doing wrong?
Update
If I change the PublisherData declaration so that it has the needed back field:
public PublisherData()
{
_Publishers = new List<Publisher>();
_PublishersDictionary = new Dictionary<string, Publisher>();
}
public List<Publisher> Publishers
{
get => _Publishers; set => _Publishers = value;
}
private List<Publisher> _Publishers;
Then this causes the data to serialize correctly and I get what is expected in the MFC application. But now my PublisherDictionary is hanging. So I added a function:
public void BuildPublisherDictionary()
{
_PublishersDictionary = _Publishers.ToDictionary(x => x.Name, x => x);
}
And adjusted the read routine:
public void ReadPublisherData(out Int64 iResult)
{
iResult = MakeResult(true);
try
{
_PublisherData.Publishers.Clear(); // Reset
XmlSerializer x = new XmlSerializer(_PublisherData.GetType());
using (StreamReader reader = new StreamReader(_strPathXML))
{
_PublisherData = (PublisherData)x.Deserialize(reader);
_PublisherData.BuildPublisherDictionary();
iResult = _PublisherData.PublisherDictionary.Count;
}
}
catch
{
iResult = MakeResult(false);
}
}
It works. But I don't know if that is the simplest way. My problem with the above is that the list / dictionary are now detached from each other. So if I add a publisher to the dictionary it will now not be in the list.
Update 2
At the moment, after reading in the XML, if I add or remove a Publisher I am applying it to both the list and the dictionary. Unless there is a simpler way.

The serializer will only call the getter of your List and then call Add for each element it finds, providing it the instance of the type it deserialized from that element.
If your requirement is to have both a list and a dictionary you'll have to provide an implementation of an type that does so and implements an applicable interface.
The following ListDict uses a List and a Dictionary as their backing stores while implementing IList to be used in the (de)serializtion.
public class ListDict<K, T>:IList<T>
{
public Dictionary<K, T> dict = new Dictionary<K, T>();
public List<T> list = new List<T>();
Func<T,K> KeyFunc;
// takes an Function that returns a key for T
public ListDict(Func<T,K> keyfunc)
{
KeyFunc = keyfunc;
}
// called by the serializer
public void Add(T value)
{
Add( KeyFunc(value), value);
}
// fill both List and Dictionary
public void Add(K key, T value)
{
list.Add(value);
dict.Add( key , value);
}
// left out other required methods from IList<T>
}
Now your PublisherData class will change as follows to leverage above class:
[XmlRoot(ElementName = "PublisherDatabase", Namespace = "http://www.publictalksoftware.co.uk/msa")]
public class PublisherData
{
private ListDict<string, Publisher> _PublishersDictionary;
public PublisherData()
{
// provide the function to generate a key for a Publisher
_PublishersDictionary = new ListDict<string,Publisher>( (p) => p.Name );
}
[XmlElement]
public ListDict<string,Publisher> Publishers
{
get { return _PublishersDictionary; }
}
[XmlIgnore]
public Dictionary<string, Publisher> PublisherDictionary
{
get {return _PublishersDictionary.dict; }
}
}
Using above classes gives me a filled list and Dictionary directly after deserialization. You'll have to make sure of course to keep the backing stores in sync in the ListDict. Maybe you can do without it but that depends on your exact usecase.

Deserialize json array stream one item at a time

I serialize an array of large objects to a json http response stream. Now I want to deserialize these objects from the stream one at a time. Are there any c# libraries that will let me do this? I've looked at json.net but it seems I'd have to deserialize the complete array of objects at once.
[{large json object},{large json object}.....]
Clarification: I want to read one json object from the stream at a time and deserialize it.

In order to read the JSON incrementally, you'll need to use a JsonTextReader in combination with a StreamReader. But, you don't necessarily have to read all the JSON manually from the reader. You should be able to leverage the Linq-To-JSON API to load each large object from the reader so that you can work with it more easily.
For a simple example, say I had a JSON file that looked like this:
[
{
"name": "foo",
"id": 1
},
{
"name": "bar",
"id": 2
},
{
"name": "baz",
"id": 3
}
]
Code to read it incrementally from the file might look something like the following. (In your case you would replace the FileStream with your response stream.)
using (FileStream fs = new FileStream(#"C:\temp\data.json", FileMode.Open, FileAccess.Read))
using (StreamReader sr = new StreamReader(fs))
using (JsonTextReader reader = new JsonTextReader(sr))
{
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject)
{
// Load each object from the stream and do something with it
JObject obj = JObject.Load(reader);
Console.WriteLine(obj["id"] + " - " + obj["name"]);
}
}
}
Output of the above would look like this:
1 - foo
2 - bar
3 - baz

I have simplified one of the samples/tests of my parser/deserializer to answer this question's use case more straightforwardly.
Here's for the test data:
https://github.com/ysharplanguage/FastJsonParser/tree/master/JsonTest/TestData
(cf. fathers.json.txt)
And here's for the sample code:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
// Our stuff
using System.Text.Json;
//...
public class FathersData
{
public Father[] fathers { get; set; }
}
public class Someone
{
public string name { get; set; }
}
public class Father : Someone
{
public int id { get; set; }
public bool married { get; set; }
// Lists...
public List<Son> sons { get; set; }
// ... or arrays for collections, that's fine:
public Daughter[] daughters { get; set; }
}
public class Child : Someone
{
public int age { get; set; }
}
public class Son : Child
{
}
public class Daughter : Child
{
public string maidenName { get; set; }
}
//...
static void FilteredFatherStreamTestSimplified()
{
// Get our parser:
var parser = new JsonParser();
// (Note this will be invoked thanks to the "filters" dictionary below)
Func<object, object> filteredFatherStreamCallback = obj =>
{
Father father = (obj as Father);
// Output only the individual fathers that the filters decided to keep (i.e., when obj.Type equals typeof(Father)),
// but don't output (even once) the resulting array (i.e., when obj.Type equals typeof(Father[])):
if (father != null)
{
Console.WriteLine("\t\tId : {0}\t\tName : {1}", father.id, father.name);
}
// Do not project the filtered data in any specific way otherwise,
// just return it deserialized as-is:
return obj;
};
// Prepare our filter, and thus:
// 1) we want only the last five (5) fathers (array index in the resulting "Father[]" >= 29,995),
// (assuming we somehow have prior knowledge that the total count is 30,000)
// and for each of them,
// 2) we're interested in deserializing them with only their "id" and "name" properties
var filters =
new Dictionary<Type, Func<Type, object, object, int, Func<object, object>>>
{
// We don't care about anything but these 2 properties:
{
typeof(Father), // Note the type
(type, obj, key, index) =>
((key as string) == "id" || (key as string) == "name") ?
filteredFatherStreamCallback :
JsonParser.Skip
},
// We want to pick only the last 5 fathers from the source:
{
typeof(Father[]), // Note the type
(type, obj, key, index) =>
(index >= 29995) ?
filteredFatherStreamCallback :
JsonParser.Skip
}
};
// Read, parse, and deserialize fathers.json.txt in a streamed fashion,
// and using the above filters, along with the callback we've set up:
using (var reader = new System.IO.StreamReader(FATHERS_TEST_FILE_PATH))
{
FathersData data = parser.Parse<FathersData>(reader, filters);
System.Diagnostics.Debug.Assert
(
(data != null) &&
(data.fathers != null) &&
(data.fathers.Length == 5)
);
foreach (var i in Enumerable.Range(29995, 5))
System.Diagnostics.Debug.Assert
(
(data.fathers[i - 29995].id == i) &&
!String.IsNullOrEmpty(data.fathers[i - 29995].name)
);
}
Console.ReadKey();
}
The rest of the bits is available here:
https://github.com/ysharplanguage/FastJsonParser
'HTH,

This is my solution (combined from different sources, but mainly based on Brian Rogers solution) to convert huge JSON file (which is an array of objects) to XML file for any generic object.
JSON looks like this:
{
"Order": [
{ order object 1},
{ order object 2},
{...}
{ order object 10000},
]
}
Output XML:
<Order>...</Order>
<Order>...</Order>
<Order>...</Order>
C# code:
XmlWriterSettings xws = new XmlWriterSettings { OmitXmlDeclaration = true };
using (StreamWriter sw = new StreamWriter(xmlFile))
using (FileStream fs = new FileStream(jsonFile, FileMode.Open, FileAccess.Read))
using (StreamReader sr = new StreamReader(fs))
using (JsonTextReader reader = new JsonTextReader(sr))
{
//sw.Write("<root>");
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartArray)
{
while (reader.Read())
{
if (reader.TokenType == JsonToken.StartObject)
{
JObject obj = JObject.Load(reader);
XmlDocument doc = JsonConvert.DeserializeXmlNode(obj.ToString(), "Order");
sw.Write(doc.InnerXml); // a line of XML code <Order>...</Order>
sw.Write("\n");
//this approach produces not strictly valid XML document
//add root element at the beginning and at the end to make it valid XML
}
}
}
}
//sw.Write("</root>");
}

With Cinchoo ETL - an open source library, you can parse large JSON efficiently with low memory footprint. Since the objects are constructed and returned in a stream based pull model
using (var p = new ChoJSONReader(** YOUR JSON FILE **))
{
foreach (var rec in p)
{
Console.WriteLine($"Name: {rec.name}, Id: {rec.id}");
}
}
For more information, please visit codeproject article.
Hope it helps.

I know that the question is old, but it appears in google search, and I needed the same thing recently. The another way to deal with stream serilization is to use JsonSerializer.DeserializeAsyncEnumerable
Usage looks like:
await using (var readStream = File.Open(filePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
await foreach (T item in JsonSerializer.DeserializeAsyncEnumerable<T>(readStream))
{
// do something withe the item
}
}

.NET binary XML with pre-shared dictionary

I'm using XmlDictionaryWriter to serialize objects to a database with data contract serializer.
It works great, both size and speed are 2 times better then using text/xml.
However, I'll have to deal with enormous count of records in my database, where any extra bytes are directly translated into the gigabytes of the DB size.
That's why I'd love to reduce the size further, by using an XML dictionary.
How do I do that?
I see that XmlDictionaryWriter.CreateBinaryWriter static method accepts the 2-nd parameter of type IXmlDictionary. The MSDN says "The XmlDictionary to use as the shared dictionary".
First I've tried to use the system-supplied implementation:
XmlDictionary dict = new XmlDictionary();
string[] dictEntries = new string[]
{
"http://schemas.datacontract.org/2004/07/MyContracts",
"http://www.w3.org/2001/XMLSchema-instance",
"MyElementName1",
"MyElementName2",
"MyElementName3",
};
foreach ( string s in dictEntries )
dict.Add( s );
The result is .NET framework completely ignores the dictionary, and still inserts the above strings as plain text instead of just referencing a corresponding dictionary entry.
Then I've created my own implementation of IXmlDictionary:
class MyDictionary : IXmlDictionary
{
Dictionary<int, string> values = new Dictionary<int, string>();
Dictionary<string, int> keys = new Dictionary<string, int>();
MyDictionary()
{
string[] dictEntries = new string[]
{
"http://schemas.datacontract.org/2004/07/MyContracts",
"http://www.w3.org/2001/XMLSchema-instance",
"MyElementName1",
"MyElementName2",
"MyElementName3",
};
foreach ( var s in dictEntries )
this.Add( s );
}
static IXmlDictionary s_instance = new MyDictionary();
public static IXmlDictionary instance { get { return s_instance; } }
void Add( string val )
{
if ( keys.ContainsKey( val ) )
return;
int id = values.Count + 1;
values.Add( id, val );
keys.Add( val, id );
}
bool IXmlDictionary.TryLookup( XmlDictionaryString value, out XmlDictionaryString result )
{
if ( value.Dictionary == this )
{
result = value;
return true;
}
return this.TryLookup( value.Value, out result );
}
bool IXmlDictionary.TryLookup( int key, out XmlDictionaryString result )
{
string res;
if ( !values.TryGetValue( key, out res ) )
{
result = null;
return false;
}
result = new XmlDictionaryString( this, res, key );
return true;
}
public bool /* IXmlDictionary. */ TryLookup( string value, out XmlDictionaryString result )
{
int key;
if ( !keys.TryGetValue( value, out key ) )
{
result = null;
return false;
}
result = new XmlDictionaryString( this, value, key );
return true;
}
}
The result is - my TryLookup methods are called OK, however DataContractSerializer.WriteObject produces an empty document.
How do I use a pre-shared dictionary?
Thanks in advance!
P.S. I don't want to mess with XmlBinaryReaderSession/XmlBinaryWriterSession: I don't have "sessions", instead I have a 10 GB+ database accessed by many threads at once. What I want is just static pre-defined dictionary.
Update: OK I've figured out that I just need to call "XmlDictionaryWriter.Flush". The only remaining question is - why doesn't the system-supplied IXmlDictionary implementation work as expected?

for the XmlDictionaryWriter you need to use session.
example:
private static Stream SerializeBinaryWithDictionary(Person person,DataContractSerializer serializer)
{
var stream = new MemoryStream();
var dictionary = new XmlDictionary();
var session = new XmlBinaryWriterSession();
var key = 0;
session.TryAdd(dictionary.Add("FirstName"), out key);
session.TryAdd(dictionary.Add("LastName"), out key);
session.TryAdd(dictionary.Add("Birthday"), out key);
session.TryAdd(dictionary.Add("Person"), out key);
session.TryAdd(dictionary.Add("http://www.friseton.com/Name/2010/06"),out key);
session.TryAdd(dictionary.Add("http://www.w3.org/2001/XMLSchema-instance"),out key);
var writer = XmlDictionaryWriter.CreateBinaryWriter(stream, dictionary, session);
serializer.WriteObject(writer, person);
writer.Flush();
return stream;
}

The only way I way able to replicate the issue with the IXmlDictionary not being used was when my class wasn't decorated with a DataContract attribute. The following app displays the difference in sizes with decorated and undecorated classes.
using System;
using System.Runtime.Serialization;
using System.Xml;
namespace XmlPresharedDictionary
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Serialized sizes");
Console.WriteLine("-------------------------");
TestSerialization<MyXmlClassUndecorated>("Undecorated: ");
TestSerialization<MyXmlClassDecorated>("Decorated: ");
Console.ReadLine();
}
private static void TestSerialization<T>(string lineComment) where T : new()
{
XmlDictionary xmlDict = new XmlDictionary();
xmlDict.Add("MyElementName1");
DataContractSerializer serializer = new DataContractSerializer(typeof(T));
using (System.IO.MemoryStream stream = new System.IO.MemoryStream())
using (var writer = XmlDictionaryWriter.CreateBinaryWriter(stream, xmlDict))
{
serializer.WriteObject(writer, new T());
writer.Flush();
Console.WriteLine(lineComment + stream.Length.ToString());
}
}
}
//[DataContract]
public class MyXmlClassUndecorated
{
public MyElementName1[] MyElementName1 { get; set; }
public MyXmlClassUndecorated()
{
MyElementName1 = new MyElementName1[] { new MyElementName1("A A A A A"), new MyElementName1("A A A A A") };
}
}
[DataContract]
public class MyXmlClassDecorated
{
public MyElementName1[] MyElementName1 { get; set; }
public MyXmlClassDecorated()
{
MyElementName1 = new MyElementName1[] { new MyElementName1("A A A A A"), new MyElementName1("A A A A A") };
}
}
[DataContract]
public class MyElementName1
{
[DataMember]
public string Value { get; set; }
public MyElementName1(string value) { Value = value; }
}
}

Storing entity in XML, using MVVM to read/write in WPF Application

Say I've a class (model) called Instance with Properties DatbaseHostname, AccessManagerHostname, DatabaseUsername and DatabasePassword
public class Instance
{
private string _DatabaseHostname;
public string DatabaseHostname
{
get { return _DatabaseHostname; }
set { _DatabaseHostname = value; }
}
private string _AccessManagerHostname;
public string AccessManagerHostname
{
get { return _AccessManagerHostname; }
set { _AccessManagerHostname = value; }
}
private string _DatabaseUsername;
public string DatabaseUsername
{
get { return _DatabaseUsername; }
set { _DatabaseUsername = value; }
}
private string _DatabasePassword;
public string DatabasePassword
{
get { return _DatabasePassword; }
set { _DatabasePassword = value; }
}
}
I'm looking for a sample code to read/write this Model to XML (preferably linq2XML) => storing 1:n instances in XML.
i can manage the the view and ViewModel part myself, although it would be nice if someone had a sample of that part too..

Well, you could use Linq to XML, but your class is a perfect candidate for XML Serialization, which is much simpler IMHO :
var list = new List<Instance>();
...
// Serialization
var xs = new XmlSerializer(typeof(List<Instance>));
using (var writer = XmlWriter.Create(filename))
{
xs.Serialize(writer, list);
}
...
// Deserialization
using (var reader = XmlReader.Create(filename))
{
list = xs.Deserialize(reader) as List<Instance>;
}

Not sure how you want your xml structured, but this should work:
List<Instance> instances = new List<Instance>();
// Get your instances here...
var baseNode = new XElement("Instances");
instances.ForEach(instance => baseNode.Add("Instance",
new XAttribute("DatabaseHostname", instance.DatabaseHostname),
new XAttribute("AccessManagerHostname", instance.AccessManagerHostname),
new XAttribute("DatabaseUsername", instance.DatabaseUsername),
new XAttribute("DatabasePassword", instance.DatabasePassword)));