I am have a bunch of long json output in individual files. I need to read these files and deserialize them into the entities that originally generated the json (I have access to the original entities). Each file has the json output that was generated by serializing an object of type IEnumerable<Response<MyEntity>>.
I am getting an exception when attempting to deserialize, but I don't understand why. Here is my attempt to deserialize:
List<string> jsonOutputStrings;
// Read json from files and populate jsonOutputStrings list
List<Response<MyEntity>> apiResponses = new List<Response<MyEntity>>();
foreach (string json in jsonOutputStrings)
{
apiResponses.AddRange(JsonConvert.DeserializeObject<List<Response<MyEntity>>>(json, new JsonSerializerSettings { TypeNameHandling = TypeNameHandling.All }).ToList());
}
I also tried deserializing to an IEnumerable instead of a list:
apiResponses.AddRange(JsonConvert.DeserializeObject<IEnumerable<Response<MyEntity>>>(json, new JsonSerializerSettings { TypeNameHandling = TypeNameHandling.All }).ToList());
I get the following exception:
A first chance exception of type
'Newtonsoft.Json.JsonSerializationException' occurred in
Newtonsoft.Json.dll
Additional information: Cannot create and populate list type
System.Linq.Enumerable+WhereSelectListIterator`2[Entities.Requirement,Entities.RequirementEntity].
Path
'$values[0].ReturnedEntity.Summaries.$values[0].Requirements.$values',
line 1, position 715.
The entire json is too long to post (and also contains some confidential data) but I did find a place in the json that has this:
"Requirements":{"$id":"7","$type":"System.Linq.Enumerable+WhereSelectListIterator`2[[Entities.Requirement, Entities],[Entities.RequirementEntity, API.Entities]], System.Core","$values":[...]}
In the entity that I'm trying to deserialize into (same one that was originally serialized), Requirements is of type IEnumerable<Entities.RequirementEntity>.
It doesn't make sense to me how serialization from MyEntity works but deserializing to the same type doesn't. How do I solve this problem?
You must browse through Response<> and MyEntity and see how collections are initialized. This tells us that one of the collections in some class is created using Where method from linq. You can reproduce this error by executing this code:
class MyEntity
{
public MyEntity()
{
Data = new List<string>().Where(x => true);
}
public IEnumerable<string> Data { get; set; }
}
class Program
{
static void Main(string[] args)
{
string data = #"[{""Data"":[""a"",""b""]}]";
var j = JsonConvert.DeserializeObject<IEnumerable<MyEntity>>(data);
}
}
Another possibility is to have metadata in json. Then you have 2 solutions:
Set TypeNameHandling to TypeNameHandling.None (as somebody mentioned in comment);
Replace not working types in string with working ones
Using TypeNameHandling.None may lead to wrong deserialization for exmaple when you have IEnumerable<BaseType> and that list contains subtype of BaseType.
In that case you should go for second option. Basically you should replace any type that is not deserializing and replace it for example with List.
Sample code:
class MyEntity
{
public IEnumerable<string> Data { get; set; }
}
class Program
{
static void Main(string[] args)
{
IList<MyEntity> entities = new MyEntity[] {
new MyEntity { Data = new [] { "1", "2" }.Where(x => x != string.Empty) },
new MyEntity { Data = new [] { "A", "B" }.AsQueryable().Where(x => x != string.Empty) },
new MyEntity { Data = new List<string> { "A", "B" } },
};
string data = JsonConvert.SerializeObject(entities, Formatting.Indented, new JsonSerializerSettings { TypeNameHandling = TypeNameHandling.All });
data = Regex.Replace(data, "\"\\$type\":\\s+\"System.Linq.Enumerable\\+WhereArrayIterator(.+?), System.Core\",", "\"$type\": \"System.Collections.Generic.List$1, mscorlib\",", RegexOptions.Singleline);
var j = JsonConvert.DeserializeObject<IEnumerable<MyEntity>>(data, new JsonSerializerSettings { TypeNameHandling = TypeNameHandling.All });
}
}
Is there a Linq result in the serialized entity? Maybe defined as an IEnumerable and filled with a Linq Where result? It seems that's where your problem lies.
You should convert it to a List or Array before serializing.
Related
I have a few columns that contain JSON data. These particular columns are an array of enums.
When querying the MembershipType table, the serialization and deserialization works well and as expected. However, I have a SQL view that nests an array of MembershipTypes and doing causes EF Core to throw an error similar to this:
System.Exception: "Error converting value \"[\"Certified\"]\" to type 'DataLayer.Models.Members.MembershipTypeCategory[]'. Path '[0].History.MembershipType.Categories', line 1, position 585."
You can see all the extra quotes and backslashes that are added to the value. I have tried string replacing the quotes and backslashes, which works, but this can have bad affects with other string data.
Any ideas?
Code below:
SQL VIEW
SELECT
hist.*
, (
select * from Member.MembershipTypes
where id = hist.MembershipTypeId and Deleted = 0
FOR JSON PATH, WITHOUT_ARRAY_WRAPPER
) as MembershipType
...
Enum
public enum MembershipTypeCategory
{
Type1,
Type2,
Type3
}
and the class it's on:
[Table("MembershipTypes", Schema = "Member")]
public class MembershipType : EntityBase, IMembershipTypeDto
{
...
[NotMapped]
public MembershipTypeCategory[] Categories { get; set; }
...
}
In the ModelBuilder I have:
var settings = new JsonSerializerSettings
{
ContractResolver = new IgnoreVirtualResolver(),
PreserveReferencesHandling = PreserveReferencesHandling.None,
ReferenceLoopHandling = ReferenceLoopHandling.Ignore,
TypeNameHandling = TypeNameHandling.All,
Formatting = Formatting.None,
Error = HandleDeserializationError
};
modelBuilder.Entity<MembershipType>()
.Property(x => x.Categories)
.HasConversion(
v => JsonConvert.SerializeObject(v, Formatting.None, settings),
v => JsonConvert.DeserializeObject<MembershipTypeCategory[]>(v, settings)
);
return modelBuilder;
I think your converter is not ok. I generate code like your (in EF code first) and read/write data is ok, but string stored in db is:
{"$type":"XXX.Model.Tests.TestType[], XXX.Model","$values":[1,2]}
You have to think about converter again ;)
sample solution:
public static string ToJson(Enum[] values)
{
return ((values?.Length ?? 0) == 0) ? return "[]" : $"['{string.Join("','", values)}']";
}
public static Enum[] FromJson(string json)
{
return JsonConvert.DeserializeObject<Enum[]>(json);
}
Or using methods from JavaScriptSerializer - JSON serialization of enum as string
I have the following code:
return (DataTable)JsonConvert.DeserializeObject(_data, (typeof(DataTable)));
Then, I tried:
var jsonSettings = new JsonSerializerSettings
{
NullValueHandling = NullValueHandling.Ignore
};
return (DataTable)JsonConvert.DeserializeObject<DataTable>(_data, jsonSettings);
The return line is throwing the error:
{"Error converting value \"\" to type 'System.Double'."}
Lots of solutions online suggesting creating custom Class with nullable types but this won't work for me. I can't expect the json to be in a certain format. I have no control over the column count, column type, or column names.
You can supply settings to JsonConvert.DeserializeObject to tell it how to handle null values, in this case, and much more:
var settings = new JsonSerializerSettings
{
NullValueHandling = NullValueHandling.Ignore,
MissingMemberHandling = MissingMemberHandling.Ignore
};
var jsonModel = JsonConvert.DeserializeObject<Customer>(jsonString, settings);
An alternative solution for Thomas Hagström, which is my prefered, is to use the property attribute on the member variables.
For example when we invoke an API, it may or may not return the error message, so we can set the NullValueHandling property for ErrorMessage:
public class Response
{
public string Status;
public string ErrorCode;
[JsonProperty(NullValueHandling = NullValueHandling.Ignore)]
public string ErrorMessage;
}
var response = JsonConvert.DeserializeObject<Response>(data);
The benefit of this is to isolate the data definition (what) and deserialization (use), the deserilazation needn’t to care about the data property, so that two persons can work together, and the deserialize statement will be clean and simple.
You can subscribe to the 'Error' event and ignore the serialization error(s) as required.
static void Main(string[] args)
{
var a = JsonConvert.DeserializeObject<DataTable>("-- JSON STRING --", new JsonSerializerSettings
{
Error = HandleDeserializationError
});
}
public static void HandleDeserializationError(object sender, ErrorEventArgs errorArgs)
{
var currentError = errorArgs.ErrorContext.Error.Message;
errorArgs.ErrorContext.Handled = true;
}
ASP.NET CORE:
The accepted answer works perfectly. But in order to make the answer apply globally, in startup.cs file inside ConfigureServices method write the following:
services.AddControllers().AddNewtonsoftJson(options =>
{
options.SerializerSettings.NullValueHandling = NullValueHandling.Ignore;
});
The answer has been tested in a .Net Core 3.1 project.
This question already has answers here:
How to handle both a single item and an array for the same property using JSON.net
(9 answers)
Closed 4 years ago.
We're dealing with an json API result. Just to make our lives difficult the provider of the API returns on of the items as an array if there are multiple objects, or as a single object if there is only one.
e.g.
If there is only one object...
{
propertyA: {
first: "A",
second: "B"
}
}
Or if there are multiple:
{
propertyA: [
{
first: "A",
second: "B"
},
{
first: "A",
second: "B"
}
]
}
Does anybody have a good way of dealing with this scenario?
Ideally we'd like to serialize both to
public class ApiResult{
public ApiItem[] PropertyA {get;set;}
}
This works for the second example but of you encounter the first example you get a A first chance exception of type 'System.MissingMethodException' occurred in System.Web.Extensions.dll
Additional information: No parameterless constructor defined for type of 'ApiItem[]'.
I assume the class definition is as below
public class ApiResult
{
public ApiItem[] PropertyA { get; set; }
}
public class ApiItem
{
public string First { get; set; }
public string Second { get; set; }
}
You can deserialize the json into a dynamic variable, then check the type of d.propertyA. If it's JArray then propertyA is an array, so you can deserialize the json into a ApiResult. If it's JObject then propertyA is a single object, so you need to manually construct ApiItem and assign it to PropertyA of ApiResult. Consider the method below
public ApiResult Deserialize(string json)
{
ApiResult result = new ApiResult();
dynamic d = JsonConvert.DeserializeObject(json);
if (d.propertyA.GetType() == typeof (Newtonsoft.Json.Linq.JObject))
{
// propertyA is a single object, construct ApiItem manually
ApiItem item = new ApiItem();
item.First = d.propertyA.first;
item.Second = d.propertyA.second;
// assign item to result.PropertyA[0]
result.PropertyA = new ApiItem[1];
result.PropertyA[0] = item;
}
else if (d.propertyA.GetType() == typeof (Newtonsoft.Json.Linq.JArray))
{
// propertyA is an array, deserialize json into ApiResult
result = JsonConvert.DeserializeObject<ApiResult>(json);
}
return result;
}
The code above will return an instance of ApiResult for both json examples.
Working demo: https://dotnetfiddle.net/wBQKrp
Building upon ekad's answer, I made the code:
Shorter: as now you don't have map one by one every property inside ApiItem
Probably faster in the case of being already an Array (by not calling to both versions of JsonConvert.DeserializeObject with the same input of the json string)
Explicit about how to introduce other properties
Handling the error case of unknown type of our propertyA (the one that can be either an array or an object).
Notice that instead of JsonConvert.DeserializeObject, I call JObject.Parse, and then ToObject<> for only the part I need in that particular case:
static ApiResult Deserialize(string json)
{
JObject j = JObject.Parse(json);
var propA = j["propertyA"];
switch (propA.Type.ToString())
{
case "Object":
return new ApiResult {
PropertyA = new[]{propA.ToObject<ApiItem>()},
SomethingElse = j["somethingElse"].ToObject<string>(),
};
case "Array":
return j.ToObject<ApiResult>();
default:
throw new Exception("Invalid json with propertyA of type " + propA.Type.ToString());
}
}
The API is pretty much the same, but I've added SomethingElse (for showing how other properties can be easily handled with this approach):
public class ApiResult
{
public ApiItem[] PropertyA { get; set; }
public string SomethingElse { get; set; }
}
public class ApiItem
{
public string First { get; set; }
public string Second { get; set; }
}
Working demo: https://dotnetfiddle.net/VLbTMu
JSON# has a very lightweight tool that allows you to achieve this. It will retrieve embedded JSON, regardless of whether or not the embedded JSON is an object, or array, from within larger JSON objects:
const string schoolMetadata = #"{ "school": {...";
var jsonParser = new JsonObjectParser();
using (var stream =
new MemoryStream(Encoding.UTF8.GetBytes(schoolMetadata))) {
Json.Parse(_jsonParser, stream, "teachers");
}
Here we retrieve a "teachers" object from within a larger "school" object.
best way to Serialize/Deserialize to/from JSON is Json.NET
Popular high-performance JSON framework for .NET
Product product = new Product();
product.Name = "Apple";
product.Expiry = new DateTime(2008, 12, 28);
product.Sizes = new string[] { "Small" };
string json = JsonConvert.SerializeObject(product);
//{
// "Name": "Apple",
// "Expiry": "2008-12-28T00:00:00",
// "Sizes": [
// "Small"
// ]
//}
string json = #"{
'Name': 'Bad Boys',
'ReleaseDate': '1995-4-7T00:00:00',
'Genres': [
'Action',
'Comedy'
]
}";
Movie m = JsonConvert.DeserializeObject<Movie>(json);
string name = m.Name;
// Bad Boys
I have a json string that was created from serializing an array of objects :
[
{
"html": "foo"
},
{
"html": "bar"
}
]
How can I deserialize it to some iterable C# structure ? I've tried this code, but I'm getting No parameterless constructor defined for type of 'System.String'. error :
string[] htmlArr = new JavaScriptSerializer().Deserialize<String[]>(html);
What I want to receive is an iterable structure to get each 'html' object.
Use a class for each JSON object. Example:
public class HtmlItem
{
[DataMember(Name = "html")]
public string Html { get; set; }
}
JavaScriptSerializer ser = new JavaScriptSerializer();
// Serialize
string html = ser.Serialize(new List<HtmlItem> {
new HtmlItem { Html = "foo" },
new HtmlItem { Html = "bar" }
});
// Deserialize and print the html items.
List<HtmlItem> htmlList = ser.Deserialize<List<HtmlItem>>(html);
htmlList.ForEach((item) => Console.WriteLine(item.Html)); // foo bar
You can use Newtonsoft Json.NET (available from NuGet)
string json = #"[{""html"": ""foo""},{""html"": ""bar""}]";
var items = JsonConvert.DeserializeObject<List<Item>>(json);
Where
public class Item
{
public string Html { get; set; }
}
The docs site apparently isn't working right now... But I would try using JSON.NET ( http://james.newtonking.com/projects/json/help/ )
There are a couple of ways you can do it. You can deserialize in a very dynamic not type strict way or you can define an object that matches the json object exactly and deserialize into that. If there are many formats of JSON you'll have to serialize I would recommend using schemas.
Answer of nekman is not completely correct, the attribute should be JsonPropery instead of DataMember. (in this case you can remove the attribute since the deserializer doesn't care about the capital H)
public class HtmlItem
{
[JsonProperty("html")]
public string Html { get; set; }
}
JavaScriptSerializer ser = new JavaScriptSerializer();
// Serialize
string html = ser.Serialize(new List<HtmlItem> {
new HtmlItem { Html = "foo" },
new HtmlItem { Html = "bar" }
});
// Deserialize and print the html items.
List<HtmlItem> htmlList = ser.Deserialize<List<HtmlItem>>(html);
htmlList.ForEach((item) => Console.WriteLine(item.Html)); // foo bar
(This is all kind of background to give you context around my problem. You can skip down to "The Problem" and read that, and then maybe come back up and skim the background if you want to get straight to the point. Sorry it's a wall of text!)
I've got a bunch of terrible, terrible JSON I need to store in a database. Essentially, someone took a large XML file, and serialized it to one, big, flat JSON object by simply using the XML's XPath. Here's an example of what I mean:
Original XML:
<statistics>
<sample>
<date>2012-5-10</date>
<duration>11.2</duration>
</sample>
<sample>
<date>2012-6-10</date>
<duration>13.1</duration>
</sample>
<sample>
<date>2012-7-10</date>
<duration>10.0</duration>
</sample>
</statistics>
The Horrible JSON I Have to Work With: (basically just the XPath from above)
{
"statistics":"",
"statistics/sample":"",
"statistics/sample/date":"2012-5-10",
"statistics/sample/duration":"11.2",
"statistics/sample#1":"",
"statistics/sample/date#1":"2012-6-10",
"statistics/sample/duration#1":"13.1",
"statistics/sample#2":"",
"statistics/sample/date#2":"2012-7-10",
"statistics/sample/duration#2":"10.0",
}
And now I need to put it in a database that contains a statistics table with date and duration columns.
How I'm Currently Doing It: (or at least a simple example of how)
Tuple<string, string>[] jsonToColumn = // maps JSON value to SQL table column
{
new Tuple<string, string>("statistics/sample/date", "date"),
new Tuple<string, string>("statistics/sample/duration", "duration")
};
// Parse the JSON text
// jsonText is just a string holding the raw JSON
JavaScriptSerializer serializer = new JavaScriptSerializer();
Dictionary<string, object> json = serializer.DeserializeObject(jsonText) as Dictionary<string, object>;
// Duplicate JSON fields have some "#\d+" string appended to them, so we can
// find these and use them to help uniquely identify each individual sample.
List<string> sampleIndices = new List<string>();
foreach (string k in json.Keys)
{
Match m = Regex.Match(k, "^statistics/sample(#\\d*)?$");
if (m.Success)
{
sampleIndices .Add(m.Groups[1].Value);
}
}
// And now take each "index" (of the form "#\d+" (or "" for the first item))
// and generate a SQL query for its sample.
foreach (string index in compressionIndices)
{
List<string> values = new List<string>();
List<string> columns = new List<string>();
foreach (Tuple<string, string> t in jsonToColumn)
{
object result;
if (json.TryGetValue(t.Item1 + index, out result))
{
columns.Add(t.Item2);
values.Add(result);
}
}
string statement = "INSERT INTO statistics(" + string.Join(", ", columns) + ") VALUES(" + string.Join(", ", values) + ");";
// And execute the statement...
}
However, I'd like to use an ADO.NET Entity Data Model (or something LINQ-ish) rather than this hackery, because I need to start performing some queries before inserting and apply some updates, and creating and executing my own SQL statements is just... cumbersome. I created an ADO.NET Entity Data Model (.edmx) file and set things up, and now I can easily use this model to interact with and write to my database.
The Problem/Question
The problem is I'm not sure how to best map from my JSON to my ADO.NET Entity Data Model Statistic object (that represents a sample/record in the statistics table). The easiest would be to change my Tuple list to use something like pointers-to-members (a la Tuple<string, Statistic::*Duration>("statistics/sample/duration", &Statistic::Duration) if this were C++), but a) I don't even think this is possible in C#, and b) even if it was it makes my Tuples all have different types.
What are some of my options here? How can I best map the JSON to my Statistic objects? I'm kinda new to the LINQ world, and am wondering if there's a way (through LINQ or something else) to map these values.
It's a sub-optimal position I'm in (working with such poor JSON), and I recognize it's possible that maybe my current method is better than anything else given my situation, and if that's the case I'd even accept that as my answer. But I would really like to explore what options there are for mapping this JSON to a C# object (and ultimately to the SQL database).
If the whole problem is mapping that "JSON" you have to POCO entities, here's an example on how to Deserialize it using a Custom JavascriptConverter:
Your POCO entities:
public class Statistics
{
public Statistics()
{
Samples = new List<Sample>();
}
public List<Sample> Samples { get; set; }
}
public class Sample
{
public DateTime Date { get; set; }
public float Duration { get; set; }
}
Your StatsConverter:
public class StatsConverter : JavaScriptConverter
{
public override object Deserialize(IDictionary<string, object> dictionary, Type type, JavaScriptSerializer serializer)
{
if (dictionary == null)
throw new ArgumentNullException("dictionary");
else if (type == typeof(Statistics))
{
Statistics statistics = null;
Sample sample = null;
{
foreach (var item in dictionary.Keys)
{
if (dictionary[item] is string && item.Contains("duration"))
sample.Duration = float.Parse(dictionary[item].ToString());
else if (dictionary[item] is string && item.Contains("date"))
sample.Date = DateTime.Parse((dictionary[item].ToString()));
else if (dictionary[item] is string && item.Contains("sample"))
{
sample = new Sample();
statistics.Samples.Add(sample);
}
else if (dictionary[item] is string && item.Contains("statistics"))
statistics = new Statistics();
}
}
return statistics;
}
return null;
}
public override IDictionary<string, object> Serialize(object obj, JavaScriptSerializer serializer)
{
throw new NotImplementedException();
}
public override IEnumerable<Type> SupportedTypes
{
get { return new ReadOnlyCollection<Type>(new List<Type>(new Type[] { typeof(Statistics)})); }
}
}
Now a sample on how to Deserialize it:
string json = #"{
""statistics"":"""",
""statistics/sample"":"""",
""statistics/sample/date"":""2012-5-10"",
""statistics/sample/duration"":""11.2"",
""statistics/sample#1"":"""",
""statistics/sample/date#1"":""2012-6-10"",
""statistics/sample/duration#1"":""13.1"",
""statistics/sample#2"":"""",
""statistics/sample/date#2"":""2012-7-10"",
""statistics/sample/duration#2"":""10.0""
}";
//These are the only 4 lines you'll require on your code
JavaScriptSerializer serializer = new JavaScriptSerializer();
StatsConverter sc = new StatsConverter();
serializer.RegisterConverters(new JavaScriptConverter[] { new StatsConverter() });
Statistics stats = serializer.Deserialize<Statistics>(json);
stats object above will deserialize to a Statistics object with 3 Sample objects in its Samples collection.
If you use a ORM (e.g. EntityFramework) then you are just working against a datamodel so assuming you would then have a model class defined something like...
public class Sample
{
public string Date { get; set; }
public double Duration { get; set; }
}
Then you could do something like this...
List<Sample> samples = new List<Sample>();
Dictionary<string, object> data = ser.DeserializeObject(json) as Dictionary<string, object>;
var keys = data.Keys.ToList();
for (int i = 0; i <keys.Count; i++)
{
string k = keys[i];
if (Regex.IsMatch(k, "^statistics/sample(#\\d*)?$"))
{
samples.Add(new Sample
{
Date = (string)data[keys[i + 1]],
Duration = double.Parse((string)data[keys[i + 2]])
});
}
}
I just populate a list for the example and again if you are using something like EntityFramework then you could could just be adding the instances directly to your repository/datacontext.