Is there anything terribly inefficient here? It seems like this process is taking way longer than it should. I am parsing many JSON files each with a JsonArray of objects. Maybe someone with more experience could point out an error in this method of parsing the JSON into objects, thereby saving me a ton of time.
Also, memory usage slowly creeps upwards MB by MB sometimes causing outofmemoryexceptions..
public void Parse(){
using (BabysFirstsUsersDataEntities db = new BabysFirstsUsersDataEntities()
{
foreach (var path in Directory.EnumerateFiles(#"C:\\examplepath\").OrderBy(f => f))
{
string jsonString = System.IO.File.ReadAllText(path);
JToken tok = JObject.Parse(jsonString);
Debug.WriteLine("path: " + path);
foreach (var x in tok.First.First)
{
JsonUserImageDTO jdto = x.ToObject<JsonUserImageDTO>();
UserImageList uil = jdto.ToDataModel();
if (uil.REID != null)
db.UserImageLists.Add(uil);
}
}
db.SaveChanges();
}
}
An example of what one of the JSON strings in each .json file looks like below. Note that there are around 1000 of these files and each can have thousands of such entries:
{
"results": [
{
"ACL": {
"asdf": {
"read": true,
"write": true
},
"role:admin": { "read": true }
},
"REID": "exampleID",
"createdAt": "datetime-string",
"email": "example",
"objectId": "example",
"updatedAt": "datetimestring",
"urlCount": 1,
"urlList": [ "exampleurl" ]
},
{
"ACL": {
"asdf": {
"read": true,
"write": true
},
"role:admin": { "read": true }
},
"REID": "exampleID",
"createdAt": "datetime-string",
"email": "example",
"objectId": "example",
"updatedAt": "datetimestring",
"urlCount": 1,
"urlList": [ "exampleurl" ]
}
]
}
It looks like there could be several places that could causing the slowness.
Deserialzing JSON
Transforming the object twice (jdto, then uil)
Saving to the database
It may be worth profiling the code to find out exactly what part is taking longer than you'd expect. That said there are some things you can do to generally improve this code.
Deserialize from a steam instead of a string. The way you have it, you basically have the object in memory twice-- once as a string, then once as tok. See the second example in the docs for how to use a stream. Actually, in your case you the same information in memory 4 times -- the string, tok, jdto, and uil. Which brings me to the next point..
Try to eliminate some of the intermediate representations of your object. Generally, the more objects you have laying around, the more time you will spend waiting on the GC.
Move the filtering on the path name to the part where you call EnumerateFiles(). There is no sense in deserializing a file if you are not going to do anything with it.
Have you actually profiled your code? See Erik Lippert's performance rant: Use a profiler or other analysis tool to determine empirically where the bottleneck is before you start investigating alternatives. For instance, you actual performance problem may be somewhere in the BabysFirstsUsersDataEntities db class.
That being said, my immediate reaction is that you have too many intermediate representations of your data, the construction, population and garbage collection of which all take time. These include:
The jsonString which may be large enough to go on the large object heap, and thus permanently impair the performance and memory use of your process.
The JToken tok representation of your entire JSON hierarchy.
Each individual JsonUserImageDTO.
What I would suggest is to eliminate as many of these intermediate representations as possible. As suggested in the documentation you should load directly from a stream rather that loading to a string and parsing that string.
You can also eliminate the JToken tok by populating your data model directly. Let's say your BabysFirstsUsersDataEntities looks like this (I'm just guessing here):
public class BabysFirstsUsersDataEntities
{
public BabysFirstsUsersDataEntities() { this.UserImageLists = new List<UserImageList>(); }
public List<UserImageList> UserImageLists { get; set; }
}
public class UserImageList
{
public string email { get; set; }
public List<string> urlList;
}
And your DTO model looks something like this model provided by http://json2csharp.com/:
public class RootObjectDTO
{
public ICollection<JsonUserImageDTO> results { get; set; }
}
public class JsonUserImageDTO
{
public ACL ACL { get; set; }
public string REID { get; set; }
public string createdAt { get; set; }
public string email { get; set; }
public string objectId { get; set; }
public string updatedAt { get; set; }
public int urlCount { get; set; }
public List<string> urlList { get; set; }
public UserImageList ToDataModel()
{
return new UserImageList { email = email, urlList = urlList };
}
}
public class Asdf
{
public bool read { get; set; }
public bool write { get; set; }
}
public class RoleAdmin
{
public bool read { get; set; }
}
public class ACL
{
public Asdf asdf { get; set; }
[JsonProperty("role:admin")]
public RoleAdmin RoleAdmin { get; set; }
}
Then create the following generic ConvertingCollection<TIn, TOut> utility class:
public class ConvertingCollection<TIn, TOut> : BaseConvertingCollection<TIn, TOut, ICollection<TIn>>
{
readonly Func<TOut, TIn> toInner;
public ConvertingCollection(Func<ICollection<TIn>> getCollection, Func<TIn, TOut> toOuter, Func<TOut, TIn> toInner)
: base(getCollection, toOuter)
{
if (toInner == null)
throw new ArgumentNullException();
this.toInner = toInner;
}
protected TIn ToInner(TOut outer) { return toInner(outer); }
public override void Add(TOut item)
{
Collection.Add(ToInner(item));
}
public override void Clear()
{
Collection.Clear();
}
public override bool IsReadOnly { get { return Collection.IsReadOnly; } }
public override bool Remove(TOut item)
{
return Collection.Remove(ToInner(item));
}
public override bool Contains(TOut item)
{
return Collection.Contains(ToInner(item));
}
}
public abstract class BaseConvertingCollection<TIn, TOut, TCollection> : ICollection<TOut>
where TCollection : ICollection<TIn>
{
readonly Func<TCollection> getCollection;
readonly Func<TIn, TOut> toOuter;
public BaseConvertingCollection(Func<TCollection> getCollection, Func<TIn, TOut> toOuter)
{
if (getCollection == null || toOuter == null)
throw new ArgumentNullException();
this.getCollection = getCollection;
this.toOuter = toOuter;
}
protected TCollection Collection { get { return getCollection(); } }
protected TOut ToOuter(TIn inner) { return toOuter(inner); }
#region ICollection<TOut> Members
public abstract void Add(TOut item);
public abstract void Clear();
public virtual bool Contains(TOut item)
{
var comparer = EqualityComparer<TOut>.Default;
foreach (var member in Collection)
if (comparer.Equals(item, ToOuter(member)))
return true;
return false;
}
public void CopyTo(TOut[] array, int arrayIndex)
{
foreach (var item in this)
array[arrayIndex++] = item;
}
public int Count { get { return Collection.Count; } }
public abstract bool IsReadOnly { get; }
public abstract bool Remove(TOut item);
#endregion
#region IEnumerable<TOut> Members
public IEnumerator<TOut> GetEnumerator()
{
foreach (var item in Collection)
yield return ToOuter(item);
}
#endregion
#region IEnumerable Members
IEnumerator IEnumerable.GetEnumerator()
{
return GetEnumerator();
}
#endregion
}
You can now populate your db directly as follows:
var rootDTO = new RootObjectDTO
{
results = new ConvertingCollection<UserImageList, JsonUserImageDTO>(() => db.UserImageLists, (x) => { throw new NotImplementedException(); }, (x) => x.ToDataModel())
};
using (var stream = File.Open(path, FileMode.Open))
using (var reader = new StreamReader(stream))
{
JsonSerializer.CreateDefault().Populate(reader, rootDTO);
}
By populating a preallocated rootDTO and ConvertingCollection<UserImageList, JsonUserImageDTO>, your db.UserImageLists will get populated with the contents of the JSON with fewer intermediate representations.
You can create the objects and then deserialize them.
Example:
JsonConvert.DeserializeObject<RootObject>(jsonString);
public class Asdf
{
public bool read { get; set; }
public bool write { get; set; }
}
public class RoleAdmin
{
public bool read { get; set; }
}
public class ACL
{
public Asdf asdf { get; set; }
public RoleAdmin { get; set; }
}
public class Result
{
public ACL ACL { get; set; }
public string REID { get; set; }
public string createdAt { get; set; }
public string email { get; set; }
public string objectId { get; set; }
public string updatedAt { get; set; }
public int urlCount { get; set; }
public List<string> urlList { get; set; }
}
public class RootObject
{
public List<Result> results { get; set; }
}
Related
Some of my actions accept models like:
public class PaymentRequest
{
public decimal Amount { get; set; }
public bool? SaveCard { get; set; }
public int? SmsCode { get; set; }
public BankCardDetails Card { get; set; }
}
public class BankCardDetails
{
public string Number { get; set; }
public string HolderName { get; set; }
public string ExpiryDate { get; set; }
public string ValidationCode { get; set; }
}
And the action method looks like:
[HttpPost]
[Route("api/v1/payment/pay")]
public Task<BankCardActionResponse> Pay([FromBody] PaymentRequest request)
{
if (request == null)
throw new HttpResponseException(HttpStatusCode.BadRequest);
return _paymentService.PayAsync(DataUserHelper.PhoneNumber, request);
}
I use Nlog. I think it's clear this is a bad idea to log all this bank data. My log config file contained the following line:
<attribute name="user-requestBody" layout="${aspnet-request-posted-body}"/>
I logged the request. I decided to refactor that and planned the following strategy. Actions that contain sensitive data into their requests I will mark with an attribute like
[RequestMethodFormatter(typeof(PaymentRequest))]
then take a look at my custom renderer:
[LayoutRenderer("http-request")]
public class NLogHttpRequestLayoutRenderer : AspNetRequestPostedBody
{
protected override void DoAppend(StringBuilder builder, LogEventInfo logEvent)
{
base.DoAppend(builder, logEvent);
var body = builder.ToString();
// Get attribute of the called action.
var type = ... // How can I get "PaymentRequest" from the [RequestMethodFormatter(typeof(PaymentRequest))]
var res = MaskHelper.GetMaskedJsonString(body, type);
// ... and so on
}
}
I think you understand the idea. I need the type from the method's RequestMethodFormatter attribute. Is it even possible to get it into the renderer? I need it because I'm going to deserialize request JSON into particular models (it's gonna be into the MaskHelper.GetMaskedJsonString), work with the models masking the data, serialize it back into JSON.
So, did I choose a wrong approach? Or it's possible to get the type from the attribute into the renderer?
After some research, I ended up with the following solution:
namespace ConsoleApp7
{
internal class Program
{
private static void Main()
{
var sourceJson = GetSourceJson();
var userInfo = JsonConvert.DeserializeObject(sourceJson, typeof(User));
Console.WriteLine("----- Serialize without Resolver-----");
Console.WriteLine(JsonConvert.SerializeObject(userInfo));
Console.WriteLine("----- Serialize with Resolver-----");
Console.WriteLine(JsonConvert.SerializeObject(userInfo, new JsonSerializerSettings
{
ContractResolver = new MaskPropertyResolver()
}));
}
private static string GetSourceJson()
{
var guid = Guid.Parse("3e92f0c4-55dc-474b-ae21-8b3dac1a0942");
return JsonConvert.SerializeObject(new User
{
UserId = guid,
Age = 19,
Name = "John",
BirthDate = new DateTime(1990, 5, 12),
Hobbies = new[]
{
new Hobby
{
Name = "Football",
Rating = 5,
DurationYears = 3,
},
new Hobby
{
Name = "Basketball",
Rating = 7,
DurationYears = 4,
}
}
});
}
}
public class User
{
[MaskGuidValue]
public Guid UserId { get; set; }
[MaskStringValue("***")] public string Name { get; set; }
public int Age { get; set; }
[MaskDateTimeValue]
public DateTime BirthDate { get; set; }
public Hobby[] Hobbies { get; set; }
}
public class Hobby
{
[MaskStringValue("----")]
public string Name { get; set; }
[MaskIntValue(replacement: 11111)]
public int Rating { get; set; }
public int DurationYears { get; set; }
}
public class MaskPropertyResolver : DefaultContractResolver
{
protected override IList<JsonProperty> CreateProperties(Type type, MemberSerialization memberSerialization)
{
var props = base.CreateProperties(type, memberSerialization);
var allowedPropertyTypes = new Type[]
{
typeof(Guid),
typeof(DateTime),
typeof(string),
typeof(int),
};
foreach (var prop in props.Where(p => allowedPropertyTypes.Contains(p.PropertyType)))
{
if (prop.UnderlyingName == null)
continue;
var propertyInfo = type.GetProperty(prop.UnderlyingName);
var attribute =
propertyInfo?.GetCustomAttributes().FirstOrDefault(x => x is IMaskAttribute) as IMaskAttribute;
if (attribute == null)
{
continue;
}
if (attribute.Type != propertyInfo.PropertyType)
{
// Log this case, cause somebody used wrong attribute
continue;
}
prop.ValueProvider = new MaskValueProvider(propertyInfo, attribute.Replacement, attribute.Type);
}
return props;
}
private class MaskValueProvider : IValueProvider
{
private readonly PropertyInfo _targetProperty;
private readonly object _replacement;
private readonly Type _type;
public MaskValueProvider(PropertyInfo targetProperty, object replacement, Type type)
{
_targetProperty = targetProperty;
_replacement = replacement;
_type = type;
}
public object GetValue(object target)
{
return _replacement;
}
public void SetValue(object target, object value)
{
_targetProperty.SetValue(target, value);
}
}
}
[AttributeUsage(AttributeTargets.Property)]
public class MaskStringValueAttribute : Attribute, IMaskAttribute
{
public Type Type => typeof(string);
public object Replacement { get; }
public MaskStringValueAttribute(string replacement)
{
Replacement = replacement;
}
}
[AttributeUsage(AttributeTargets.Property)]
public class MaskIntValueAttribute : Attribute, IMaskAttribute
{
public object Replacement { get; }
public Type Type => typeof(int);
public MaskIntValueAttribute(int replacement)
{
Replacement = replacement;
}
}
[AttributeUsage(AttributeTargets.Property)]
public class MaskGuidValueAttribute : Attribute, IMaskAttribute
{
public Type Type => typeof(Guid);
public object Replacement => Guid.Empty;
}
[AttributeUsage(AttributeTargets.Property)]
public class MaskDateTimeValueAttribute : Attribute, IMaskAttribute
{
public Type Type => typeof(DateTime);
public object Replacement => new DateTime(1970, 1, 1);
}
public interface IMaskAttribute
{
Type Type { get; }
object Replacement { get; }
}
}
I hope somebody will find it helpful.
You can try nuget package https://www.nuget.org/packages/Slin.Masking and https://www.nuget.org/packages/Slin.Masking.NLog.
It can easily be integrated with DotNet projects with slight changes, and you can define your rules for it. But the document needs some improvement.
As a suggestion, you can use two files:
masking.json (can be a generic one, that shared across all projects)
masking.custom.json (can be used with particular rules for specific projects)
Good Morning, I'm falling on a problem that apparently should be easy but I cannot find a solution.
I have already red this and this posts, nothing helped.
I have an API, that returns data in JSON format with this schema, it has an indexer (I know it's idiotic, I didn't develop it) of the result before each object:
{
"total":2,
"0":{
"id_property":5028080,
"id_company":11719097,
....
},
"1":{
"id_property":4996958,
"id_company":11719097,
....
},
"status":"success"
}
I'm trying to deserialize it with System.Text.Json but it fails also with Newtonsoft.
According to the previous links I made a C# schema like this:
public class RootDTO
{
public int total { get; set; }
public Dictionary<string, PropertyDTO> Property { get; set; }
public string status { get; set; }
}
public class PropertyDTO
{
public int id { get; set; }
public string url { get; set; }
// OTHER PROPERTIES ...
}
I tried to deserialize it with the root Object and directly with the Dictionary in this way, either way it is failing. Null property in the first case, exception in the second case since it finds a "total" property not matching the schema.
RootDTO result = JsonSerializer.Deserialize<RootDTO>(content);
Dictionary<string, PropertyDTO> result = JsonSerializer.Deserialize<Dictionary<string, PropertyDTO>>(content);
Any idea on how to solve this apparently simple problem?
Many thanks everybody
Another thing I wasn't aware is that inside I have a SubObject with the same idiotic format
....
"galleries": [
{
"id": 4441310,
"0": {
"id": 146843541,
"url": "xxx",
"description": "",
"filename": "83732120220325094904.jpg",
"position": 1,
"url_big": "yyy",
"url_original": "kkk"
},
"1": {
"id": 146843542,
"url": "xxx",
"description": "",
"filename": "83732220220325094904.jpg",
"position": 2,
"url_big": "yyy",
"url_original": "kkk"
}
....
}
....
The issue there is that the objects are not represented by properties of the target type RootDTO.
You can handle that using the concept of overflow JSON: https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json-handle-overflow?pivots=dotnet-6-0
public class RootDTO
{
public int total { get; set; }
public string status { get; set; }
//all the "overflowing" properties will be here
[System.Text.Json.Serialization.JsonExtensionData]
public Dictionary<string, JsonElement> Properties { get; set; }
}
RootDTO result = System.Text.Json.JsonSerializer.Deserialize<RootDTO>(json);
You can later deserialize the JsonElement in your PropertyDTO type.
You can get the values using a method like this where we deserialize to a dict of string keys and take any object so the code won't complain about the various types we have. Then further break it down into the specific fields you need.
EDIT - Try the code here
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
public class Program
{
public static void Main()
{
var json = #"{
""total"": 2,
""0"": {
""id_property"": 5028080,
""id_company"": 11719097
},
""1"": {
""id_property"": 4996958,
""id_company"": 11719097
},
""status"": ""success""
}";
var data = JsonSerializer.Deserialize<Dictionary<string,object>>(json);
foreach (var obj in data.Where(t => t.Key != "total" && t.Key != "status")) {
var dataObj = JsonSerializer.Deserialize<PropertyDTO>(obj.Value.ToString());
Console.WriteLine(dataObj.id_property);
Console.WriteLine(dataObj.id_company);
}
}
}
public class PropertyDTO
{
public int id_property { get; set; }
public int id_company { get; set; }
// OTHER PROPERTIES ...
}
the easiest way would be to use Newtonsoft.Json. You can try to use JsonExtensionData , but in this case you will have only a generic < string, object > dictionary. This code creates a typed dictionary
using Newtonsoft.Json;
var jsonParsed = JObject.Parse(json);
RootDTO rootDto = new RootDTO
{
total = (int)jsonParsed["total"],
properties = jsonParsed.Properties().Where(p => p.Value.Type is JTokenType.Object)
.ToDictionary(p =>p.Name, p=> p.Value.ToObject<PropertyDTO>());
status = (string)jsonParsed["status"]
}
classes
public class RootDTO
{
public int total { get; set; }
public Dictionary<string, PropertyDTO> properties { get; set; }
public string status { get; set; }
}
public class PropertyDTO
{
[JsonProperty("id_property")]
public int properyId { get; set; }
[JsonProperty("id_company")]
public int companyId { get; set; }
}
A work around can be the following:
Create another class that wraps desired values
public class IdioticWrapper<TValue> : IDictionary<string, JsonElement>
{
private IDictionary<string, TValue> desiredValues
= new Dictionary<string, TValue>();
private IDictionary<string, JsonElement> properties
= new Dictionary<string, JsonElement>();
public JsonElement this[string key]
{
get => this.properties[key];
set
{
// TODO: some checks like is null can deserialize ...
this.desiredValues[key] = JsonSerializer.Deserialize<TValue>(value.GetRawText());
}
}
// There are desired models
public IEnumerable<TValue> DesiredValues
=> this.desiredValues.Values;
public ICollection<string> Keys => this.properties.Keys;
public ICollection<JsonElement> Values => this.properties.Values;
public int Count => this.properties.Count;
public bool IsReadOnly => this.properties.IsReadOnly;
public void Add(string key, JsonElement value) => this.properties.Add(key, value);
public void Add(KeyValuePair<string, JsonElement> item) => this.properties.Add(item);
public void Clear() => this.properties.Clear();
public bool Contains(KeyValuePair<string, JsonElement> item) => properties.Contains(item);
public bool ContainsKey(string key) => this.properties.ContainsKey(key);
public void CopyTo(KeyValuePair<string, JsonElement>[] array, int arrayIndex) => this.properties.CopyTo(array, arrayIndex);
public IEnumerator<KeyValuePair<string, JsonElement>> GetEnumerator() => this.properties.GetEnumerator();
public bool Remove(string key) => this.properties.Remove(key);
public bool Remove(KeyValuePair<string, JsonElement> item) => this.properties.Remove(item);
public bool TryGetValue(string key, [MaybeNullWhen(false)] out JsonElement value) => this.properties.TryGetValue(key, out value);
IEnumerator IEnumerable.GetEnumerator() => this.GetEnumerator();
}
2 use that wraper wehenever you need to wrap nested data of that kind
public class PropertyDTO
{
public int id_property { get; set; }
public int id_company { get; set; }
[JsonExtensionData]
public IdioticWrapper<object> properties { get; set; }
}
public class RootDTO
{
public int total { get; set; }
public string status { get; set; }
[JsonExtensionData]
public IdioticWrapper<PropertyDTO> properties { get; set; }
public IEnumerable<PropertyDTO> Values
=> this.properties.DesiredValues;
}
and at the end you shold get the correct result
var result = JsonSerializer.Deserialize<RootDTO>(jsonContent);
Console.WriteLine(result.total);
Console.WriteLine(result.status);
var values = result.Values;
// ... and so on
I'm currently building an application where I have some tasks that are sharing a Dictionary<Object, Guid>.
To avoid conflicts, the actual Dictionaries are contained within a DictionaryHandler-class with three methods:
class DictionaryHandler
{
private object lockProcesses = new object();
private Dictionary<Process, Guid> processes = new Dictionary<Process, Guid>();
public Dictionary<Process, Guid> GetProcesses()
{
lock (lockProcesses)
{
// TODO
}
}
public void AddToProcesses(Process process, Guid guid)
{
lock (lockProcesses)
{
processes.Add(process, guid);
}
}
public void RemoveFromProcesses(Process process)
{
lock (lockProcesses)
{
processes.Remove(process);
}
}
}
For context, this is the Process-class:
public class Process
{
public string Name { get; }
public bool Enabled { get; }
public TimeSpan RuntimeWindowStart { get; }
public TimeSpan RuntimeWindowEnd { get; }
public TimeSpan Cooldown { get; }
public int Priority { get; }
public string Username { get; }
public string ExceptionDate { get; }
public string ExceptionDay { get; }
public string AllowedWorkdays { get; }
public string SpecificResource { get; }
public string Assigned { get; set; }
public DateTime? Timestamp { get; set; }
public Process(string name, bool enabled, TimeSpan runtimeWindowStart, TimeSpan runtimeWindowEnd, TimeSpan cooldown, int priority, string username, string exceptionDate, string exceptionDay, string allowedWorkdays, string specificResource, string assigned, DateTime? timestamp)
{
Name = name;
Enabled = enabled;
RuntimeWindowStart = runtimeWindowStart;
RuntimeWindowEnd = runtimeWindowEnd;
Cooldown = cooldown;
Priority = priority;
Username = username;
ExceptionDate = exceptionDate;
ExceptionDay = exceptionDay;
AllowedWorkdays = allowedWorkdays;
SpecificResource = specificResource;
Assigned = assigned;
Timestamp = timestamp;
}
}
My main issue is that I want to find a way to return a copy of the Dictionary through the GetProcesses()-method, without returning a reference to the "actual" dictionary.
As far as I can see, the optimal way to do this is to Serialize and Deserialize the Dictionary and return that. But I'm having a hard time doing this, as I'm unable to find an example that matches my case.
I've read this and this and tried to combine the two - unfortunately without luck.
My main issue is that I want to find a way to return a copy of the Dictionary through the GetProcesses()-method, without returning a reference to the "actual" dictionary.
If you don't need to clone the values, you can use the constructor overload to Dictionary which takes an existing IDictionary:
new Dictionary<Process, Guid>(processes);
If you do need to clone the values, you can use something like:
public static Dictionary<Process, Guid> DeepClone<TKey, TValue>(Dictionary<Process, Guid> source)
{
var ret = new Dictionary<Process, Guid>(source.Count, source.Comparer);
foreach (var entry in source)
{
ret.Add(entry.Key, entry.Value);
}
return ret;
}
If you do need create a copy of Process class instance while copying dictionary, you may use:
public class Process
{
// some properties here
public Process ShallowCopy()
{
return (Process) this.MemberwiseClone();
}
}
public static Dictionary<Process, Guid> DeepClone<TKey, TValue>(Dictionary<Process, Guid> source)
{
var ret = new Dictionary<Process, Guid>(source.Count, source.Comparer);
foreach (var entry in source)
{
ret.Add(entry.Key.ShallowCopy(), entry.Value);
}
return ret;
}
Use String instead of Guid.
Guid is structure type. Therefore serialize and deserialize methods might not work correctly.
This question already has answers here:
How can I parse a JSON string that would cause illegal C# identifiers?
(3 answers)
Closed 8 years ago.
I am trying to deserialize a Json response from an API.
The data looks like this
{
"response": {
"6112": {
"ID": 6112,
"Title": "AdditionalPhotos"
},
"5982": {
"ID": 5982,
"Title": "BikeRide"
},
"total_records": "20",
"returned_count": 10,
"returned_records": "1-10"
}
}
C# class:
public class Products
{
public class Product
{
public string Id { get; set; }
public string Title { get; set; }
}
public Product product { get; set; }
}
public class ss
{
public Dictionary<string, Products.Product> Response { get; set; }
public string total_records { get; set; }
}
Serialization code
ss res = Newtonsoft.Json.JsonConvert.DeserializeObject<ss>(jsonData());
I can get it to work without the total_records entry and below by deserializng to a Dictionary <string , Product>. But I cannot figure out how to get it to work. This is the error I get
Error converting value "20" to type 'Products+Product'. Path 'response.total_records'
I know why I get the error, but I'm unsure how I can proceed without going in and substringing from total_records down. I have no control over the API data.
Edit: you guys are fast, I was still getting to putting the classes up
First you json is not valid one, it should look like this
{
"response":{
"6112":{
"ID":"6112",
"Title":"Additional Photos",
},
"5982":{
"ID":"5982",
"Title":"Bike Ride",
},
"total_records": "20",
"returned_count": "10",
"returned_records": "1-10",
}
}
If you mean the response to contain list it should look like this
{
"response":{
"myArray": [
{
"ID":"6112",
"Title":"Additional Photos",
},
{
"ID":"5982",
"Title":"Bike Ride",
}
],
"total_records": "20",
"returned_count": "10",
"returned_records": "1-10",
}
}
So your code look like this
public class MyArray
{
public string ID { get; set; }
public string Title { get; set; }
}
public class Response
{
public List<MyArray> myArray { get; set; }
public string total_records { get; set; }
public string returned_count { get; set; }
public string returned_records { get; set; }
}
public class RootObject
{
public Response response { get; set; }
}
If you have control over API response then please refer to Mzf's answer.
If you don't have control over API then it may not be possible to do this particular deserialization on one go. You might have to loop.
Here's my take.
Update
Modified my approach:
Created a class Response which inherits from Dictionary<string, Product>, and added the metadata parts like total_records, records_count to it's public properties. And created a JsonConverter that can deserialize JObject to Response class.
The logic used for deserialization is quite simple:
Extract the metadata parts like total_records, records_count to variables.
Then remove those metadata from the JObject, so that the key values becomes homogeneous.
Now Json.net will be easily able to serialize JObject to Response object, as key values are homogenous.
Assign the metadata extracted previously to the properties of Response object
public void Deserialize()
{
var json = #"{
'response':{
'6112':{
'ID':6112,
'Title':'Additional Photos',
},
'5982':{
'ID':5982,
'Title':'Bike Ride',
},
'total_records': '20',
'returned_count': 10,
'returned_records': '1-10',
}
}";
var responseObj = Newtonsoft.Json.JsonConvert.DeserializeObject<ss>(json, new ResponseConverter());
}
public class Response : Dictionary<string, Product>
{
public int total_records { get; set; }
public int returned_count { get; set; }
public string returned_records { get; set; }
}
public class Product
{
public string Id { get; set; }
public string Title { get; set; }
}
public class ss
{
public Response Response { get; set; }
}
public class ResponseConverter : Newtonsoft.Json.JsonConverter
{
private Response CreateResponse(Newtonsoft.Json.Linq.JObject jObject)
{
//preserve metadata values into variables
int total_records = jObject["total_records"].ToObject<int>();
var returned_records = jObject["returned_records"].ToObject<string>();
var returned_count = jObject["returned_count"].ToObject<int>();
//remove the unwanted keys
jObject.Remove("total_records");
jObject.Remove("returned_records");
jObject.Remove("returned_count");
//once, the metadata keys are removed, json.net will be able to deserialize without problem
var response = jObject.ToObject<Response>();
//Assign back the metadata to response object
response.total_records = total_records;
response.returned_count = returned_count;
response.returned_records = returned_records;
//.. now person can be accessed like response['6112'], and
// metadata can be accessed like response.total_records
return response;
}
public override bool CanConvert(Type objectType)
{
return objectType == typeof(Response);
}
public override object ReadJson(Newtonsoft.Json.JsonReader reader, Type objectType, object existingValue, Newtonsoft.Json.JsonSerializer serializer)
{
var jObject = Newtonsoft.Json.Linq.JObject.Load(reader);
Response target = CreateResponse(jObject);
serializer.Populate(jObject.CreateReader(), target);
return target;
}
public override void WriteJson(Newtonsoft.Json.JsonWriter writer, object value, Newtonsoft.Json.JsonSerializer serializer)
{
throw new NotImplementedException();
}
}
In my opinion this is how the JSON file should look like:
{
"response": {
"5982": {
"ID": 5982,
"Title": "BikeRide"
},
"6112": {
"ID": 6112,
"Title": "AdditionalPhotos"
},
"total_records": "20",
"returned_count": 10,
"returned_records": "1-10"
}
}
and this is how the class should look like
public class __invalid_type__5982
{
public int ID { get; set; }
public string Title { get; set; }
}
public class __invalid_type__6112
{
public int ID { get; set; }
public string Title { get; set; }
}
public class Response
{
public __invalid_type__5982 __invalid_name__5982 { get; set; }
public __invalid_type__6112 __invalid_name__6112 { get; set; }
public string total_records { get; set; }
public int returned_count { get; set; }
public string returned_records { get; set; }
}
public class RootObject
{
public Response response { get; set; }
}
I am using Newtonsoft.Json .Net for 4.0 for this project
Parent class:
public class CLiveThing
{
private object lawk = new object();
public Action<double> hp_cur_changed;
public Action<double> hp_max_changed;
public double hp_max { get; private set; }
public double hp_cur { get; private set; }
public void change_hp_max(double val)
{
lock (lawk)
{
hp_max += val;
if (hp_max_changed != null)
hp_max_changed(hp_max);
}
}
public void change_hp_cur(double val)
{
lock (lawk)
{
hp_cur += val;
if (hp_cur_changed != null)
hp_cur_changed(hp_cur);
}
}
}
Child class:
public class CPlayer : CLiveThing
{
public int id { get; private set; }
public CPlayer(int id)
{
this.id = id;
}
/*
* Network
*/
public string Serialize()
{
return Newtonsoft.Json.JsonConvert.SerializeObject(this);
}
public static CPlayer Deserialize(string val)
{
return Newtonsoft.Json.JsonConvert.DeserializeObject<CPlayer>(val);
}
}
Server (uses Players.CPlayers to manage all players with a generic collection)
Players.CPlayers.Serialize()
Players.CPlayers.Serialize serializes all players in the server's memory, one per line
Like so:
public static string Serialize()
{
players_lock.AcquireReaderLock(Timeout.Infinite);
string str = "";
foreach (CPlayer player in players.Values)
{
str += player.Serialize();
str += Environment.NewLine;
}
players_lock.ReleaseReaderLock();
return str;
}
Client
I put a break line in the Players.CPlayers.Deserialize loop, which reverses what the server did.
foreach (string line in split)
{
if (line.Length > 0)
{
CPlayer player = CPlayer.Deserialize(line);
addOrReplace(player.id, player);
}
}
Here's an example of one line:
What goes in:
"{\"hp_cur_changed\":null,\"hp_max_changed\":null,\"id\":1,\"hp_max\":100.0,\"hp_cur\":100.0}"
What comes out of CPlayer.Deserialize():
It only Deserialized the ID and ignored the properties in the parent class. Which is weird because the server-side did Serialize it properly. Can anyone tell me how to fix this?
I was not able to find an official reference why it's working like this but there are at least two way to solve your problem:
Declare your base class property setters as public
public double hp_cur { get; set; }
public double hp_max { get; set; }
Or annotate them with the JsonProperty attribute:
[JsonProperty]
public double hp_max { get; private set; }
[JsonProperty]
public double hp_cur { get; private set; }