Develop Reference

Why does deserialization append to an initialized `List<T>` instead of overwriting it?

I have a configuration-related data model which is required to have default values for all properties to ensure proper execution of consuming operations. Take for example this very simple model:
public class Sample {
public List<int> Integers { get; set; }
public Sample() =>
Integers = new List<int> { 1 };
}
This model is expected to contain at least one value so we specify a default processing value based on predefined requirements. However, the value doesn't have to be 1 specifically and if a client specifies a different value or many different values, our process should respect their input. Assume the client specifies the following json configuration to be deserialized into the model:
{
"integers": [ 2, 3 ]
}
During runtime, we load the configuration directly, but let's use a local string variable for the sake of this question:
using Newtonsoft.Json
...
string configuration = "{ \"integers\": [ 2, 3 ] }";
var sample = JsonConvert.DeserializeObject<Sample>(configuration);
Console.WriteLine(string.Join(",", sample.Integers));
The above code snippet should produce an output of:
1,2,3
As you can see in my screenshot below, that is the case:
However, my question is why... Why does the deserialization process append to the collection instead of overwriting it?

You can point how to deserialize the json
var serializerSettings = new JsonSerializerSettings {
ObjectCreationHandling = ObjectCreationHandling.Replace};
var sample = JsonConvert.DeserializeObject<Sample>(configuration,serializerSettings);
Console.WriteLine(string.Join(",", sample.Integers)); // [2,3]
by default it is auto, and if there is something it tries to add.

C#'s version of JS's JSON [duplicate]

This question already has answers here:
Serialize and Deserialize Json and Json Array in Unity
(9 answers)
Closed 3 years ago.
I recently got into C# using Unity. I previously worked with JavaScript and know that they defiantly have their differences, but still have some similarities. I have used JSON with JS and it works great. Now with Unity, I want to store data of upcoming "stages" in JSON in an infinite runner game. But from my experience, JSON does not work nearly as well with C# as it does with JS. I have looked at Unity's JSON Utility but haven't figured out if it's possible to simply have a string and then convert it into an object which you could access like object.item.array[0].item which is how you'd do it in JS. Another thing that I looked at was this but as a novice to C#, I couldn't make heads or tails of it. So does C# have something like JSON, but its more integrated? I've used C# lists, can you get 3D lists with items and not just arrays? I know that they are very different languages, and what works well on one, might not on another.

I think closest to what you describe in your questions and the comments as
simply convert a JSON string into a JSONObject
would maybe be the good old SimpleJSON. Simply copy the SimpleJSON.cs and depending on your needs maybe SimpleJSONUnity.cs(provides some extensions for directly parsing to and from Vector2, Vector3, Vector4, Quaternion, Rect, RectOffset and Matrix4x4) somewhere into your Assets folder.
Then given the example json
{
"version": "1.0",
"data": {
"sampleArray": [
"string value",
5,
{
"name": "sub object"
}
]
}
}
you can simply access single fields like
using SimpleJSON;
...
var jsonObject = JSON.Parse(the_JSON_string);
string versionString = jsonObject["version"].Value; // versionString will be a string containing "1.0"
float versionNumber = jsonObject["version"].AsFloat; // versionNumber will be a float containing 1.0
string val = jsonObject["data"]["sampleArray"][0]; // val contains "string value"
string name = jsonObject["data"]["sampleArray"][2]["name"]; // name will be a string containing "sub object"
...
Using this you don't have to re-create the entire c# class representation of the JSON data which might sometimes be a huge overhead if you just want to access a single value from the JSON string.
However if you want to serialze and deserialize entire data structures you won't get happy using SimpleJSON. Given the example above this is how you would use Unity's JsonUtility
Create the c# class representation of the data yu want to store. In this case e.g. something like
[Serializable]
public class RootObject
{
public string version = "";
public Data data = new Data();
}
[Serializable]
public class Data
{
public List<object> sampleArray = new List<object>();
}
[Serializeable]
public class SubData
{
public string name = "";
}
Then fill it with values and parse it to JSON like
var jsonObject = new RootObject()
{
version = "1.0",
data = new Data()
{
sampleArray = new List<object>()
{
"string value",
5,
new SubData(){ name = "sub object" }
}
}
};
var jsonString = JsonUtility.ToJson(jsonObject);
And to convert it back to c# either if jsonObject was not created yet
jsonObject = JsonUtility.FromJson<RootObject>(jsonString);
otherwise
JsonUtility.FromJsonOverwrite(jsonString, jsonObject);

JSON is just how objects are represented in JavaScript. While C# can use JSON, you'll probably have a much easier time defining your own custom classes. Using classes, you can define all the properties you'll need, string them together into a list, etc.. Hope this helps.

Parsing JSON with 2D arrays in Unity C#

I am working with Unity and Python and sending data over UDP to my Unity client. I am sending a JSON which reaches my Unity application successfully. I can't figure out how to properly parse this JSON. I have tried :
JsonUtility.FromJson<T>(...)
but the result wasn't correct. I implemented a [System.Serializable] class which contained data types
List<List<int>> landmarks;
List<double> eulerAngles;
List<List<double>> boxPoints;
And initialized these in my constructor using
landmarks = new List<List<int>>();
...
Every time I try to print the result, all my variables are empty.
My JSON looks like this -
{
"eulerAngles": [6.2255, -15.8976, -0.5881],
"boxPoints": [
[194.05965, 194.15782],
[182.35829, 189.05042],
[6 more...]
],
"landmarks": [
[196, 225],
[197, 242],
[199, 258],
[65 more..]
]
}
Any help appreciated.

Your JSON is still strangly formatted and I don't understand your data structure.
Why are you storing everything in List<int[]> while the values you get are clearly float (or double) values?
If something it should probably be
[Serializable]
public class JsonData
{
// here maybe int is ok at least from your data it seems so
public List<int[]> landmarks = new List<int[]>();
// for those you clearly want floats
public List<float[]> eulerAngles = new List<float[]>();
public List<float[]> boxPoints = new List<float[]>();
}
Have a look at this JSON to c# converter!
BUT anyway Unity's JsonUtility seems not to support the deserialization of nested arrays and lists! I guess the reason is that while other primitive data types have a default value that is different from null, List and arrays are null be default and have to be initialized before filling them. JsonUtility seems to not support that. I think it is related to
Note that while it is possible to pass primitive types to this method, the results may not be what you expect; instead of serializing them directly, the method will attempt to serialize their public instance fields, producing an empty object as a result. Similarly, passing an array to this method will not produce a JSON array containing each element, but an object containing the public fields of the array object itself (of which there are none). To serialize the actual content of an array or primitive type, it is necessary to wrap it in a class or struct. (JsonUtility.ToJson)
and
a field of type T will have value default(T) - it will not be given any value specified as a field initializer, as the constructor for the object is not executed during deserialization (JsonUtility.FromJson)
However, why even take them as arrays where every of your arrays only has one element anyway when it seems (and your names and title say) you rather want to store tuples of allways 2 and 3 numbers?
In case you can change the JSON format it seems like you rather would want something like
[Serializable]
public class JsonData
{
// here maybe int is ok at least from your data it seems so
public List<Vector2Int> landmarks = new List<Vector2Int>();
// for those you clearly want floats
public List<Vector3> eulerAngles = new List<Vector3>();
public List<Vector2> boxPoints = new List<Vector2>();
}
And your JSON should than rather look like
{
"eulerAngles": [
{"x":6.2255, "y":-15.8976, "z":-0.5881},
...
],
"boxPoints": [
{"x":194.05965, "y":194.15782},
{"x":182.35829, "y":189.05042},
...
],
"landmarks": [
{"x":196, "y":225},
{"x":197, "y":242},
{"x":199, "y":258},
...
]
}
The simplest way to see how your JSON should actually look like would be to take your JsonData class and serialize it once so you see what Unity expects as JSON.
var testData = new JsonData();
// fill it with example values
testData.eulerAngles = new List<Vector3>()
{
new Vector3(6.2255f, -15.8976f, -0.5881f)
};
testData.boxPoints = new List<Vector2>()
{
new Vector2(194.05965f, 194.15782f),
new Vector2(182.35829f, 189.05042f)
};
testData.landmarks = new List<Vector2Int>()
{
new Vector2Int(196, 225),
new Vector2Int(197, 242),
new Vector2Int(199, 258)
}
var result = JsonUtility.ToJson(testData);
If changing the JSON format is not an option you could instead have a look at SimpleJSON which doesn't deserialize directly into a class instance but rather to a JSONNode which can contain nested lists and arrays e.g.
var jsonData = JSON.Parse(yourJson);
var firstEuerAngles = jsonData["eulerAngles"][0];

JSON.Net, AnonymousTypes and Dashes [duplicate]

This question already has answers here:
How can I parse a JSON string that would cause illegal C# identifiers?
(3 answers)
Can you have a property name containing a dash
(3 answers)
Closed 7 years ago.
In C#, a valid variable name cannot contain dashes. But in Json, all property names are based off of strings, so what would be considered an invalid character for a C# variable name, could be considered valid in Json.
My question is, how does JSON.Net handle having a Dash, or other invalid data inside of a Property name when attempting to deserialize to an Anonymous Type, and more importantly, what do you replace invalid characters with in the Anonymous Type to capture it.
If example data is required, i can provide it but quite frankly just add a Dash (-) to a Json Property name, and you've got my situation in a nutshell.
P.S: I cannot change the Json itself, because it is being consumed from an API.

You can use a ContractResolver to manipulate how JSON.Net maps C# property names to JSON names.
For your example this code does that:
class DashContractResolver : Newtonsoft.Json.Serialization.DefaultContractResolver
{
protected override string ResolvePropertyName(string propertyName)
{
// Count capital letters
int upperCount = propertyName.Skip(1).Count(x => char.IsUpper(x));
// Create character array for new name
char[] newName = new char[propertyName.Length + upperCount];
// Copy over the first character
newName[0] = char.ToLowerInvariant(propertyName[0]);
// Fill the character, and an extra dash for every upper letter
int iChar = 1;
for (int iProperty = 1; iProperty < propertyName.Length; iProperty++)
{
if (char.IsUpper(propertyName[iProperty]))
{
// Insert dash and then lower-cased char
newName[iChar++] = '-';
newName[iChar++] = char.ToLowerInvariant(propertyName[iProperty]);
}
else
newName[iChar++] = propertyName[iProperty];
}
return new string(newName, 0, iChar);
}
}
class Program
{
static void Main(string[] args)
{
string json = #"{""text-example"":""hello""}";
var anonymous = new { textExample = "" };
var obj = JsonConvert.DeserializeAnonymousType(json, anonymous,
new JsonSerializerSettings
{
ContractResolver = new DashContractResolver()
});
}
}
It converts UpperCamelCase and lowerCamelCase to lower-dash-case. Therefore mapping to your JSON input.
This overload of DeserializeAnonymousType has not always been available, and isn't available in the version released with Visual Studio 2013. The current (stable) NuGet package has this overload in it.

I'd suggest looking at the Dynamic rather than Anonymous UI for Json.Net, which can deserialise your data to ExpandoObject, which is a dynamic type that behaves like a dictionary - i.e. similar to a JavaScript object. This will then mean the range of allowed property names goes up, because they become dictionary keys rather than .Net properties.
Kind of like: Deserialize a property as an ExpandoObject using JSON.NET

Using Protobuf-net, I suddenly got an exception about an unknown wire-type

(this is a re-post of a question that I saw in my RSS, but which was deleted by the OP. I've re-added it because I've seen this question asked several times in different places; wiki for "good form")
Suddenly, I receive a ProtoException when deserializing and the message is: unknown wire-type 6
What is a wire-type?
What are the different wire-type values and their description?
I suspect a field is causing the problem, how to debug this?

First thing to check:
IS THE INPUT DATA PROTOBUF DATA? If you try and parse another format (json, xml, csv, binary-formatter), or simply broken data (an "internal server error" html placeholder text page, for example), then it won't work.
What is a wire-type?
It is a 3-bit flag that tells it (in broad terms; it is only 3 bits after all) what the next data looks like.
Each field in protocol buffers is prefixed by a header that tells it which field (number) it represents,
and what type of data is coming next; this "what type of data" is essential to support the case where
unanticipated data is in the stream (for example, you've added fields to the data-type at one end), as
it lets the serializer know how to read past that data (or store it for round-trip if required).
What are the different wire-type values and their description?
0: variant-length integer (up to 64 bits) - base-128 encoded with the MSB indicating continuation (used as the default for integer types, including enums)
1: 64-bit - 8 bytes of data (used for double, or electively for long/ulong)
2: length-prefixed - first read an integer using variant-length encoding; this tells you how many bytes of data follow (used for strings, byte[], "packed" arrays, and as the default for child objects properties / lists)
3: "start group" - an alternative mechanism for encoding child objects that uses start/end tags - largely deprecated by Google, it is more expensive to skip an entire child-object field since you can't just "seek" past an unexpected object
4: "end group" - twinned with 3
5: 32-bit - 4 bytes of data (used for float, or electively for int/uint and other small integer types)
I suspect a field is causing the problem, how to debug this?
Are you serializing to a file? The most likely cause (in my experience) is that you have overwritten an existing file, but have not truncated it; i.e. it was 200 bytes; you've re-written it, but with only 182 bytes. There are now 18 bytes of garbage on the end of your stream that is tripping it up. Files must be truncated when re-writing protocol buffers. You can do this with FileMode:
using(var file = new FileStream(path, FileMode.Truncate)) {
// write
}
or alternatively by SetLength after writing your data:
file.SetLength(file.Position);
Other possible cause
You are (accidentally) deserializing a stream into a different type than what was serialized. It's worth double-checking both sides of the conversation to ensure this is not happening.

Since the stack trace references this StackOverflow question, I thought I'd point out that you can also receive this exception if you (accidentally) deserialize a stream into a different type than what was serialized. So it's worth double-checking both sides of the conversation to ensure this is not happening.

This can also be caused by an attempt to write more than one protobuf message to a single stream. The solution is to use SerializeWithLengthPrefix and DeserializeWithLengthPrefix.
Why this happens:
The protobuf specification supports a fairly small number of wire-types (the binary storage formats) and data-types (the .NET etc data-types). Additionally, this is not 1:1, nor is is 1:many or many:1 - a single wire-type can be used for multiple data-types, and a single data-type can be encoded via any of multiple wire-types. As a consequence, you cannot fully understand a protobuf fragment unless you already know the scema, so you know how to interpret each value. When you are, say, reading an Int32 data-type, the supported wire-types might be "varint", "fixed32" and "fixed64", where-as when reading a String data-type, the only supported wire-type is "string".
If there is no compatible map between the data-type and wire-type, then the data cannot be read, and this error is raised.
Now let's look at why this occurs in the scenario here:
[ProtoContract]
public class Data1
{
[ProtoMember(1, IsRequired=true)]
public int A { get; set; }
}
[ProtoContract]
public class Data2
{
[ProtoMember(1, IsRequired = true)]
public string B { get; set; }
}
class Program
{
static void Main(string[] args)
{
var d1 = new Data1 { A = 1};
var d2 = new Data2 { B = "Hello" };
var ms = new MemoryStream();
Serializer.Serialize(ms, d1);
Serializer.Serialize(ms, d2);
ms.Position = 0;
var d3 = Serializer.Deserialize<Data1>(ms); // This will fail
var d4 = Serializer.Deserialize<Data2>(ms);
Console.WriteLine("{0} {1}", d3, d4);
}
}
In the above, two messages are written directly after each-other. The complication is: protobuf is an appendable format, with append meaning "merge". A protobuf message does not know its own length, so the default way of reading a message is: read until EOF. However, here we have appended two different types. If we read this back, it does not know when we have finished reading the first message, so it keeps reading. When it gets to data from the second message, we find ourselves reading a "string" wire-type, but we are still trying to populate a Data1 instance, for which member 1 is an Int32. There is no map between "string" and Int32, so it explodes.
The *WithLengthPrefix methods allow the serializer to know where each message finishes; so, if we serialize a Data1 and Data2 using the *WithLengthPrefix, then deserialize a Data1 and a Data2 using the *WithLengthPrefix methods, then it correctly splits the incoming data between the two instances, only reading the right value into the right object.
Additionally, when storing heterogeneous data like this, you might want to additionally assign (via *WithLengthPrefix) a different field-number to each class; this provides greater visibility of which type is being deserialized. There is also a method in Serializer.NonGeneric which can then be used to deserialize the data without needing to know in advance what we are deserializing:
// Data1 is "1", Data2 is "2"
Serializer.SerializeWithLengthPrefix(ms, d1, PrefixStyle.Base128, 1);
Serializer.SerializeWithLengthPrefix(ms, d2, PrefixStyle.Base128, 2);
ms.Position = 0;
var lookup = new Dictionary<int,Type> { {1, typeof(Data1)}, {2,typeof(Data2)}};
object obj;
while (Serializer.NonGeneric.TryDeserializeWithLengthPrefix(ms,
PrefixStyle.Base128, fieldNum => lookup[fieldNum], out obj))
{
Console.WriteLine(obj); // writes Data1 on the first iteration,
// and Data2 on the second iteration
}

Previous answers already explain the problem better than I can. I just want to add an even simpler way to reproduce the exception.
This error will also occur simply if the type of a serialized ProtoMember is different from the expected type during deserialization.
For instance if the client sends the following message:
public class DummyRequest
{
[ProtoMember(1)]
public int Foo{ get; set; }
}
But what the server deserializes the message into is the following class:
public class DummyRequest
{
[ProtoMember(1)]
public string Foo{ get; set; }
}
Then this will result in the for this case slightly misleading error message
ProtoBuf.ProtoException: Invalid wire-type; this usually means you have over-written a file without truncating or setting the length
It will even occur if the property name changed. Let's say the client sent the following instead:
public class DummyRequest
{
[ProtoMember(1)]
public int Bar{ get; set; }
}
This will still cause the server to deserialize the int Bar to string Foo which causes the same ProtoBuf.ProtoException.
I hope this helps somebody debugging their application.

Also check the obvious that all your subclasses have [ProtoContract] attribute. Sometimes you can miss it when you have rich DTO.

I've seen this issue when using the improper Encoding type to convert the bytes in and out of strings.
Need to use Encoding.Default and not Encoding.UTF8.
using (var ms = new MemoryStream())
{
Serializer.Serialize(ms, obj);
var bytes = ms.ToArray();
str = Encoding.Default.GetString(bytes);
}

If you are using SerializeWithLengthPrefix, please mind that casting instance to object type breaks the deserialization code and causes ProtoBuf.ProtoException : Invalid wire-type.
using (var ms = new MemoryStream())
{
var msg = new Message();
Serializer.SerializeWithLengthPrefix(ms, (object)msg, PrefixStyle.Base128); // Casting msg to object breaks the deserialization code.
ms.Position = 0;
Serializer.DeserializeWithLengthPrefix<Message>(ms, PrefixStyle.Base128)
}

This happened in my case because I had something like this:
var ms = new MemoryStream();
Serializer.Serialize(ms, batch);
_queue.Add(Convert.ToBase64String(ms.ToArray()));
So basically I was putting a base64 into a queue and then, on the consumer side I had:
var stream = new MemoryStream(Encoding.UTF8.GetBytes(myQueueItem));
var batch = Serializer.Deserialize<List<EventData>>(stream);
So though the type of each myQueueItem was correct, I forgot that I converted a string. The solution was to convert it once more:
var bytes = Convert.FromBase64String(myQueueItem);
var stream = new MemoryStream(bytes);
var batch = Serializer.Deserialize<List<EventData>>(stream);