I have a cache service that holds multiple Price objects which are updated as new price deltas arrive, sometimes multiple times a second.
Each object holds it's various prices in a collection assigned to an ID. If someone subscribes to a particular price I need to serialize the latest price object into JSON each time a new price arrives in order to send it over RMQ. The problem I am having is that in some cases I receive the following error message while serializing because a new price has arrived and updated the collection on the object during the serialization of the previous.
"Collection was modified; enumeration operation may not execute."
I've tried various ways of serializing the object (it needs to be as fast as possible) but I still get the same issue.
What would be the best and most efficient way of solving this so I can serialize even if the object changes.
The simplified objects are:
//This is the collection on an object that holds the prices which are being updated
public ConcurrentDictionary<Id, Prices> Asset{ get; set; }
//Class that holds the ever updating prices
[Serializable]
public class Prices
{
public Prices()
{
Prices1 = new List<PriceVolume>();
Prices2 = new List<PriceVolume>();
}
}
thanks in advance!
Instead of serializing the actual object you pull from the concurrent dictionary you should create a deep copy of it and serialize the copy. unfortunately you will still have to put the code to get the copy inside a mutex. The ConcurrentDic is only protecting you from having an item changed or deleted while you are retrieving it, it does not protect the object from manipulation after you have retrieved your reference to it.
You could probably benefit from locking the element while you are serializing it.
Locking prevents other threads from modifying the element while you are inside the lock.
This will make the operations trying to change the prices wait until you are done serializing it to modify them.
[Serializable]
public class Prices
{
public string Serialize()
{
lock (this)
{
// logic for serilization here
}
}
}
Just a thought, but how about looking into Serialization Callbacks (some refer to this as Serialization Hooks) and Implementing the ISerializable Interface. You seem to need more fine-grained control over the serialization of your object. Have a look at this link:
Custom Serialization
Look at the following
OnDeserializingAttribute (Before deserialization)
OnDeserializedAttribute (After deserialization)
OnSerializingAttribute (Before serialization)
OnSerializedAttribute (After serialization)
Also have a look at this link:
Version Tolerant Serialization
You can consider if the object warrants the use of the OptionalFieldAttribute or NonSerializedAttribute on certain fields to control if they need to be serialized or optionally serialized. Just be wary of the use of the NonSerializedAttribute. Have a look at the best practices mentioned in the article (reproduced here for reference):
To ensure proper versioning behavior, follow these rules when modifying a type from version to version:
Never remove a serialized field
Never apply the NonSerializedAttribute attribute to a field if the attribute was not applied to the field in the previous version
Never change the name or the type of a serialized field
When adding a new serialized field, apply the OptionalFieldAttribute attribute
When removing a NonSerializedAttribute attribute from a field (that was not serializable in a previous version), apply the OptionalFieldAttribute attribute
For all optional fields, set meaningful defaults using the serialization callbacks unless 0 or null as defaults are acceptable
To ensure that a type will be compatible with future serialization engines, follow these guidelines:
Always set the VersionAdded property on the OptionalFieldAttribute attribute correctly
Avoid branched versioning
Related
I have a set of objects that contain fields & properties that need to be inspectable in the output of serialization but not read back in when deserialized.
This is purely for debugging/confirmation purposes. We are creating hundreds of files and I want to spot check that serialization is occurring correctly by adding supplementary information. I do not want this supplementary information to be read in during deserialization - it's impossible to do so in fact.
I also need to do this with equal facility across different serialization formats, so we can assess which one is working best. I have a generic serialization approach where the desired format is passed in as an argument, so don't want anything too messy or intricate for each different format.
I've hunted around and found various things on related topics - mostly to do with the opposite: not writing certain fields during serialization. What's out there seems to be quite complicated and at times hacky.
Is it possible to serialize an object differently to deserializing it using Json.Net?
JsonConvert .NET Serialize/Deserialize Read Only
Serialize Property, but Do Not Deserialize Property in Json.Net
Also it appears any approach is inconsistent between serialization formats. i.e. unlike the [*Ignore] attributes, there are no [*SerializeOnly] attributes (where * = JSON, XML, YAML).
Is there an easy way to do this across these serialization formats? Is there a single family of attributes that can help? Or is it idiosyncratic and hacky in each case?
I have tested and applied this only to XML serialization, but it works for me:
When I want a property to be serialized, but not read back, I just declare an empty setter.
public String VersionOfApplicationThatHasWrittenThisFile
{
get
{
return "1.0";
}
set
{
// Leave empty
}
}
I'm receiving messages over a network using JSON.NET. The message format is somewhat dynamic, in that the messages will be represented by many different classes, each inheriting from a parent message. For example:
{
MessageName: "MessageType1",
Data1: 124,
Data2: "Something"
}
{
MessageName: "MessageType2",
OtherData: "Some data",
MoreData: "Even more",
ANumber: 25
}
The problem I'm having is that in JSON.NET, I have no idea how to figure out the name of the class (MessageType1/MessageType2/etc) in order to deserialize it into an instance of the class without deserializing it twice. There's a few options I've considered; the one I'm currently using is to use a container class containing the message name and the actual json message serialized to string, but this seems wasteful.
Another method I've considered is deserializing into a string/string dictionary and then performing the population of the class on my own, which seems messy and unnecessary considering JSON.NET can do that for me... as long as I know the class first.
I'm really hoping there's an easy way to have JSON.NET figure out a class name by examining the MessageName property and then continue to populate a class after examining that one property.
Thanks for the help!
JSON can deserialize into a well known class only. You need to specify the data layout (i.e. the class/type)
There are two alternatives:
1.) go one level deeper. Use the JSON Token parser to read the tokens from your JSON stream and act based on the tokens you find.
2.) as you suggested: Use a class layout flexible enough to hold all your possible variations like a key/value dictionary.
Lets say my c# model updated while correspondent collection still contains old documents, I want old and new documents to coexist in the collection, while using only new version of c# model to read them. I wish no inheritance is used if possible. So I wonder which of this issues are solvable and how:
there is a new property in c# model which does not present in database. I think it never should be an issue, Mongo knows nothing about it, and it will be initialized with default value. The only issue here is to initialize it with particular value for all old documents, anybody knows how?
one of property has gone from model. I want MongoDb to find out there is no more property in c# class to map the field of old document to, and to ignore it instead of crashing. This scenario probably sounds a bit strange as it would mean some garbage left in database, but anyway, is the behavior possible to implement/configure?
type if changed, new type is convertible to old one, like integer->string. Is there any way to configure mapping for old docs?
I can consider using inheritance for second case if it is not solvable otherwise
Most of the answers to your questions are found here.
BsonDefaultValue("abc") attribute on properties to handle values not present in the database, and to give them a default value upon deserialization
BsonIgnoreExtraElements attribute on the class to ignore extra elements found during deserialization (to avoid the exception)
A custom serializer is required to handle if the type of a member is changed, or you need to write an upgrade script to fix the data. It would probably be easier to leave the int on load, and save to a string as needed. (That will mean that you'll need a new property name for the string version of the property.)
I'm wondering if there's a way in which I can create a tree/view of a serialised object graph, and whether anyone has any pointers? EDIT The aim being that should we encounter a de-serialization problem for some reason, that we can actually view/produce a report on the serialized data to help us identify the cause of the problem before having to debug the code. Additionally I want to extend this in the future to take two streams (version 1, version 2) and highlight differences between the two of them to help ensure that we don't accidently remove interesting information during code changes. /EDIT
Traditionally we've used Soap or XML serialization, but these are becoming too restricted for our needs, and Binary serialization would generally do all that we need. The reason that this hasn't been adopted, is because it's much harder to view the serialized contents to help fix upgrade issues etc.
So I've started looking into trying to create a view on the serialized information. I can do this from an ISerializable constructor to a certain extent :
public A(SerializationInfo info, StreamingContext context)
{}
Given the serialization info I can reflect the m_data member and see the actual serialized contents. The problem with this approach is
It will only display a branch from the tree, I want to display the entire tree from the root and it's not really possible to do from this position.
It's not a convenient place to interrogate the information, I'd like to pass a stream to a class and do the work there.
I've seen the ObjectManager class but this works on an existing object graph, whereas I need to be able to work from the stream of data. I've looked through the BinaryFormatted which uses an ObjectReader and a __BinaryParser, hooking into the ObjectManager (which I think will then have the entire contents, just maybe in a flat list), but to replicate this or invoke it all via reflection (2 of those 3 classes are internal) seems like quite a lot of work, so I'm wondering if there's a better approach.
You could put a List<Child class> in every parent class (Even if there the same)
and when you create a child you immediately place it in that list or better yet declare it whilst adding it the list
For instance
ListName.Add(new Child(Constructer args));
Using this you would serialize them as one file which contains the hierarchy of the objects and the objects themselves.
If the parent and child classes are the same there is no reason why you cannot have dynamic and multi leveled hierarchy.
In order to achieve what you describe you would have to deserialize whole object graph from stream without knowing a type from which it was serialized. But this is not possible, because serializer doesn't store such information.
AFAIK it works in a following way. Suppose you have a couple of types:
class A { bool p1 }
class B { string p1; string p2; A p3}
// instantiate them:
var b = new B { p1 = "ppp1", p2 = "ppp2", p3 = new A { p1 = true} };
When serializer is writing this object, it starts walking object graph in some particular order (I assume in alphabetic order) and write object type and then it's contents. So your binary stream will like this:
[B:[string:ppp1][string:ppp2][A:[bool:true]]]
You see, here there are only values and their types. But order is implicit - like it is written.
So, if you change your object B, to suppose
class B { A p1; string p3; string p3;}
Serialzer will fail, because it will try to assing instance of string (which was serialized first) to pointer to A. You may try to reverse engineer how binary serialization works, then you may be able to create a dynamic tree of serialized objects. But this will require considerable effort.
For this purpose I would create class similar to this:
class Node
{
public string NodeType;
public List<Node> Children;
public object NodeValue;
}
Then while you will be reading from stream, you can create those nodes, and recreate whole serialized tree and analyze it.
I have a class MyClass containing a private List<MySecondClass> myList. The list is exposed through a getter as follows:
public IEnumerable<MySecondClass> MyList
{
get { return myList.Select(a => a); }
}
The list is modified through public AddItem(MySecondClass itemToAdd) and ClearItems() methods. I believe that this is a properly encapsulated list.
The problem lies in that I need to pass an object of type MyClass (containing myList) via SOAP to a web service, which fills myList (using the AddItem() method), and then returns the object.
However, when the webmethod returns the class, after serialization myList is empty. I am suspecting this is because I do not have a setter for myList, which is causing the list not to be set during serialization.
Is this a good assumption, or am I way off? If the problem is what I think it is, is there a way to allow for the list to be successfully passed from the webmethod without breaking encapsulation (I do not want to expose a generic list)?
Without trying this directly myself, I believe that you could definitely be correct.
serialization in .NET makes utilizing read only properties a fun circus.because the .net default serialization process requires a setter property in order to "deserialize" the object. Without a setter property the serialization piece will still work allowing you to serialize to a drive or across the network. But, it is the deserialization process that will fail which could definitely be why your collection is empty. Im just amazed it doesn't error out to be honest.
Have you tried to add a simple setter just to verify that this is in fact the issue just so that we know with 100% certainty that this is the problem before working to solve it.
While I never really solved the initial problem, what I did do to get it working was simplify the data that was being passed to the web method. Instead of passing an entire object to the web method, I instead passed a unique identifier. The webmethod then returns the list I need, and I handle actually adding the items in this list to the object client-side.
The XML Serializer used by ASMX services only serializes public read/write properties.