Develop Reference

MongoDB Changestream Filters

Currently using the MongoDB.Driver v2.11.0 in a .Net Core 3.1 application. Trying to read values from a changestream and have been following the documentation here:
https://mongodb.github.io/mongo-csharp-driver/2.11/reference/driver/change_streams/
My problem is i'm trying to only get objects with a specific type and updated field, but cannot get both filters working. I can get the changestream to grab only objects that have a type of Z from the fulldocument, but cannot get any updateDescription.updatedFields filters to work properly.
Object
public class Abc
{
[BsonElement("d")]
public D D{ get; set; }
[BsonElement("e")]
public E E{ get; set; }
}
public class D
{
[BsonElement("type")]
public string Type{ get; set; }
}
public class E
{
[BsonElement("e")]
public F F{ get; set; }
}
public class F
{
[BsonElement("status")]
public string Status { get; set; }
}
So after connecting to mongo and getting the collection here is the code to setup the change stream.
protected ChangeStreamOptions _changeStreamOptions => new ChangeStreamOptions { FullDocument = ChangeStreamFullDocumentOption.UpdateLookup };
public IChangeStreamCursor<ChangeStreamDocument<Abc>> GetChangeStreamCursor()
{
return _collection.Watch(ConfigurePipeline(), _changeStreamOptions);
}
private PipelineDefinition<ChangeStreamDocument<Abc>, ChangeStreamDocument<Abc>> ConfigurePipeline()
{
List<IPipelineStageDefinition> pipeline = new List<IPipelineStageDefinition>();
pipeline.Add(PipelineStageDefinitionBuilder.Match(ConfigureFilters()));
return pipeline;
}
private FilterDefinition<ChangeStreamDocument<Abc>> ConfigureFilters()
{
var builder = Builders<ChangeStreamDocument<Abc>>.Filter;
//here is where I build the filters and having the issues.
//if its just based on the object type It works.
return builder.Eq("fullDocument.d.type", "z");
}
This works and have no problems getting only objects that had a type of Z that were updated.
If I try to then also add a filter to only return objects of the Z and that had the field Status to have been updated to a specific type.
Here is what I've tried for that:
builder.Eq("fullDocument.d.type", "z") & builder.AnyEq(x => x.UpdateDescription.UpdatedFields.Values, "Action");
builder.Eq("updateDescription.updatedFields.e.f.status", "Action");
builder.Eq("e.f.status", "Action");
The application will run with this but just never picks up any changes.
Also only tried looking at the updated fields since the UpdateDescription.UpdatedFields is an array.
builder.ElemMatch(x => x.UpdateDescription.UpdatedFields, x => x.Name == "e.f.status" && x.Value == "Action")
This fails in the program main with this error
Unable to determine the serialization information for x => x.UpdateDescription.UpdatedFields.'
Simply put I need to create a change stream and only get back objects that have a specific property type in the fulldocument and where another property was updated to a specific type.
Any help would be appreciated!
Edit Update:
If I change the types to be a BsonDocument I can pass this as the pipeline and it works:
So I got it working with a BsonObject and using this:
var filter = "{ $and: [ { operationType: 'update' }, " +
"{ 'fullDocument.d.type' : 'z'}" +
"{ 'updateDescription.updatedFields': { 'e.f.status': 'Action' } } ] }";
But I would like to keep it typed if possible.

So I figured out a solution that is working for me the way I described above.
private FilterDefinition<ChangeStreamDocument<Abc>> ConfigureFilters()
{
var builder = Builders<ChangeStreamDocument<Abc>>.Filter;
var filters = builder.And(builder.Eq("fullDocument.d.type", "z") & new BsonDocument("e.f.status", new BsonDocument("$eq", "Action")));
return filters;
}

c# parse json value, check if null

So I have this Web API that calls a service which in turn, returns a json string.
The string looks a bit like this:
{
"title": "Test",
"slug": "test",
"collection":{ },
"catalog_only":{ },
"configurator":null
}
I have cut this down considerably to make it easier to see my issue.
When I make my API call, I then use a Factory method to factorize the response which looks something like this:
/// <summary>
/// Used to create a product response from a JToken
/// </summary>
/// <param name="model">The JToken representing the product</param>
/// <returns>A ProductResponseViewModel</returns>
public ProductResponseViewModel Create(JToken model)
{
// Create our response
var response = new ProductResponseViewModel()
{
Title = model["title"].ToString(),
Slug = model["slug"].ToString()
};
// Get our configurator property
var configurator = model["configurator"];
// If the configurator is null
if (configurator == null)
throw new ArgumentNullException("Configurator");
// For each item in our configurator data
foreach (var item in (JObject)configurator["data"])
{
// Get our current option
var option = item.Value["option"].ToString().ToLower();
// Assign our configuration values
if (!response.IsConfigurable) response.IsConfigurable = (option == "configurable");
if (!response.IsDesignable) response.IsDesignable = (option == "designable");
if (!response.HasGraphics) response.HasGraphics = (option == "graphics");
if (!response.HasOptions) response.HasOptions = (option == "options");
if (!response.HasFonts) response.HasFonts = (option == "fonts");
}
// Return our Product response
return response;
}
}
Now, as you can see I am getting my configurator property and then checking it to see if it's null.
The json string shows the configurator as null, but when I put a breakpoint on the check, it actually shows it's value as {}.
My question is how can I get it to show the value (null) as opposed to this bracket response?

You're asking for the JToken associated with the configurator key. There is such a token - it's a null token.
You can check this with:
if (configurator.Type == JTokenType.Null)
So if you want to throw if either there's an explicit null token or configurator hasn't been specified at all, you could use:
if (configurator == null || configurator.Type == JTokenType.Null)

For those who use System.Text.Json in newer .Net versions (e.g. .Net 5):
var jsonDocument = JsonDocument.Parse("{ \"configurator\": null }");
var jsonElement = jsonDocument.RootElement.GetProperty("configurator");
if (jsonElement.ValueKind == JsonValueKind.Null)
{
Console.WriteLine("Property is null.");
}

public class Collection
{
}
public class CatalogOnly
{
}
public class RootObject
{
public string title { get; set; }
public string slug { get; set; }
public Collection collection { get; set; }
public CatalogOnly catalog_only { get; set; }
public object configurator { get; set; }
}
var k = JsonConvert.SerializeObject(new RootObject
{
catalog_only =new CatalogOnly(),
collection = new Collection(),
configurator =null,
slug="Test",
title="Test"
});
var t = JsonConvert.DeserializeObject<RootObject>(k).configurator;
Here configurator is null.

Force binary deserialization to fail when type modified

I'm looking for a non-intrusive way to enforce deserialization to fail under the following circumstances:
The type is not defined in a strongly named assembly.
BinaryFormatter is used.
Since serialized, the type has been modified (e.g. a property has been added).
Below is an illustration/repro of the problem in form of a failing NUnit test. I'm looking for a generic way to make this pass without modifying the Data class, preferably by just setting up the BinaryFormatter during serialization and/or deserialization. I also don't want to involve serialization surrogates, as this is likely to require specific knowledge for each affected type.
Can't find anything in the MSDN docs that helps me though.
[Serializable]
public class Data
{
public string S { get; set; }
}
public class DataSerializationTests
{
/// <summary>
/// This string contains a Base64 encoded serialized instance of the
/// original version of the Data class with no members:
/// [Serializable]
/// public class Data
/// { }
/// </summary>
private const string Base64EncodedEmptyDataVersion =
"AAEAAAD/////AQAAAAAAAAAMAgAAAEtTc2MuU3Rvcm0uRGF0YS5UZXN0cywgV"+
"mVyc2lvbj0xLjAuMC4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1YmxpY0tleVRva2"+
"VuPW51bGwFAQAAABlTc2MuU3Rvcm0uRGF0YS5UZXN0cy5EYXRhAAAAAAIAAAAL";
[Test]
public void Deserialize_FromOriginalEmptyVersionFails()
{
var binaryFormatter = new BinaryFormatter();
var memoryStream = new MemoryStream(Convert.FromBase64String(Base64EncodedEmptyDataVersion));
memoryStream.Seek(0L, SeekOrigin.Begin);
Assert.That(
() => binaryFormatter.Deserialize(memoryStream),
Throws.Exception
);
}
}

I'd recommend a "Java" way here - declare int field in every single serializable class like private int _Serializable = 0; and check that your current version & serialized version match; manually increase when you change properties. If you insist on automated way you'll have to store a lot of metadata and check if current metadata & persisted metadata matches (extra burden on performance/size of serialized data).
Here is the automatic descriptor. Basically you'll have to store TypeDescriptor instance as a part of your binary data & on retrieve check if persisted TypeDescriptor is valid for serialization (IsValidForSerialization) against current TypeDescriptor.
var persistedDescriptor = ...;
var currentDescriptor = Describe(typeof(Foo));
bool isValid = persistedDescriptor.IsValidForSerialization(currentDescriptor);
[Serializable]
[DataContract]
public class TypeDescriptor
{
[DataMember]
public string TypeName { get; set; }
[DataMember]
public IList<FieldDescriptor> Fields { get; set; }
public TypeDescriptor()
{
Fields = new List<FieldDescriptor>();
}
public bool IsValidForSerialization(TypeDescriptor currentType)
{
if (!string.Equals(TypeName, currentType.TypeName, StringComparison.Ordinal))
{
return false;
}
foreach(var field in Fields)
{
var mirrorField = currentType.Fields.FirstOrDefault(f => string.Equals(f.FieldName, field.FieldName, StringComparison.Ordinal));
if (mirrorField == null)
{
return false;
}
if (!field.Type.IsValidForSerialization(mirrorField.Type))
{
return false;
}
}
return true;
}
}
[Serializable]
[DataContract]
public class FieldDescriptor
{
[DataMember]
public TypeDescriptor Type { get; set; }
[DataMember]
public string FieldName { get; set; }
}
private static TypeDescriptor Describe(Type type, IDictionary<Type, TypeDescriptor> knownTypes)
{
if (knownTypes.ContainsKey(type))
{
return knownTypes[type];
}
var descriptor = new TypeDescriptor { TypeName = type.FullName, Fields = new List<FieldDescriptor>() };
knownTypes.Add(type, descriptor);
if (!type.IsPrimitive && type != typeof(string))
{
foreach (var field in type.GetFields(BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public).OrderBy(f => f.Name))
{
var attributes = field.GetCustomAttributes(typeof(NonSerializedAttribute), false);
if (attributes != null && attributes.Length > 0)
{
continue;
}
descriptor.Fields.Add(new FieldDescriptor { FieldName = field.Name, Type = Describe(field.FieldType, knownTypes) });
}
}
return descriptor;
}
public static TypeDescriptor Describe(Type type)
{
return Describe(type, new Dictionary<Type, TypeDescriptor>());
}
I also though about some mechanism of shortening size of persisted metadata - like calculating MD5 from xml-serialized or json-serialized TypeDescriptor; but in that case new property/field will mark your object as incompatible for serialization.

JavaScriptSerializer throwing AmbiguousMatchException when “new” used in subclass

I know that the same problem is faced by a lot of people in one way or another but what I'm confused about is that how come Newtonsoft JSON Serializer is able to correctly handle this case while JavaScriptSerializer fails to do so.
I'm going to use the same code sample used in one of the other stackoverflow thread (JavascriptSerializer serializing property twice when "new" used in subclass)
void Main()
{
System.Web.Script.Serialization.JavaScriptSerializer serializer = new System.Web.Script.Serialization.JavaScriptSerializer();
var json = serializer.Serialize(new Limited());
Limited status = serializer.Deserialize<Limited>(json); --> throws AmbiguousMatchException
}
public class Full
{
public String Stuff { get { return "Common things"; } }
public FullStatus Status { get; set; }
public Full(bool includestatus)
{
if(includestatus)
Status = new FullStatus();
}
}
public class Limited : Full
{
public new LimitedStatus Status { get; set; }
public Limited() : base(false)
{
Status = new LimitedStatus();
}
}
public class FullStatus
{
public String Text { get { return "Loads and loads and loads of things"; } }
}
public class LimitedStatus
{
public String Text { get { return "A few things"; } }
}
But if I use Newtonsoft Json Serializer, everythings works fine. Why? And is it possible to achieve the same using JavaScriptSerializer?
void Main()
{
var json = JsonConvert.SerializeObject(new Limited());
Limited status = JsonConvert.DeserializeObject<Limited>(json); ----> Works fine.
}

The reason this works in Json.NET is that it has specific code to handle this situation. From JsonPropertyCollection.cs:
/// <summary>
/// Adds a <see cref="JsonProperty"/> object.
/// </summary>
/// <param name="property">The property to add to the collection.</param>
public void AddProperty(JsonProperty property)
{
if (Contains(property.PropertyName))
{
// don't overwrite existing property with ignored property
if (property.Ignored)
return;
JsonProperty existingProperty = this[property.PropertyName];
bool duplicateProperty = true;
if (existingProperty.Ignored)
{
// remove ignored property so it can be replaced in collection
Remove(existingProperty);
duplicateProperty = false;
}
else
{
if (property.DeclaringType != null && existingProperty.DeclaringType != null)
{
if (property.DeclaringType.IsSubclassOf(existingProperty.DeclaringType))
{
// current property is on a derived class and hides the existing
Remove(existingProperty);
duplicateProperty = false;
}
if (existingProperty.DeclaringType.IsSubclassOf(property.DeclaringType))
{
// current property is hidden by the existing so don't add it
return;
}
}
}
if (duplicateProperty)
throw new JsonSerializationException("A member with the name '{0}' already exists on '{1}'. Use the JsonPropertyAttribute to specify another name.".FormatWith(CultureInfo.InvariantCulture, property.PropertyName, _type));
}
Add(property);
}
As you can see above, there is specific code here to prefer derived class properties over base class properties of the same name and visibility.
JavaScriptSerializer has no such logic. It simply calls Type.GetProperty(string, flags)
PropertyInfo propInfo = serverType.GetProperty(memberName,
BindingFlags.Instance | BindingFlags.IgnoreCase | BindingFlags.Public);
This method is documented to throw an exception in exactly this situation:
Situations in which AmbiguousMatchException occurs include the following:
A type contains two indexed properties that have the same name but different numbers of parameters. To resolve the ambiguity, use an overload of the GetProperty method that specifies parameter types.
A derived type declares a property that hides an inherited property with the same name, using the new modifier (Shadows in Visual Basic). To resolve the ambiguity, include BindingFlags.DeclaredOnly to restrict the search to members that are not inherited.
I don't know why Microsoft didn't add logic for this to JavaScriptSerializer. It's really a very simple piece of code; perhaps it got eclipsed by DataContractJsonSerializer?
You do have a workaround, which is to write a custom JavaScriptConverter:
public class LimitedConverter : JavaScriptConverter
{
const string StuffName = "Stuff";
const string StatusName = "Status";
public override object Deserialize(IDictionary<string, object> dictionary, Type type, JavaScriptSerializer serializer)
{
var limited = new Limited();
object value;
if (dictionary.TryGetValue(StuffName, out value))
{
// limited.Stuff = serializer.ConvertToType<string>(value); // Actually it's get only.
}
if (dictionary.TryGetValue(StatusName, out value))
{
limited.Status = serializer.ConvertToType<LimitedStatus>(value);
}
return limited;
}
public override IDictionary<string, object> Serialize(object obj, JavaScriptSerializer serializer)
{
var limited = (Limited)obj;
if (limited == null)
return null;
var dict = new Dictionary<string, object>();
if (limited.Stuff != null)
dict.Add(StuffName, limited.Stuff);
if (limited.Status != null)
dict.Add(StatusName, limited.Status);
return dict;
}
public override IEnumerable<Type> SupportedTypes
{
get { return new [] { typeof(Limited) } ; }
}
}
And then use it like:
try
{
System.Web.Script.Serialization.JavaScriptSerializer serializer = new System.Web.Script.Serialization.JavaScriptSerializer();
serializer.RegisterConverters(new JavaScriptConverter[] { new LimitedConverter() });
var json = serializer.Serialize(new Limited());
Debug.WriteLine(json);
var status = serializer.Deserialize<Limited>(json);
var json2 = serializer.Serialize(status);
Debug.WriteLine(json2);
}
catch (Exception ex)
{
Debug.Assert(false, ex.ToString()); // NO ASSERT.
}

C# JSON deserializer does only check format, it doesn't care about object data

I want to deserialize a JSON message received from TCP/IP in my C# application. I have really simple classes with one or two Properties at most, but they are derived. For example:
public class SemiCommand
{
public SemiCommand()
{
UID = string.Empty;
}
public SemiCommand(string uid)
{
UID = uid;
}
public string UID
{
get;
set;
}
public new string ToString()
{
return new JavaScriptSerializer().Serialize(this);
}
}
public class ResponseCommand : SemiCommand
{
protected const string DEFAULT_INFO = "OK";
public ResponseCommand()
{
UID = string.Empty;
Info = DEFAULT_INFO;
}
public ResponseCommand(string uid)
{
UID = uid;
Info = DEFAULT_INFO;
}
public string Response
{
get;
set;
}
public string Info
{
get;
set;
}
public class GetInfoResponseCommand : ResponseCommand
{
public GetInfoResponseCommand()
: base()
{
Response = "GetInfoResponse";
}
public List<ClientProcess> Processes
{
get;
set;
}
}
I made a few attempts with DataMember and DataContract attributes to solve my problem, but it didn't fix it.
The problem lies here:
private Type[] _deserializationTypes = new Type[] { typeof(StartProcessesRequestCommand), typeof(RequestCommand) };
public RequestCommand TranslateTCPMessageToCommand(string messageInString)
{
RequestCommand requestCommand = null;
for (int i = 0; i < _deserializationTypes.Length; i++)
{
try
{
requestCommand = TryDeserialize(_deserializationTypes[i], messageInString);
break;
}
catch
{
}
}
if (requestCommand == null || (requestCommand != null && requestCommand.Command == string.Empty))
{
throw new System.Runtime.Serialization.SerializationException("Unable to deserialize received message");
}
else
{
return requestCommand;
}
}
If i want to deserialize with a proper format, containing all the necessary properties, it works:
This works: {"Response":"SomeResponse","Info":"Information","UID":"123"}
However, this also works: {"nonexistingProperty":"value"}
The second one also creates a RequestCommand with property values set to null.
1st question: How can I force my messagetranslator function to make RequestCommand only if it receives all of the required properties
2nd question: If i have multiple command types which are derived from one or more ancestors, how is it possible, to deserialize it automatically from the "deepest" class if the received properties allow it.
EDIT:
I serialize/deserialize with these two
private RequestCommand TryDeserialize(Type type, string messageInString)
{
JavaScriptSerializer js = new JavaScriptSerializer();
return (RequestCommand)js.Deserialize(messageInString, type);
}
public string TranslateCommandToTCPMessage(ResponseCommand command)
{
JavaScriptSerializer js = new JavaScriptSerializer();
return js.Serialize(command);
}
The break in the try catch supposed to end the loop if the TrySerialize doesn't throw an exception, therefore the deserialization was successful.

If you actually used DataContractJsonSerializer, and you decorated the members / types in question with DataMember / DataContract, you can actually get this to happen. To do this, you can decorate ALL data members with [IsRequired=true]. This would throw an exception if the members weren't present on the wire.
Another option you have is to use IObjectReference, which let you translate one object to another post-serialization, and you can do this depending on what is deserialized from the wire.