Develop Reference

MongoDB Changestream Filters

Currently using the MongoDB.Driver v2.11.0 in a .Net Core 3.1 application. Trying to read values from a changestream and have been following the documentation here:
https://mongodb.github.io/mongo-csharp-driver/2.11/reference/driver/change_streams/
My problem is i'm trying to only get objects with a specific type and updated field, but cannot get both filters working. I can get the changestream to grab only objects that have a type of Z from the fulldocument, but cannot get any updateDescription.updatedFields filters to work properly.
Object
public class Abc
{
[BsonElement("d")]
public D D{ get; set; }
[BsonElement("e")]
public E E{ get; set; }
}
public class D
{
[BsonElement("type")]
public string Type{ get; set; }
}
public class E
{
[BsonElement("e")]
public F F{ get; set; }
}
public class F
{
[BsonElement("status")]
public string Status { get; set; }
}
So after connecting to mongo and getting the collection here is the code to setup the change stream.
protected ChangeStreamOptions _changeStreamOptions => new ChangeStreamOptions { FullDocument = ChangeStreamFullDocumentOption.UpdateLookup };
public IChangeStreamCursor<ChangeStreamDocument<Abc>> GetChangeStreamCursor()
{
return _collection.Watch(ConfigurePipeline(), _changeStreamOptions);
}
private PipelineDefinition<ChangeStreamDocument<Abc>, ChangeStreamDocument<Abc>> ConfigurePipeline()
{
List<IPipelineStageDefinition> pipeline = new List<IPipelineStageDefinition>();
pipeline.Add(PipelineStageDefinitionBuilder.Match(ConfigureFilters()));
return pipeline;
}
private FilterDefinition<ChangeStreamDocument<Abc>> ConfigureFilters()
{
var builder = Builders<ChangeStreamDocument<Abc>>.Filter;
//here is where I build the filters and having the issues.
//if its just based on the object type It works.
return builder.Eq("fullDocument.d.type", "z");
}
This works and have no problems getting only objects that had a type of Z that were updated.
If I try to then also add a filter to only return objects of the Z and that had the field Status to have been updated to a specific type.
Here is what I've tried for that:
builder.Eq("fullDocument.d.type", "z") & builder.AnyEq(x => x.UpdateDescription.UpdatedFields.Values, "Action");
builder.Eq("updateDescription.updatedFields.e.f.status", "Action");
builder.Eq("e.f.status", "Action");
The application will run with this but just never picks up any changes.
Also only tried looking at the updated fields since the UpdateDescription.UpdatedFields is an array.
builder.ElemMatch(x => x.UpdateDescription.UpdatedFields, x => x.Name == "e.f.status" && x.Value == "Action")
This fails in the program main with this error
Unable to determine the serialization information for x => x.UpdateDescription.UpdatedFields.'
Simply put I need to create a change stream and only get back objects that have a specific property type in the fulldocument and where another property was updated to a specific type.
Any help would be appreciated!
Edit Update:
If I change the types to be a BsonDocument I can pass this as the pipeline and it works:
So I got it working with a BsonObject and using this:
var filter = "{ $and: [ { operationType: 'update' }, " +
"{ 'fullDocument.d.type' : 'z'}" +
"{ 'updateDescription.updatedFields': { 'e.f.status': 'Action' } } ] }";
But I would like to keep it typed if possible.

So I figured out a solution that is working for me the way I described above.
private FilterDefinition<ChangeStreamDocument<Abc>> ConfigureFilters()
{
var builder = Builders<ChangeStreamDocument<Abc>>.Filter;
var filters = builder.And(builder.Eq("fullDocument.d.type", "z") & new BsonDocument("e.f.status", new BsonDocument("$eq", "Action")));
return filters;
}

Trying to deep clone the row values with in the entity except Id

I have project entity structure like this as below with project name and project number and along with some list objects as well as properties.
public class DesignProject
{
[Key, GraphQLNonNullType]
public string ProjectNumber { get; set; }
public string Name { get; set; }
[Column(TypeName = "jsonb")]
public ProjectSectionStatus SectionStatuses { get; set; } = new ProjectSectionStatus();
[ForeignKey("AshraeClimateZone"), GraphQLIgnore]
public Guid? AshraeClimateZoneId { get; set; }
public virtual AshraeClimateZone AshraeClimateZone { get; set; }
[Column(TypeName = "jsonb")]
public List<ProjectObject<classA>> classAList{ get; set; } = new List<ProjectObject<classA>>();
[Column(TypeName = "jsonb")]
public List<ProjectObject<classB>> classBList{ get; set; } = new List<ProjectObject<classB>>();
......
......
......
Some more json columns
}
and the project object class like this
public class ProjectObject<T>
{
public Guid? Id { get; set; }
public T OriginalObject { get; set; }
public T ModifiedObject { get; set; }
[GraphQLIgnore, JsonIgnore]
public T TargetObject
{
get
{
return ModifiedObject ?? OriginalObject;
}
}
}
and ClassA entity structure like as below
public class ClassA
{
public string Edition { get; set; }
public string City { get; set; }
}
and i have some similar children entities (ClassA) same like as above, I want to copy the contents and statuses from one project entity to other project entity.
I have project entity with ProjectNumber 1212 and have another project having ProjectNumber like 23323 so i would like copy entire project contents from 1212 to 23323. So is there any way we can achieve this with C# and i am using .Net Core with Entity framework core.
Here the source design project that i am going to copy have same structure with destination design project and i am fine with overriding the destination project values and i don't want to update the project number here.
Could any one please let me know the how can i achieve this copying? Thanks in advance!!
Please let me know if i need to add any details for this question
Update : Deep copy related code
public InsertResponse<string> CopyBookmarkproject(string SourceProjectNumber, string destinationProjectNumber)
{
var sourceDesignProject = this._dbContext.DesignProject.Where(a => a.ProjectNumber == SourceProjectNumber).SingleOrDefault();
var destinationProject = this._dbContext.DesignProject.Where(a => a.ProjectNumber == destinationProjectNumber).SingleOrDefault();
CopyProject(sourceDesignProject, destinationProject);
// Need to update the Db context at here after deep copy
}
private void CopyProject(DesignProject sourceDesignProject, DesignProject destinationProject)
{
destinationProject.classAList= sourceDesignProject.classAList; // Not sure whether this copy will works
destinationProject.AshraeClimateZone = sourceDesignProject.AshraeClimateZone; // not sure whether this copy will works also
}
Updated solution 2:
var sourceDesignProject = this._dbContext.DesignProjects.AsNoTracking()
.Where(a => a.ProjectNumber == sourceProjectNumber)
.Include(a => a.PrimaryBuildingType)
.Include(a => a.AshraeClimateZone).SingleOrDefault();
var targetDesignProject = this._dbContext.DesignProjects.Where(a => a.ProjectNumber == targetProjectNumber).SingleOrDefault();
sourceDesignProject.ProjectNumber = targetDesignProject.ProjectNumber;
sourceDesignProject.SectionStatuses.AirSystemsSectionStatus = Entities.Enums.ProjectSectionStage.INCOMPLETE;
sourceDesignProject.SectionStatuses.CodesAndGuidelinesSectionStatus = Entities.Enums.ProjectSectionStage.INCOMPLETE;
sourceDesignProject.SectionStatuses.ExecutiveSummarySectionStatus = Entities.Enums.ProjectSectionStage.INCOMPLETE;
sourceDesignProject.SectionStatuses.ExhaustEquipmentSectionStatus = Entities.Enums.ProjectSectionStage.INCOMPLETE;
sourceDesignProject.SectionStatuses.InfiltrationSectionStatus = Entities.Enums.ProjectSectionStage.INCOMPLETE;
this._dbContext.Entry(sourceDesignProject).State = EntityState.Modified; // getting the below error at this line
this._dbContext.SaveChanges();
ok = true;
getting an error like as below
The instance of entity type 'DesignProject' cannot be tracked because another instance with the same key value for {'ProjectNumber'} is already being tracked. When attaching existing entities, ensure that only one entity instance with a given key value is attached.

There's a simple and short solution which uses .AsNoTracking() for source entry, assigning destination's ProjectNumber to the source and then changing the EntityState as Modified for it. It will copy all the properties to the destination entry :
var sourceDesignProject = this._dbContext.DesignProject.AsNoTracking().Where(a => a.ProjectNumber == SourceProjectNumber).SingleOrDefault();
var destinationProject = this._dbContext.DesignProject.Where(a => a.ProjectNumber == destinationProjectNumber).SingleOrDefault();
sourceDesignProject.ProjectNumber = destinationProject.ProjectNumber;
this._dbContext.Entry(sourceDesignProject).State = System.Data.Entity.EntityState.Modified;
this._dbContext.SaveChanges();
Make sure your relational properties are loaded before.

You can make use of Reflection. As you have both, source and destination objects and are of the same type. You can just traverse through source object and fetch the properties and replace the values.
Something like :
if (inputObjectA != null && inputObjectB != null)
{
//create variables to store object values
object value1, value2;
PropertyInfo[] properties = inputObjectA.GetType().GetProperties(BindingFlags.Public | BindingFlags.Instance);
//get all public properties of the object using reflection
foreach (PropertyInfo propertyInfo in properties)
{
//get the property values of both the objects
value1 = propertyInfo.GetValue(inputObjectA, null);
value2 = propertyInfo.GetValue(inputObjectB, null);
}
Ignore the properties that are not needed to be copied. The method will have a signature like below :
CopyObjects(object inputObjectA, object inputObjectB, string[] ignorePropertiesList)

Two choices bring to mind for deep copy of objects:
1- Using BinaryFormatter (Needs Serializable attribute on your class, See original answer):
public static T DeepClone<T>(this T obj)
{
using (var ms = new MemoryStream())
{
var formatter = new BinaryFormatter();
formatter.Serialize(ms, obj);
ms.Position = 0;
return (T) formatter.Deserialize(ms);
}
}
2- Using JsonSerializers like NewtoneSoft:
public static T CloneJson<T>(this T source)
{
// Don't serialize a null object, simply return the default for that object
if (Object.ReferenceEquals(source, null))
{
return default(T);
}
// initialize inner objects individually
// for example in default constructor some list property initialized with some values,
// but in 'source' these items are cleaned -
// without ObjectCreationHandling.Replace default constructor values will be added to result
var deserializeSettings = new JsonSerializerSettings {ObjectCreationHandling = ObjectCreationHandling.Replace};
return JsonConvert.DeserializeObject<T>(JsonConvert.SerializeObject(source), deserializeSettings);
}
Update
Here's the code you can use with NewtonSoft Json library:
private void CopyProject(DesignProject sourceDesignProject, ref DesignProject destinationProject)
{
if (Object.ReferenceEquals(sourceDesignProject, null))
{
destinationProject = null;
return;
}
var deserializeSettings = new JsonSerializerSettings {ObjectCreationHandling = ObjectCreationHandling.Replace};
var destProjNumber = destinationProject.ProjectNumber;
destinationProject = JsonConvert.DeserializeObject<DesignProject>
(JsonConvert.SerializeObject(sourceDesignProject), deserializeSettings);
destinationProject.ProjectNumber = destProjNumber;
}

I always prefer using AutoMapper for these type of work. It's optimized and lower risk of making mistakes. And better than serialize/deserialize method since that would cause performance issues.
In your constructor (or in startup.cs and passing via dependency injection)
var config = new MapperConfiguration(cfg => {
cfg.CreateMap<DesignProject, DesignProject>().ForMember(x => x.Id, opt => opt.Ignore());
cfg.CreateMap<ProjectObject<ClassA>, ProjectObject<ClassA>>().ForMember(x => x.Id, opt => opt.Ignore()));
cfg.CreateMap<ProjectObject<ClassA>, ProjectObject<ClassB>>().ForMember(x => x.Id, opt => opt.Ignore()));
cfg.CreateMap<ProjectObject<ClassB>, ProjectObject<ClassB>>().ForMember(x => x.Id, opt => opt.Ignore()));
cfg.CreateMap<ProjectObject<ClassB>, ProjectObject<ClassA>>().ForMember(x => x.Id, opt => opt.Ignore()));
});
var mapper = config.CreateMapper();
In your method :
var newProject = mapper.Map(sourceProject, new DesignProject());
OR
mapper.Map(sourceProject, targetProject);

This shouldn't work. If I understand you want to copy all elements from list into new list, but to make new object. Than maybe you can use extension method to clone complete list as described here link. For updating navigation properties check this post link.

Filtering from list of C# or LINQ

I am trying to filter from attachList the taxheaderID, it comes from my database which is structured as such.
public int attachmentID { get; set; }
public int headerID { get; set; }
public string uploadedfilename { get; set; }
public string originalfilename { get; set; }
public string foldername { get; set; }
Here is the code that gets data from the database:
public JsonResult GetAllAttach()
{
using (car_monitoringEntities contextObj = new car_monitoringEntities())
{
var attachList = contextObj.car_taxcomputationattachment.ToList();
return Json(attachList, JsonRequestBehavior.AllowGet);
}
}
These are my attempts:
attachList
.Select(x => x.headerID)
.Where(x => x == x)
.Take(1);
and:
attachList = attachList
.Where(al => attachList
.Any(alx => al.taxheaderID == alx.headerID
&& al.headerID == alx.headerID));
The problem is I want to parse multiple attach on a single headerID or filter them base on headerID. For example:
Problem to fix:
This is the table
Desired output:
Combined
data table:
data table
data table 2
Here is the actual solution that was made to get the output, but my coworker told me that it is not a good practice that's why I'm trying to filter it in the function itself. apologies for the trouble, thanks!
<div ng-repeat="att in attach|filter:{headerID:header.headerID}:true">
{{att.uploadedfilename}} <br />
</div>

To get attachments by Id
public JsonResult GetAllAttach(int headerId)
{
using (car_monitoringEntities contextObj = new car_monitoringEntities())
{
var attachList = contextObj.car_taxcomputationattachment
.Where(x => x.headerID == headerId)
.ToList();
return Json(attachList, JsonRequestBehavior.AllowGet);
}
}
If you want to have all data in one JSON result, then you need to create a nested view model.

Assuming you have the header id on which you want to filter in a local variable, you are almost correct
int headerIdToFind = 19;
// think of x as a local variable inside a foreach loop which
// iterates over each item in the attachList (it does not exist
// outside the where method)
// this is what you got wrong when you compared the item to itself
var filteredAttach = attachList.Where(x => x.headerId = headerIdToFind);
// if you want to select only some properties based on header id
// you can use select to project those properties
var filteredAttach = attachList.Where(x => x.headerId = headerIdToFind).
Select(x => new {x.attachmentId, x.folderName});
// based on last image, you only want to select (project) header id and the
// filename. so you do not need where (filter) at all
// you can put all the properties you need in the select clause
var filteredAttach = attachList.Select(x => new {x.headerId, x.attachmentId});
// you can enumerate the filtered attach list of convert it into a list
var filteredAttach = filteredAttach.ToList();

Cosmos DB - Use ARRAY_CONTAINS in LINQ

I have collection of documents in Cosmos DB. Document can have inner array of objects. So model look like this:
public class Document
{
public string Id { get; set; }
public IList<InnerDocument> InnerDocuments { get; set; }
}
public class InnerDocument
{
public string Type { get; set; }
public string Created { get; set; }
}
I need to get all inner documents if at least one of them has certain type.
If I create query like this:
var innerDocument = new InnerDocument()
{
Type = "foo"
};
context.CreateDocumentQuery<Document>(uri, feedOptions)
.Where(d => d.id == "sample" && d.InnerDocuments.Contains(innerDocument));
it translate like this:
SELECT * FROM root
WHERE (root[\"id\"] = "sample"
AND ARRAY_CONTAINS(root[\"innerDocuments\"], {\"type\":\"foo\"}))
but it returns nothing, because no inner document look like this (all inner documents has also Created) so I need to add third parameter to ARRAY_CONTAINS (which tell that only part match on document is enough) so it should look like this:
SELECT * FROM root
WHERE (root[\"id\"] = "sample"
AND ARRAY_CONTAINS(root[\"innerDocuments\"], {\"type\":\"foo\"}, true))
My problem is that I did not figure out how to pass third parameter in linq. I also tried write IEqualityComparer, which always return true but with no effect (well efect was that I got exception..).
Do you have any idea how could I pass that param in linq?
Thanks.

as far as I know, unfortunately there is no LINQ equivalent for the ARRAY_CONTAINS (<arr_expr>, <expr> , bool_expr) overload. To achieve your scenarios, for now you can use SQL query. We are currently working on a set of changes that will enable LINQ for this scenario.
Edit: the available alternative is to use the Any operator with the filters on the property you want to match. For example, the SQL filter: ARRAY_CONTAINS(root.addresses, {"city": "Redmond"}, TRUE) is equivalent to this LINQ expression: addresses.Any(address => address.city == "Redmond")

If I understand correctly, you wish to retrieve all documents that have any inner document in the array with a given property value ("foo" in this example).
Normally, you would use .Where(d => d.InnerDocuments.Any(i => i.Type == "foo")), but Any is not supported yet by the Cosmos LINQ provider.
Instead, you can use this construct as a work-around:
context.CreateDocumentQuery<Document>(uri, feedOptions)
.Where(d => d.Id == "sample")
.SelectMany(d => d.InnerDocuments.Where(i => i.Type == "foo").Select(i => d));
According to this thread Microsoft has recently started working on a real Any feature for the Cosmos LINQ provider.

My solution was slightly more of a hack than a solution, but it works temporarily until the full functionality for .Any() exists.
I use Expressions to dynamically build the Where predicate for my documents, allowing me pass in a CosmosSearchCriteria object which has a list of CosmosCriteria objects as below:
public class CosmosCriteria
{
public CosmosCriteria()
{
ContainsValues = new List<string>();
}
public CosmosCriteriaType CriteriaType { get; set; }
public string PropertyName { get; set; }
public string PropertyValue { get; set; }
public ConvertedRuleComparitor Comparitor { get; set; }
public DateRange Dates { get; set; }
public List<string> ContainsValues { get; set; }
}
This allows me to query any property of the Contact model by essentially passing in the PropertyName and PropertyValue.
I haven't looked into the other workaround in here to see if I can make it work with my expression tree building, at the minute I can't afford the time to investigate.
public async Task<CosmosSearchResponse<Model.Contact>>
GetContactsBySearchCriteriaAsync(int pageSize, long companyId,
CosmosSearchCriteria searchCriteria, string continuationToken = null)
{
var collectionName = CreateCollectionName(companyId, Constants.CollectionType.Contacts);
var feedOptions = new FeedOptions { MaxItemCount = pageSize };
if (!String.IsNullOrEmpty(continuationToken))
{
feedOptions.RequestContinuation = continuationToken;
}
var collection = UriFactory.CreateDocumentCollectionUri(
Configuration.GetValue<string>(Constants.Settings.COSMOS_DATABASE_SETTING),
collectionName);
IOrderedQueryable<Model.Contact> documents = Client.CreateDocumentQuery<Model.Contact>(
collection,
feedOptions
);
documents = (IOrderedQueryable<Model.Contact>)documents.Where(document => document.deleted != true);
bool requiresConcatenation = false;
foreach (var criteria in searchCriteria.Criteria)
{
switch (criteria.CriteriaType)
{
case Constants.CosmosCriteriaType.ContactProperty:
// This is where predicates for the documents.Where(xxxx)
// clauses are built dynamically with Expressions.
documents = AddContactPropertyClauses(documents, criteria);
break;
case Constants.CosmosCriteriaType.PushCampaignHistory:
requiresConcatenation = true;
break;
}
}
documents = (IOrderedQueryable<Model.Contact>)documents.AsDocumentQuery();
/*
From this point onwards, we have to do some wizardry to get around the fact that there is no Linq to SQL
extension overload for the Cosmos DB function ARRAY_CONTAINS (<arr_expr>, <expr> , bool_expr).
The feature is planned for development but is not yet ready.
Keep an eye on the following for updates:
https://stackoverflow.com/questions/52412557/cosmos-db-use-array-contains-in-linq
https://feedback.azure.com/forums/263030-azure-cosmos-db/suggestions/11503872-support-linq-any-or-where-for-child-object-collect
*/
if (requiresConcatenation)
{
var sqlString = documents.ToString();
var jsonDoc = JsonConvert.DeserializeObject<dynamic>(sqlString); // Have to do this to remove the escaping
var q = (string)jsonDoc.query;
var queryRootAlias = Util.GetAliasNameFromQuery(q);
if (queryRootAlias == string.Empty)
{
throw new FormatException("Unable to parse root alias from query.");
}
foreach (var criteria in searchCriteria.Criteria)
{
switch (criteria.CriteriaType)
{
case Constants.CosmosCriteriaType.PushCampaignHistory:
q += string.Format(" AND ARRAY_CONTAINS({0}[\"CampaignHistory\"], {{\"CampaignType\":1,\"CampaignId\":{1}, \"IsOpened\": true }}, true) ", queryRootAlias, criteria.PropertyValue);
break;
}
}
documents = (IOrderedQueryable<Model.Contact>)Client.CreateDocumentQuery<Model.Contact>(
collection,
q,
feedOptions
).AsDocumentQuery();
}
var returnValue = new CosmosSearchResponse<Model.Contact>();
returnValue.Results = new List<Model.Contact>();
Console.WriteLine(documents.ToString());
var resultsPage = await ((IDocumentQuery<Model.Contact>)documents).ExecuteNextAsync<Model.Contact>();
returnValue.Results.AddRange(resultsPage);
if (((IDocumentQuery<Model.Contact>)documents).HasMoreResults)
{
returnValue.ContinuationToken = resultsPage.ResponseContinuation;
}
return returnValue;
}
Hope this helps, or if someone has a better way, please do tell!
Dave

C# MongoDB LINQ: Cannot query nested list

I'm trying to query a MongoDB collection using official C# driver. Here's the object structure I've created:
IMongoDatabase db = mongoClient.GetDatabase("appdb");
IMongoCollection<MusicFile> musicfiles = db.GetCollection<MusicFile>("files");
public class MusicFile
{
public ObjectId Id { get; set; }
public string Name { get; set; }
public IList<Comment> Comments { get; set; }
}
public class Comment
{
public string Text { get; set; }
}
This is the query I'm trying to get any MusicFile objects that contains a Comment object with property Text = "Comment1":
musicfiles.AsQueryable().Where(f => f.Comments != null && f.Comments.Any(c => c.Text == "Comment1")).ToList();
I can't get this query to work, it always returns an empty list. I also tried this, which too didn't work:
musicfiles.Find(f => f.Comments.Any(c => c.Text == "Comment1")).ToList()
But, if I get the complete collection is memory, the query works:
musicfiles.Find(FilterDefinition<MusicFile>.Empty).ToList().Where(f => f.Comments != null && f.Comments.Any(c => c.Text == "Comment1")).ToList();
This seems like a very inefficient way to query. Any suggestions?

OK. I'm back at home. Try this:
var musicFilter = Builders<MusicFile>.Filter;
var commentFilter = Builders<Comment>.Filter;
var files = musicfiles
.Find(
musicFilter.NE(m => m.Comments, null)
& musicFilter.ElemMatch(m => m.Comments, commentFilter.Eq(c => c.Text, "Comment1"))
)
.ToEnumerable()
.ToList();
Note I call .ToList() because, otherwise, if you iterate through files multiple times, you'll get multiple calls to the database for the same objects.