Develop Reference

MongoDB C# Case Insensitive Sort and Index

So far I've been using this code to find my documents and then sort them:
var options = new FindOptions
{
Modifiers = new BsonDocument("$hint", "PathTypeFilenameIndex")
};
return await Collection
.Find(f => f.Metadata["path"] == path, options)
.SortBy(f => f.Metadata["type"])
.ThenBy(f => f.Filename)
.ToListAsync();
I have a class that has Metadata field with path and type fields, also the class has a Filename field. I want all documents with a given path inside the metadata sorted by type and then by Filename.
An example result would be a list of documents ordered by the Name field like this:
a, Ab, B, c, D
Unfortunately, I get something like this:
Ab, B, D, a, c
And that's because MongoDB sorts the data with a simple binary comparison, where 'A' < 'a' because of their ASCII codes.
So my question is: Is there a way to make a case insensitive sort and keep using the "$hint"?
That options I pass to the Find method should tell MongoDB which index to use. I found this post: MongoDB and C#: Case insensitive search but the method here doesn't work for sorting and I couldn't tell MongoDB which index to use.

I think you can use aggregation pipeline with $addFields, $toLower (to convert filename to lowercase in temporary field), and $sort to sort them irrespective of the case
In mongodb shell you would write something like this :
db.collection.aggregate([{
$addFields : {
"lowercaseFileName" : {
$loLower : "$fileName"
}
},{
$sort : {
"metadata.type" : 1,
lowercaseFileName : 1
}
}
}])
Please write the similar code in c#, and see if it works. I dont know c#, otherwise i would have given you the exact query, but i cant.
The idea is to transform the filename to lowercase, save it in temporary field, using addFields and sort by that field.
Hope this helps you out.
Read more about $addFields, $toLower here.
Update
For whoever wants a working code in C# , thanks to #kaloyan-manev
You can use this :
return await Collection.Aggregate()
.Match(f => f.Metadata["path"] == path)
.AppendStage<BsonDocument>(new BsonDocument("$addFields", new BsonDocument("lowercaseFileName", new BsonDocument("$toLower", "$filename"))))
.AppendStage<GridFSFileInfo>(new BsonDocument("$sort", new BsonDocument { {"metadata.type", 1}, {"lowercaseFileName", 1} }))
.ToListAsync();

Did you try to set the CollationStrenght = 2?
Your code would be similar all you need is to set the Collation in the FindObject:
var options = new FindOptions
{
Modifiers = new BsonDocument("$hint", "PathTypeFilenameIndex"),
Collation = new Collation("en", strength: CollationStrength.Secondary)
};

How to create MongoDB MultiKey index on attribute of items in an array .NET Driver

I have a MongoDB collection "foos" containing items which each have an array of "bars". That is, "foo" has the following schema:
{
"id": UUID
"name": string
...
"bars": [
"id": UUID
"key": string
...
]
}
I need to create an index on name and bar.key using the MongoDB C# .NET Mongo driver.
I presumed I could use a Linq Select function to do this as follows:
Indexes.Add(Context.Collection<FooDocument>().Indexes.CreateOne(
Builders<FooDocument>.IndexKeys
.Descending(x => x.Bars.Select(y => y.Key))));
However this results in an InvalidOperationException:
System.InvalidOperationException: 'Unable to determine the serialization information for x => x.Bars.Select(y => y.Id).'
The Mongo documentation on MultiKey indexes shows how to create such an index using simple dot notation, i.e.
db.foos.createIndex( { "name": 1, "bars.key": 1 } )
However the MongoDB driver documentation seems to suggest that using a Linq function as I'm doing is correct.
How can I create a multikey index on my collection using the MongoDB .NET Driver, preferably using a Linq function?

This is an example how to do it with C#
var indexDefinition = Builders<FooDocument>.IndexKeys.Combine(
Builders<FooDocument>.IndexKeys.Ascending(f => f.Key1),
Builders<FooDocument>.IndexKeys.Ascending(f => f.Key2));
await collection.Indexes.CreateOneAsync(indexDefinition);
UPDATE
Regarding index within the array, closest what i was able to find is to use "-1" as index whene you building your index key. As i understand from github source code is is a valid option in case of building queries.
var indexDefinition = Builders<FooDocument>.IndexKeys.Combine(
Builders<FooDocument>.IndexKeys.Ascending(f => f.Key1),
Builders<FooDocument>.IndexKeys.Ascending(f => f.Key2[-1].Key));
await collection.Indexes.CreateOneAsync(indexDefinition);
"-1" is a hardcoded constant in side mongodb C# drivers which means "$" (proof). So this code would try to create index:
{ "Key1": 1, "Key2.$.Key": 1 }
which is fine for querying info from database, but not allowed (will throw an exception "Index key contains an illegal field name: field name starts with '$'") to use in indexes. So i assume it should be changed in mongodb drivers to make it work. Something like "-2" means empty operator. In that case we could use
var indexDefinition = Builders<FooDocument>.IndexKeys.Combine(
Builders<FooDocument>.IndexKeys.Ascending(f => f.Key1),
Builders<FooDocument>.IndexKeys.Ascending(f => f.Key2[-2].Key));
await collection.Indexes.CreateOneAsync(indexDefinition);
which would generate index like:
{ "Key1": 1, "Key2.Key": 1 }
So basically i don't think it is possible right now to build index you want with pure Linq without changing mongo C# drivers.
So i think your only option do like this, still C# but without Linq
await collection.Indexes.CreateOneAsync(new BsonDocument {{"name", 1}, {"bars.key", 1}});

This appears to be a requested feature for the C# driver, although it hasn't seen any progress lately. That said, someone did submit a rough-and-ready solution there on the JIRA thread, so perhaps that will do the job for you.

You can create a string index and use nameof() in C# 6:
Indexes.Add(Context.Collection<FooDocument>().Indexes.CreateOne($"{nameof(FooDocument.Bars)}.{nameof(Bars.Key)}"));

As of "MongoDB.Driver" Version="2.8.0" syntax has been changed and some of the methods has been deprecated.Following is the way to achieve the same .
Please note the CreateOneAsync(new CreateIndexModel<FooDocument>(indexDefinition)); part
var indexDefinition = Builders<FooDocument>.IndexKeys.Combine(
Builders<FooDocument>.IndexKeys.Ascending(f => f.Key1),
Builders<FooDocument>.IndexKeys.Ascending(f => f.Key2[-1].Key));
await collection.Indexes.CreateOneAsync(new CreateIndexModel<FooDocument>(indexDefinition));

As per a previous comment, C# driver still doesn't support a strongly typed way of doing multikey indexes.
Also, using something like [-1] seems bit hacky and not really what you're after as it'll substitute it with $.
As such I suggest doing this (as per MongoDB.Driver 2.8.0 onwards):
var indexDefinition = Builders<FooDocument>.IndexKeys.Combine(
Builders<FooDocument>.IndexKeys.Ascending(f => f.Key1),
Builders<FooDocument>.IndexKeys.Ascending($"{nameof(FooDocument.Key2)}.{nameof(BarType.Key)}"));
await collection.Indexes.CreateOneAsync(new CreateIndexModel<FooDocument>(indexDefinition), cancellationToken: token);

mongodb c# select specific field dot notation

In addition for my previous question:
mongodb c# select specific field.
I'm writing a generic method for selecting a specific field.
the requirements are:
Field can be of any type
Return type is T
Field can be inside a sub field
Field can be inside array items - in that case its OK to select the specific field of all the items in the array
for shorts, im looking for the "select" / dot notation capability.
for example:
the wanted method:
T GetFieldValue<T>(string id, string fieldName)
the document:
persons
{
"id": "avi"
"Freinds" : [
{
"Name" : "joni",
"age" : "33"
},
{
"Name" : "daniel",
"age" : "27"
}]
}
The goal is to call the method like this:
string[] myFriends = GetFieldValue<string[]>("avi", "Freinds.Name");
myFriends == ["joni","daniel"]
as far as i know, using projection expression with lambda is no good for items in array,
I was thinking more dot notation way.
note:
I'm using the new c# driver (2.0)
Thanks A lot.

I don't see good approach with don notation in string, because it has more issues with collections than generic approach:
For example Persion.Friends.Name
Which element is array in this chain?
You should apply explicit conversion for collection elements (possible place of bugs)
Generic methods are more reliable in support and using:
var friends = await GetFieldValue<Person, Friend[]>("avi", x => x.Friends);
var names = friends.Select(x=>x.Name).ToArray();

How to determine if $addToSet actually added a new item into a MongoDB document or if the item already existed?

I'm using the C# driver (v1.8.3 from NuGet), and having a hard time determining if an $addtoSet/upsert operation actually added a NEW item into the given array, or if the item was already existing.
Adding a new item could fall into two cases, either the document didn't exist at all and was just created by the upsert, or the document existed but the array didn't exist or didn't contain the given item.
The reason I need to do this, is that I have large sets of data to load into MongoDB, which may (shouldn't, but may) break during processing. If this happens, I need to be able to start back up from the beginning without doing duplicate downstream processing (keep processing idempotent). In my flow, if an item is determined to be newly added, I queue up downstream processing of that given item, if it is determined to already have been added in the doc, then no more downstream work is required. My issue is that the result always returns saying that the call modified one document, even if the item was already existing in the array and nothing was actually modified.
Based on my understanding of the C# driver api, I should be able to make the call with WriteConcern.Acknowledged, and then check the WriteConcernResult.DocumentsAffected to see if it indeed updated a document or not.
My issue is that in all cases, the write concern result is returning back that 1 document was updated. :/
Here is an example document that my code is calling $addToSet on, which may or may not have this specific item in the "items" list to start with:
{
"_id" : "some-id-that-we-know-wont-change",
"items" : [
{
"s" : 4,
"i" : "some-value-we-know-is-static",
}
]
}
My query always uses an _id value which is known based on the processing metadata:
var query = new QueryDocument
{
{"_id", "some-id-that-we-know-wont-change"}
};
My update is as follows:
var result = mongoCollection.Update(query, new UpdateDocument()
{
{
"$addToSet", new BsonDocument()
{
{ "items", new BsonDocument()
{
{ "s", 4 },
{ "i", "some-value-we-know-is-static" }
}
}
}
}
}, new MongoUpdateOptions() { Flags = UpdateFlags.Upsert, WriteConcern = WriteConcern.Acknowledged });
if(result.DocumentsAffected > 0 || result.UpdatedExisting)
{
//DO SOME POST PROCESSING WORK THAT SHOULD ONLY HAPPEN ONCE PER ITEM
}
If i run this code one time on an empty collection, the document is added and response is as expected ( DocumentsAffected = 1, UpdatedExisting = false). If I run it again (any number of times), the document doesn't appear to be updated as it remains unchanged but the result is now unexpected (DocumentsAffected = 1, UpdatedExisting = true).
Shouldn't this be returning DocumentsAffected = 0 if the document is unchanged?
As we need to do many millions of these calls a day, I'm hesitant to turn this logic into multiple calls per item (first checking if the item exists in the given documents array, and then adding/queuing or just skipping) if at all possible.
Is there some way to get this working in a single call?

Of course what you are doing here is actually checking the response which does indicate whether a document was updated or inserted or in fact if neither operation happened. That is your best indicator as for an $addToSet to have performed an update the document would then be updated.
The $addToSet operator itself cannot produce duplicates, that is the nature of the operator. But you may indeed have some problems with your logic:
{
"$addToSet", new BsonDocument()
{
{ "items", new BsonDocument()
{
{ "id", item.Id },
{ "v", item.Value }
}
}
}
}
So clearly you are showing that an item in your "set" is composed of two fields, so if that content varies in any way ( i.e same id but different value) then the item is actually a "unique" member of the set and will be added. There would be no way for instance for the $addToSet operator to not add new values purely based on the "id" as a unique identifier. You would have to actually roll that in code.
A second possibility here for a form of duplicate is that your query portion is not correctly finding the document that has to be updated. The result of this would be creating a new document that contains only the newly specified member in the "set". So a common usage mistake is something like this:
db.collection.update(
{
"id": ABC,
"items": { "$elemMatch": {
"id": 123, "v": 10
}},
{
"$addToSet": {
"items": {
"id": 123, "v": 10
}
}
},
{ "upsert": true }
)
The result of that sort of operation would always create a new document because the existing document did not contain the specified element in the "set". The correct implementation is to not check for the presence of the "set" member and allow $addToSet to do the work.
If indeed you do have true duplicate entries occurring in the "set" where all elements of the sub-document are exactly the same, then it has been caused by some other code either present or in the past.
Where you are sure there a new entries being created, look through the code for instances of $push or indeed and array manipulation in code that seems to be acting on the same field.
But if you are using the operator correctly then $addToSet does exactly what it is intended to do.

Sorting by aggregate of two fields

I have a mongo database with documents that look like this:
{
PublishedDate: [date],
PublishedDateOverride: [NullableDate],
...
}
The reason I have the override as a separate field is that it is important to know the original published date as well as the overridden one.
When I get these documents back I want to sort them by their "apparent" published date. That is if there is an override it should use that, otherwise use the original.
Our current system just sorts by PublishedDateOverride and then by PublishedDate which of course groups all of those with a null override together.
For a concrete example take the following four documents:
A = {
PublishedDate: 2014-03-14,
PublishedDateOverride: 2014-03-24,
...
}
B = {
PublishedDate: 2014-01-21,
PublishedDateOverride: 2014-02-02,
...
}
C = {
PublishedDate: 2014-03-01,
PublishedDateOverride: null,
...
}
D = {
PublishedDate: 2014-03-27,
PublishedDateOverride: null,
...
}
The desired sort order would be D (2014-03-27), A (2014-03-14), C (2014-03-01), B (2014-02-02).
I need to be able to do this in the database since I am also paging this data so I can't just sort after getting it out of the database.
So the question:
What is the best way to achieve this goal? Is there a way to sort by an expression? Is there a way to have a calculated field such that whenever I update a document it will put the appropriate date in there to sort on?
I'm doing this in C# in case that is relevant but I would assume any solution would be a mongo one, not in my client code.

If you want a projection of only the valid and greater date then use aggregate with the $cond operator and the $gt operator. A basic shell example for translation (which is not hard) :
db.collection.aggregate([
{ "$project": {
"date": { "$cond": [
{ "$gt": [
"$PublishedDate",
"$PublishedDateOverride"
]},
"$PublishedDate",
"$PublishedDateOverride"
]}
}},
{ "$sort": { "date": 1 } }
])
So that basically breaks down your documents to having the "date" field set to which ever of those two fields had the greater value. Then you can sort on the result. All processed server side.

try
datesCollection.OrderBy(d => d.PublishedDateOverride!= null? d.PublishedDateOverride: d.PublishedDate)

Use Indexes to Sort Query Results
To sort on multiple fields, create a compound index.