Develop Reference

Mongo error when trying to update nested arrays: No array filter found for identifier

I am trying to update a document in Mongo that represents a community with the following scenario.
A community has a collection of blocks
A block has a collection of floors
A floor has a collection of doors
A door has a collection of label names
Given a document Id and information about the labels that must be placed into each door, I want to use the MongoDb C# driver v2.10.4 and mongo:latest to update nested lists (several levels).
I've reading the documentation, about array filters, but I can't have it working.
I've created a repository from scratch to reproduce the problem, with instructions on the Readme on how to run the integration test and a local MongoDB with docker.
But as a summary, my method groupds the labels so that I can bulk place names on the desired door and then it iterates over these groups and updates on Mongo the specific document setting the desired value inside some levels deep nested object. I couldn't think of a more efficient way.
All the code in the above repo.
The DB document:
public class Community
{
public Guid Id { get; set; }
public IEnumerable<Block> Blocks { get; set; } = Enumerable.Empty<Block>();
}
public class Block
{
public string Name { get; set; } = string.Empty;
public IEnumerable<Floor> Floors { get; set; } = Enumerable.Empty<Floor>();
}
public class Floor
{
public string Name { get; set; } = string.Empty;
public IEnumerable<Door> Doors { get; set; } = Enumerable.Empty<Door>();
}
public class Door
{
public string Name { get; set; } = string.Empty;
public IEnumerable<string> LabelNames = Enumerable.Empty<string>();
}
The problematic method with array filters:
public async Task UpdateDoorNames(Guid id, IEnumerable<Label> labels)
{
var labelsGroupedByHouse =
labels
.ToList()
.GroupBy(x => new { x.BlockId, x.FloorId, x.DoorId })
.ToList();
var filter =
Builders<Community>
.Filter
.Where(x => x.Id == id);
foreach (var house in labelsGroupedByHouse)
{
var houseBlockName = house.Key.BlockId;
var houseFloorName = house.Key.FloorId;
var houseDoorName = house.Key.DoorId;
var names = house.Select(x => x.Name).ToList();
var update =
Builders<Community>
.Update
.Set($"Blocks.$[{houseBlockName}].Floors.$[{houseFloorName}].Doors.$[{houseDoorName}].LabelNames", names);
await _communities.UpdateOneAsync(filter, update);
}
}
The exception is
MongoDB.Driver.MongoWriteException with the message "A write operation resulted in an error.
No array filter found for identifier 'Block 1' in path 'Blocks.$[Block 1].Floors.$[Ground Floor].Doors.$[A].LabelNames'"
Here's a more visual sample on how the nested structure looks like in the database. Notice the value I want to update is the LabelNames, which is an array of string.
I appreciate any help to have this working and suggestions on whether it's the right approach assuming that I cannot change the repository's method signature.
SOLUTION RESULT:
Thanks for the quick answer #mickl, it works perfectly.
Result at this repo's specific point of history exactly as suggested.

The $[{houseBlockName}] expects an identifier which acts as a placeholder and has a corresponding filter defined within arrayfilters (positional filtered). It seems like you're trying to pass the filter value directly which is incorrect.
Your C# code can look like this:
var houseBlockName = house.Key.BlockId;
var houseFloorName = house.Key.FloorId;
var houseDoorName = house.Key.DoorId;
var names = house.Select(x => x.Name).ToList();
var update = Builders<Community>.Update.Set("Blocks.$[block].Floors.$[floor].Doors.$[door].LabelNames", names);
var arrayFilters = new List<ArrayFilterDefinition>();
ArrayFilterDefinition<BsonDocument> blockFilter = new BsonDocument("block.Name", new BsonDocument("$eq", houseBlockName));
ArrayFilterDefinition<BsonDocument> floorFilter = new BsonDocument("floor.Name", new BsonDocument("$eq", houseFloorName));
ArrayFilterDefinition<BsonDocument> doorFilter = new BsonDocument("door.Name", new BsonDocument("$eq", houseDoorName));
arrayFilters.Add(blockFilter);
arrayFilters.Add(floorFilter);
arrayFilters.Add(doorFilter);
var updateOptions = new UpdateOptions { ArrayFilters = arrayFilters };
var result = _communities.UpdateOne(filter, update, updateOptions);

{
var filterCompany = Builders<CompanyInfo>.Filter.Eq(x => x.Id, Timekeepping.CompanyID);
var update = Builders<CompanyInfo>.Update.Set("LstPersonnel.$[i].Timekeeping.$[j].CheckOutDate", DateTime.UtcNow);
var arrayFilters = new List<ArrayFilterDefinition>
{
new BsonDocumentArrayFilterDefinition<BsonDocument>(new BsonDocument("i.MacAddress",new BsonDocument("$eq", Timekeepping.MacAddress) )),
new BsonDocumentArrayFilterDefinition<BsonDocument>(new BsonDocument("j.Id", new BsonDocument("$eq", timeKeeping.Id)))
};
var updateOptions = new UpdateOptions { ArrayFilters = arrayFilters};
var updateResult = await _companys.UpdateOneAsync(filterCompany, update, updateOptions);
return updateResult.ModifiedCount != 0;
}

How to return two specific columns from database using LINQ?

I have table called Asset. It has lot of columns. I only want to select two of them and use them separately.
Both of these columns are strings.
Linq query :
public static List<string> GetAssetIdsWithNames()
{
using (var db = DbManager.Get())
{
var result = db.Assets.SelectMany(i=> new[] { i.AssetName, i.AssetId }).Distinct().ToList();
return result;
}
}
Where I want to use them :
var assetList = AssetManager.GetAssetIdsWithNames();
//CURRENCYBOX IS A DROPDOWN
CurrencyBox.DataSource = assetList;
CurrencyBox.DataBind();
foreach (var item in assetList)
{
CurrencyBox.DataValueField = //asset id goes here
CurrencyBox.DataTextField =//asset name goes here
break;
}

You cannot access the anonymous type outside of the local scope.
Anonymous types can only be returned as Object outside their local scope and their properties inspected via reflection.
So in this scenario, you are likely better off to use a typed data contract and map from your Asset entity instead and then access it from your calling method.
Your use of SelectMany seems odd too, you probably are after Select instead.
public class AssetDto
{
public string Name { get;set; }
public string Id { get; set; }
}
public static List<AssetDto> GetAssetIdsWithNames()
{
using (var db = DbManager.Get())
{
var result = db.Assets.Select(i=> new AssetDto { Name = i.AssetName, Id = i.AssetId }).ToList();
return result;
}
}

You could use named value tuples for that so you don't need to create an extra class
public static List<(string Name, int Id)> GetAssetWithIds()
{
using (var db = DbManager.Get())
{
var result = db.Assets
.Select(a => new { a.AssetName, a.AssetId })
.Distinct().AsEnumerable()
.Select(a => (a.AssetName, a.AssetId))
.ToList();
return result;
}
}
You will need to add System.ValueTuple

Cosmos DB - Use ARRAY_CONTAINS in LINQ

I have collection of documents in Cosmos DB. Document can have inner array of objects. So model look like this:
public class Document
{
public string Id { get; set; }
public IList<InnerDocument> InnerDocuments { get; set; }
}
public class InnerDocument
{
public string Type { get; set; }
public string Created { get; set; }
}
I need to get all inner documents if at least one of them has certain type.
If I create query like this:
var innerDocument = new InnerDocument()
{
Type = "foo"
};
context.CreateDocumentQuery<Document>(uri, feedOptions)
.Where(d => d.id == "sample" && d.InnerDocuments.Contains(innerDocument));
it translate like this:
SELECT * FROM root
WHERE (root[\"id\"] = "sample"
AND ARRAY_CONTAINS(root[\"innerDocuments\"], {\"type\":\"foo\"}))
but it returns nothing, because no inner document look like this (all inner documents has also Created) so I need to add third parameter to ARRAY_CONTAINS (which tell that only part match on document is enough) so it should look like this:
SELECT * FROM root
WHERE (root[\"id\"] = "sample"
AND ARRAY_CONTAINS(root[\"innerDocuments\"], {\"type\":\"foo\"}, true))
My problem is that I did not figure out how to pass third parameter in linq. I also tried write IEqualityComparer, which always return true but with no effect (well efect was that I got exception..).
Do you have any idea how could I pass that param in linq?
Thanks.

as far as I know, unfortunately there is no LINQ equivalent for the ARRAY_CONTAINS (<arr_expr>, <expr> , bool_expr) overload. To achieve your scenarios, for now you can use SQL query. We are currently working on a set of changes that will enable LINQ for this scenario.
Edit: the available alternative is to use the Any operator with the filters on the property you want to match. For example, the SQL filter: ARRAY_CONTAINS(root.addresses, {"city": "Redmond"}, TRUE) is equivalent to this LINQ expression: addresses.Any(address => address.city == "Redmond")

If I understand correctly, you wish to retrieve all documents that have any inner document in the array with a given property value ("foo" in this example).
Normally, you would use .Where(d => d.InnerDocuments.Any(i => i.Type == "foo")), but Any is not supported yet by the Cosmos LINQ provider.
Instead, you can use this construct as a work-around:
context.CreateDocumentQuery<Document>(uri, feedOptions)
.Where(d => d.Id == "sample")
.SelectMany(d => d.InnerDocuments.Where(i => i.Type == "foo").Select(i => d));
According to this thread Microsoft has recently started working on a real Any feature for the Cosmos LINQ provider.

My solution was slightly more of a hack than a solution, but it works temporarily until the full functionality for .Any() exists.
I use Expressions to dynamically build the Where predicate for my documents, allowing me pass in a CosmosSearchCriteria object which has a list of CosmosCriteria objects as below:
public class CosmosCriteria
{
public CosmosCriteria()
{
ContainsValues = new List<string>();
}
public CosmosCriteriaType CriteriaType { get; set; }
public string PropertyName { get; set; }
public string PropertyValue { get; set; }
public ConvertedRuleComparitor Comparitor { get; set; }
public DateRange Dates { get; set; }
public List<string> ContainsValues { get; set; }
}
This allows me to query any property of the Contact model by essentially passing in the PropertyName and PropertyValue.
I haven't looked into the other workaround in here to see if I can make it work with my expression tree building, at the minute I can't afford the time to investigate.
public async Task<CosmosSearchResponse<Model.Contact>>
GetContactsBySearchCriteriaAsync(int pageSize, long companyId,
CosmosSearchCriteria searchCriteria, string continuationToken = null)
{
var collectionName = CreateCollectionName(companyId, Constants.CollectionType.Contacts);
var feedOptions = new FeedOptions { MaxItemCount = pageSize };
if (!String.IsNullOrEmpty(continuationToken))
{
feedOptions.RequestContinuation = continuationToken;
}
var collection = UriFactory.CreateDocumentCollectionUri(
Configuration.GetValue<string>(Constants.Settings.COSMOS_DATABASE_SETTING),
collectionName);
IOrderedQueryable<Model.Contact> documents = Client.CreateDocumentQuery<Model.Contact>(
collection,
feedOptions
);
documents = (IOrderedQueryable<Model.Contact>)documents.Where(document => document.deleted != true);
bool requiresConcatenation = false;
foreach (var criteria in searchCriteria.Criteria)
{
switch (criteria.CriteriaType)
{
case Constants.CosmosCriteriaType.ContactProperty:
// This is where predicates for the documents.Where(xxxx)
// clauses are built dynamically with Expressions.
documents = AddContactPropertyClauses(documents, criteria);
break;
case Constants.CosmosCriteriaType.PushCampaignHistory:
requiresConcatenation = true;
break;
}
}
documents = (IOrderedQueryable<Model.Contact>)documents.AsDocumentQuery();
/*
From this point onwards, we have to do some wizardry to get around the fact that there is no Linq to SQL
extension overload for the Cosmos DB function ARRAY_CONTAINS (<arr_expr>, <expr> , bool_expr).
The feature is planned for development but is not yet ready.
Keep an eye on the following for updates:
https://stackoverflow.com/questions/52412557/cosmos-db-use-array-contains-in-linq
https://feedback.azure.com/forums/263030-azure-cosmos-db/suggestions/11503872-support-linq-any-or-where-for-child-object-collect
*/
if (requiresConcatenation)
{
var sqlString = documents.ToString();
var jsonDoc = JsonConvert.DeserializeObject<dynamic>(sqlString); // Have to do this to remove the escaping
var q = (string)jsonDoc.query;
var queryRootAlias = Util.GetAliasNameFromQuery(q);
if (queryRootAlias == string.Empty)
{
throw new FormatException("Unable to parse root alias from query.");
}
foreach (var criteria in searchCriteria.Criteria)
{
switch (criteria.CriteriaType)
{
case Constants.CosmosCriteriaType.PushCampaignHistory:
q += string.Format(" AND ARRAY_CONTAINS({0}[\"CampaignHistory\"], {{\"CampaignType\":1,\"CampaignId\":{1}, \"IsOpened\": true }}, true) ", queryRootAlias, criteria.PropertyValue);
break;
}
}
documents = (IOrderedQueryable<Model.Contact>)Client.CreateDocumentQuery<Model.Contact>(
collection,
q,
feedOptions
).AsDocumentQuery();
}
var returnValue = new CosmosSearchResponse<Model.Contact>();
returnValue.Results = new List<Model.Contact>();
Console.WriteLine(documents.ToString());
var resultsPage = await ((IDocumentQuery<Model.Contact>)documents).ExecuteNextAsync<Model.Contact>();
returnValue.Results.AddRange(resultsPage);
if (((IDocumentQuery<Model.Contact>)documents).HasMoreResults)
{
returnValue.ContinuationToken = resultsPage.ResponseContinuation;
}
return returnValue;
}
Hope this helps, or if someone has a better way, please do tell!
Dave

Fluent Validation changing CustomAsync to MustAsync

Could some one please help me to resolved this? i'm trying to change CustomAsync to MustAsync, but i couldn't make things to work. Below is my custom method
RuleFor(o => o).MustAsync(o => {
return CheckIdNumberAlreadyExist(o)
});
private static async Task<ValidationFailure> CheckIdNumberAlreadyExist(SaveProxyCommand command)
{
if (command.Id > 0)
return null;
using (IDbConnection connection = new SqlConnection(ConnectionSettings.LicensingConnectionString))
{
var param = new DynamicParameters();
param.Add("#idnumber", command.IdNumber);
var vehicle = await connection.QueryFirstOrDefaultAsync<dynamic>("new_checkDuplicateProxyIdNumber", param, commandType: CommandType.StoredProcedure);
return vehicle != null
? new ValidationFailure("IdNumber", "Id Number Already Exist")
: null;
}
}

To make it work with the latest version of the FluentValidation, I had to use the codes like below.
RuleFor(ws => ws).MustAsync((x, cancellation) => UserHasAccess(x)).WithMessage("User doesn't have access to perform this action");
Please notice the lambda expression here MustAsync((x, cancellation) => UserHasAccess(x)), without this I was always getting an error as cannot convert from 'method group' to 'Func<Worksheet, CancellationToken, Task<bool>>
Below is my custom UserHasAccess function.
private async Task <bool> UserHasAccess(Worksheet worksheet) {
var permissionObject = await _dataProviderService.GetItemAsync(worksheet.FileItemId);
if (permissionObject is null) return false;
if (EditAccess(permissionObject.Permission)) return true;
return false;
}

I'm assuming you're using a version of FluentValidation prior to version 6, as you're not passing in a Continuation Token, so I've based my answer on version 5.6.2.
Your example code does not compile, for starters, as you're missing a semi-colon in your actual rule. You are also evaluating two different properties on the SaveProxyCommand parameter.
I've built a very small POC based on some assumptions:
Given 2 classes:
public class SaveProxyCommand {
public int Id { get; set; }
}
public class ValidationFailure {
public string PropertyName { get; }
public string Message { get; }
public ValidationFailure(string propertyName, string message){
Message = message;
PropertyName = propertyName;
}
}
And a validator:
public class SaveProxyCommandValidator : AbstractValidator<SaveProxyCommand>{
public SaveProxyCommandValidator()
{
RuleFor(o => o).MustAsync(CheckIdNumberAlreadyExists)
.WithName("Id")
.WithState(o => new ValidationFailure(nameof(o.IdNumber), "Id Number Already Exist"));
}
private static async Task<bool> CheckIdNumberAlreadyExists(SaveProxyCommand command) {
if (command.Id > 0)
return true;
var existingIdNumbers = new[] {
1, 2, 3, 4
};
// This is a fudge, but you'd make your db call here
var isNewNumber = !(await Task.FromResult(existingIdNumbers.Contains(command.IdNumber)));
return isNewNumber;
}
}
I didn't include the call to the database, as that's not part of your problem. There are a couple of things of note here:
You're not setting the .WithName annotation method, but when you're setting up a validation rule for an object you have to do this, as FluentValidation expects you to specify specific properties to be validated by default, if you pass in an entire object it just doesn't know how to report errors back.
Must/MustAsync need to return a bool/Task<bool> instead of a custom object. To get around this, you can specify a custom state to be returned when failing validation.
You can then get access to this like this:
var sut = new SaveProxyCommand { Id = 0, IdNumber = 3 };
var validator = new SaveProxyCommandValidator();
var result = validator.ValidateAsync(sut).GetAwaiter().GetResult();
var ValidationFailures = result.Errors?.Select(s => s.CustomState).Cast<ValidationFailure>();
The above does not take into account empty collections, it's just an example of how to dig into the object graph to retrieve custom state.
As a suggestion, fluentvalidation works best if you set up individual rules per property, instead of validating the entire object. My take on this would be something like this:
public class SaveProxyCommandValidator : AbstractValidator<SaveProxyCommand>{
public SaveProxyCommandValidator()
{
RuleFor(o => o.IdNumber).MustAsync(CheckIdNumberAlreadyExists)
.Unless(o => o.Id > 0)
.WithState(o => new ValidationFailure(nameof(o.IdNumber), "Id Number Already Exist"));
}
private static async Task<bool> CheckIdNumberAlreadyExists(int numberToEvaluate) {
var existingIdNumbers = new[] {
1, 2, 3, 4
};
// This is a fudge, but you'd make your db call here
var isNewNumber = !(await Task.FromResult(existingIdNumbers.Contains(numberToEvaluate)));
return isNewNumber;
}
}
This read more like a narrative, it uses the .Unless construct to only run the rule if Id is not more than 0, and does not require the evaluation of the entire object.

RavenDB - stream index query results in exception

We're currently trying to use the Task<IAsyncEnumerator<StreamResult<T>>> StreamAsync<T>(IQueryable<T> query, CancellationToken token = null), running into some issues.
Our document look something like:
public class Entity
{
public string Id { get; set; }
public DateTime Created { get; set; }
public Geolocation Geolocation { get; set; }
public string Description { get; set; }
public IList<string> SubEntities { get; set; }
public Entity()
{
this.Id = Guid.NewGuid().ToString();
this.Created = DateTime.UtcNow;
}
}
In combination we've a view model, which is also the model were indexing:
public class EntityViewModel
{
public string Id { get; set; }
public DateTime Created { get; set; }
public Geolocation Geolocation { get; set; }
public string Description { get; set; }
public IList<SubEntity> SubEntities { get; set; }
}
And ofcourse, the index, with the resulttype inheriting from the viewmodel, to enable that SubEntities are mapped and output correctly, while enabling the addition of searchfeatures such as fulltext etc.:
public class EntityWithSubentitiesIndex : AbstractIndexCreationTask<Entity, EntityWithSubentitiesIndex.Result>
{
public class Result : EntityViewModel
{
public string Fulltext { get; set; }
}
public EntityWithSubentitiesIndex ()
{
Map = entities => from entity in entities
select new
{
Id = entity.Id,
Created = entity.Created,
Geolocation = entity.Geolocation,
SubEntities = entity.SubEntities.Select(x => LoadDocument<SubEntity>(x)),
Fulltext = new[]
{
entity.Description
}.Concat(entity.SubEntities.Select(x => LoadDocument<SubEntity>(x).Name)),
__ = SpatialGenerate("__geolokation", entity.Geolocation.Lat, entity.Geolocation.Lon)
};
Index(x => x.Created.Date, FieldIndexing.Analyzed);
Index(x => x.Fulltext, FieldIndexing.Analyzed);
Spatial("__geolokation", x => x.Cartesian.BoundingBoxIndex());
}
}
Finally we're querying like this:
var query = _ravenSession.Query<EntityWithSubentitiesIndex.Result, EntityWithSubentitiesIndex>()
.Customize(c =>
{
if (filter.Boundary == null) return;
var wkt = filter.Boundary.GenerateWkt().Result;
if (!string.IsNullOrWhiteSpace(wkt))
{
c.RelatesToShape("__geolokation", wkt, SpatialRelation.Within);
}
})
.AsQueryable();
// (...) and several other filters here, removed for clarity
var enumerator = await _ravenSession.Advanced.StreamAsync(query);
var list = new List<EntityViewModel>();
while (await enumerator.MoveNextAsync())
{
list.Add(enumerator.Current.Document);
}
When doing so we're getting the following exception:
System.InvalidOperationException: The query results type is 'Entity'
but you expected to get results of type 'Result'. If you want to
return a projection, you should use
.ProjectFromIndexFieldsInto() (for Query) or
.SelectFields() (for DocumentQuery) before calling to
.ToList().
According to the documentation, the Streaming API should support streaming via an index, and querying via an IQueryable at once.
How can this be fixed, while still using an index, and the streaming API, to:
Prevent having to page through the normal query, to work around the default pagesize
Prevent having to load the subentities one at a time when querying
Thanks in advance!

Try to use:
.As<Entity>()
(or .OfType<Entity>()) in your query. That should work in the regular stream.
This is a simple streaming query using "TestIndex" that is an index over an entity Test and I'm using a TestIndex.Result to look like your query. Note that this is actually not what the query will return, it's only there so you can write typed queries (ie. .Where(x => x.SomethingMapped == something))
var queryable = session.Query<TestIndex.Result, TestIndex>()
.Customize(c =>
{
//do stuff
})
.As<Test>();
var enumerator = session.Advanced.Stream(queryable);
while (enumerator.MoveNext())
{
var entity = enumerator.Current.Document;
}
If you instead want to retrieve the values from the index and not the actual entity being indexed you have to store those as fields and then project them into a "view model" that matches your mapped properties. This can be done by using .ProjectFromIndexFieldsInto<T>() in your query. All the stored fields from the index will be mapped to the model you specify.
Hope this helps (and makes sense)!
Edit: Updated with a, for me, working example of the Streaming API used with ProjectFromIndexFieldsInto<T>() that returns more than 128 records.
using (var session = store.OpenAsyncSession())
{
var queryable = session.Query<Customers_ByName.QueryModel, Customers_ByName>()
.Customize(c =>
{
//just to do some customization to look more like OP's query
c.RandomOrdering();
})
.ProjectFromIndexFieldsInto<CustomerViewModel>();
var enumerator = await session.Advanced.StreamAsync(queryable);
var customerViewModels = new List<CustomerViewModel>();
while (await enumerator.MoveNextAsync())
{
customerViewModels.Add(enumerator.Current.Document);
}
Console.WriteLine(customerViewModels.Count); //in my case 504
}
The above code works great for me. The index has one property mapped (name) and that property is stored. This is running the latest stable build (3.0.3800).

As #nicolai-heilbuth stated in the comments to #jens-pettersson's answer, it seems to be a bug in the RavenDB client libraries from version 3 onwards.
Bug report filed here: http://issues.hibernatingrhinos.com/issue/RavenDB-3916