Develop Reference

C# MongoDB - Filtering nested array data

i am new on MongoDB and i am trying to use it in C# context. Let´s say, i have documents like this:
[
{
"Number": "2140007529",
"Name": "ABC",
"IsInactive": true,
"EntryList": [
{
"Timestamp": "2022-06-01T14:00:00.000+00:00",
"Value": 21564.0
},
{
"Timestamp": "2022-07-01T21:31:00.000+00:00",
"Value": 21568.0
},
{
"Timestamp": "2022-08-02T21:21:00.000+00:00",
"Value": 21581.642
},
{
"Timestamp": "2022-09-02T15:42:00.000+00:00",
"Value": 21593.551
},
{
"Timestamp": "2022-09-26T13:00:00.000+00:00",
"Value": 21603
}
]
},
{
"Number": "2220000784",
"Name": "XYZ",
"IsInactive": false,
"EntryList": [
{
"Timestamp": "2022-09-26T13:00:00.000+00:00",
"Value": 0.0
},
{
"Timestamp": "2022-10-01T08:49:00.000+00:00",
"Value": 5.274
},
{
"Timestamp": "2022-11-01T09:56:00.000+00:00",
"Value": 76.753
},
{
"Timestamp": "2022-12-01T19:43:00.000+00:00",
"Value": 244.877
},
{
"Timestamp": "2023-01-01T11:54:00.000+00:00",
"Value": 528.56
},
{
"Timestamp": "2023-02-01T17:21:00.000+00:00",
"Value": 802.264
}
]
}
]
I want to get the document where the IsInactive flag is false. But for the EntryList there should be returned entries greater than Timestamp 2022-12-31 only.I should look like this:
{
"Number": "2220000784",
"Name": "XYZ",
"IsInactive": false,
"EntryList": [
{
"Timestamp": "2023-01-01T11:54:00.000+00:00",
"Value": 528.56
},
{
"Timestamp": "2023-02-01T17:21:00.000+00:00",
"Value": 802.264
}
]
}
So, here is my question. How can i filter nested arrays in return value with C#. Thanks for help!
I tried to get the result with aggregate function of MongoDB in MongoDB Compass. I got it work with but not in C#.

I think you are looking for a query similar to this one.
So you can try something like this code:
var desiredTimestamp = new DateTime(2022, 12, 31);
var results = collection.AsQueryable()
.Where(x => x.IsInactive == false && x.EntryList.Any(e => e.Timestamp >= desiredTimestamp))
.Select(obj => new
{
Number = obj.Number,
Name = obj.Name,
IsInactive = obj.IsInactive,
EntryList = obj.EntryList
.Where(e => e.Timestamp >= desiredTimestamp)
.ToList()
}).ToList()
Note that I'm assuming your Timestamp is a date type, otherwise you can't compare date and string.

How to order by group "row_number" in ElasticSearch

I have a database with Products and each product has an Id, Name, ManufacturerId, CategoryId and UserScore.
I want to retrieve all Products by a given Category sorted by UserScore, but avoiding many products of same Manufacturer listed together.
With the following query they all stuck together:
SELECT
P.ProductId, P.Name, P.ManufacturerId, P.UserScore
FROM Products P
WHERE P.CategoryId = 1
ORDER BY P.UserScore
This is the result in T-SQL
In T-SQL I came up with a solution like the following, where Products are grouped in no more than 2 elements by Manufacturer, and it suits perfectly my needs:
SELECT T.*
FROM (
SELECT
P.ProductId, P.Name, P.ManufacturerId, P.UserScore,
ROW_NUMBER() OVER (PARTITION BY P.ManufacturerId ORDER BY P.UserScore DESC) RN
FROM Products P
WHERE P.CategoryId = 1
) T
ORDER BY T.UserScore / CEILING(RN/2.0) DESC
How could I implement a ElasticSearch Query to mimic this behaviour?
Any ideas?
The index in elasticsearch would be like this, this is just an abstract example:
{"ProductId": "157072", "Name": "Product 157072", "ManufacturerId": "7790", "UserScore": "100000", "CategoryId": "1"},
{"ProductId": "296881", "Name": "Product 296881", "ManufacturerId": "6921", "UserScore": "35400", "CategoryId": "1"},
{"ProductId": "353924", "Name": "Product 353924", "ManufacturerId": "54616", "UserScore": "25000", "CategoryId": "1"},
...

You can use the collapse search function to group all the manufacturers:
https://www.elastic.co/guide/en/elasticsearch/reference/current/collapse-search-results.html
Visit "inner_hits" to control the collapsed results behavior.
# Indexing Documents
POST test_so/_bulk
{ "index" : {} }
{"ProductId": "157072", "Name": "Product 157072", "ManufacturerId": "7790", "UserScore": 100000, "CategoryId": "1"}
{ "index" : {} }
{"ProductId": "296881", "Name": "Product 296881", "ManufacturerId": "6921", "UserScore": 35400, "CategoryId": "1"}
{ "index" : {} }
{"ProductId": "353924", "Name": "Product 353924", "ManufacturerId": "54616", "UserScore": 25000, "CategoryId": "1"}
# Filtering by Category: 1, collapsing by Manufacturer and sorting by UserScore
POST test_so/_search
{
"query": {
"term": {
"CategoryId.keyword": {
"value": "1"
}
}
},
"collapse": {
"field": "ManufacturerId.keyword"
},
"sort": [
{
"UserScore": {
"order": "desc"
}
}
]
}
Results
{
"took": 22,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": null,
"hits": [
{
"_index": "test_so",
"_id": "0amBPYQBJRm5qR4vd6NE",
"_score": null,
"_source": {
"ProductId": "157072",
"Name": "Product 157072",
"ManufacturerId": "7790",
"UserScore": 100000,
"CategoryId": "1"
},
"fields": {
"ManufacturerId.keyword": [
"7790"
]
},
"sort": [
100000
]
},
{
"_index": "test_so",
"_id": "0qmBPYQBJRm5qR4vd6NE",
"_score": null,
"_source": {
"ProductId": "296881",
"Name": "Product 296881",
"ManufacturerId": "6921",
"UserScore": 35400,
"CategoryId": "1"
},
"fields": {
"ManufacturerId.keyword": [
"6921"
]
},
"sort": [
35400
]
},
{
"_index": "test_so",
"_id": "06mBPYQBJRm5qR4vd6NE",
"_score": null,
"_source": {
"ProductId": "353924",
"Name": "Product 353924",
"ManufacturerId": "54616",
"UserScore": 25000,
"CategoryId": "1"
},
"fields": {
"ManufacturerId.keyword": [
"54616"
]
},
"sort": [
25000
]
}
]
}
}

Try following which is assuming all items in Group have same values. So I used First()
var results = products.Where(x => x.CategoryId == 1)
.OrderByDescending(x => x.UserScore)
.GroupBy(x => x.ManufacturerId)
.Select(x => new {ProductId = x.ProductId.First(), Name = x.Name.First(), ManufacturerId = x.Key, UserScore = x.UserScore.First()})

How to make GroupBy Table return only One value in Linq

Hi so I have 2 Table with one to many Relationship, I use ado net and Join query to get the Table and the referenced one, after that I cast my DataTable as Enumerable and then use the Linq operation to return it as nested JSON. But the problem is only the GroupBy field will return one data using the key and every else will return an array as many as the record have like this
{
"id": 23,
"date": [
"2018-01-01T00:00:00",
"2018-01-01T00:00:00",
"2018-01-01T00:00:00",
"2018-01-01T00:00:00"
],
"total_room_sold": [
41,
41,
41,
41
],
"total_rom_revenue": [
19340082,
19340082,
19340082,
19340082
],
"Segment": {
"segment_name": [
"BFR",
"DIS",
"PAR",
"LON"
],
"room_sold": [
4,
2,
1,
0
],
"revenue_by_segment": [
1904628,
686605,
461157,
0
]
}
}
And for the N:1 Table do I need to GroupBy again so I can get like this for the segment
{
"id": 23,
"date": [
"2018-01-01T00:00:00",
"2018-01-01T00:00:00",
"2018-01-01T00:00:00",
"2018-01-01T00:00:00"
],
"total_room_sold": [
41,
41,
41,
41
],
"total_rom_revenue": [
19340082,
19340082,
19340082,
19340082
],
"Segment": [
{
"segment_name": "BFR",
"rooom_sold" : 4,
"revenue_by_segment" : 412313213
},
{
"segment_name": "BFR",
"rooom_sold" : 2,
"revenue_by_segment" : 12312313
}
]
}
My code for now
var dt = dt.AsEnumerable().GroupBy(x => x.Field<dynamic>(id)).Select(x => new {
id = x.key,
date = x.Field<dynamic>("date"),
total_room_sold = x.Field<dynami>(total_room_sold),
Segment = x.Select(s => new {
segment_name = s.Field<dynamic>("segment_name"),
room_sold = s.Field<dynamic>("room_sold"),
})
})

How do I get UNIQUE categories from all documents in CosmosDB?

I have millions of documents in CosmosDB using SQL API, and I need to find the unique categories from all documents.
The documents looks like follows, you can see the categories array just under the description, I dont care in what order they are I just need to know all the unique ones from all documents in the collection, I need this so that later on I can create queries on the categories but thats a later question I first need to get them all out so I know what all the possible options are, but I am unable to figure out the query to do this so that I get only the category names.
{
"id": "56d934d3-90bf-4f5a-b602-e515fefa599f",
"_id": "5bf6705f9568cf00013cd13c",
"vendor": "XXX",
"updatedAt": "2018-11-23T03:55:30.044Z",
"locales": [
{
"title": "Cold shoulder t-shirt",
"description": "Because collar bones. Trending cold shoulder t-shirt in 100% organic cotton. Classic, wide and boxy t-shirt fit with cut-out details. In black, because black tees and fashion are like this (insert friendly hand gesture). This style is online exclusive.",
"categories": [
"Women",
"clothing",
"tops"
],
"brand": null,
"images": [
"https://lp.xxx.com/app002prod?set=source[01_0659881_001_102],type[ECOMLOOK],device[hdpi],quality[80],ImageVersion[2018081]&call=url[file:/product/main]",
"https://lp.xxx.com/app002prod?set=source[01_0659881_001_203],type[ECOMLOOK],device[hdpi],quality[80],ImageVersion[2018081]&call=url[file:/product/main]",
"https://lp.xxx.com/app002prod?set=source[01_0659881_001_301],type[ECOMLOOK],device[hdpi],quality[80],ImageVersion[2018081]&call=url[file:/product/main]",
"https://lp.xxx.com/app002prod?set=source[02_0659881_001_101],type[PRODUCT],device[hdpi],quality[80],ImageVersion[1.0]&call=url[file:/product/main]"
],
"country": "SE",
"currency": "SEK",
"language": "en",
"variants": [
{
"artno": "0659881001",
"urls": [
"https://click.linksynergy.com/link?id=INtcw3sexSw&offerid=491018&type=2&murl=https%3A%2F%2Fwww.xxx.com%2Fen_sek%2Fclothing%2Ftops%2Fproduct.cold-shoulder-t-shirt-black-magic.0659881001.html"
],
"price": 80,
"stock": 0,
"attributes": {
"size": "XXS",
"color": "Black magic"
}
},
{
"artno": "xxx",
"urls": [
"https://click.linksynergy.com/link?id=INtcw3sexSw&offerid=491018&type=2&murl=https%3A%2F%2Fwww.xxx.com%2Fen_sek%2Fclothing%2Ftops%2Fproduct.cold-shoulder-t-shirt-black-magic.0659881001.html"
],
"price": 80,
"stock": 0,
"attributes": {
"size": "XS",
"color": "Black magic"
}
},
{
"artno": "0659881001",
"urls": [
"https://click.linksynergy.com/link?id=INtcw3sexSw&offerid=491018&type=2&murl=https%3A%2F%2Fwww.xxx.com%2Fen_sek%2Fclothing%2Ftops%2Fproduct.cold-shoulder-t-shirt-black-magic.0659881001.html"
],
"price": 80,
"stock": 0,
"attributes": {
"size": "XL",
"color": "Black magic"
}
},
{
"artno": "0659881001",
"urls": [
"https://click.linksynergy.com/link?id=INtcw3sexSw&offerid=491018&type=2&murl=https%3A%2F%2Fwww.xxx.com%2Fen_sek%2Fclothing%2Ftops%2Fproduct.cold-shoulder-t-shirt-black-magic.0659881001.html"
],
"price": 80,
"stock": 0,
"attributes": {
"size": "S",
"color": "Black magic"
}
},
{
"artno": "0659881001",
"urls": [
"https://click.linksynergy.com/link?id=INtcw3sexSw&offerid=491018&type=2&murl=https%3A%2F%2Fwww.xxx.com%2Fen_sek%2Fclothing%2Ftops%2Fproduct.cold-shoulder-t-shirt-black-magic.0659881001.html"
],
"price": 80,
"stock": 1,
"attributes": {
"size": "M",
"color": "Black magic"
}
},
{
"artno": "0659881001",
"urls": [
"https://click.linksynergy.com/link?id=INtcw3sexSw&offerid=491018&type=2&murl=https%3A%2F%2Fwww.xxx.com%2Fen_sek%2Fclothing%2Ftops%2Fproduct.cold-shoulder-t-shirt-black-magic.0659881001.html"
],
"price": 80,
"stock": 0,
"attributes": {
"size": "L",
"color": "Black magic"
}
}
]
}
],
"_rid": "QEwcALNbIz8GAAAAAAAAAA==",
"_self": "dbs/QEwcAA==/colls/QEwcALNbIz8=/docs/QEwcALNbIz8GAAAAAAAAAA==/",
"_etag": "\"6a0003c6-0000-0000-0000-5bf7958c0000\"",
"_attachments": "attachments/",
"_ts": 1542952332
}

Please see my test, it could get all the unique categories names.
Sample document:
[
{
"id": "1",
"locales": [
{
"categories": [
"Women",
"clothing",
"tops"
]
}
]
},
{
"id": "2",
"locales": [
{
"categories": [
"Men",
"test",
"tops"
]
}
]
}
]
SQL:
SELECT distinct cat FROM c
join l in c.locales
join cat in l.categories
Output:
[
{
"cat": "Women"
},
{
"cat": "clothing"
},
{
"cat": "tops"
},
{
"cat": "Men"
},
{
"cat": "test"
}
]
If you don't want to case sensitive,just use LOWER function in sql.
SELECT distinct Lower(cat) FROM c
join l in c.locales
join cat in l.categories
If you want to get ["Women","clothing","tops","Men","test"], it can't be parsed as an array in single sql directly, you could use stored procedure to parse the output array.
For example, add below code in stored procedure.
var returnArray = [];
for(var i=0 ;i<array.size;i++){
returnArray.push(array[i].value)
}
return returnArray;

Multiple max/min aggregation in ElasticSearch (C# Nest)

I have been searching here and I didn't find anything similar... However I apologize in advanced if it have escaped me, and I hope you can help out finding the correct direction.
I was looking for a way to implement the following in NEST C#:
"aggs": {
"sys_created_on_max": {
"max": {
"field": "sys_created_on"
}
},
"sys_created_on_min":{
"min": {
"field": "sys_created_on"
}
},
"sys_updated_on_max": {
"max": {
"field": "sys_updated_on"
}
},
"sys_updated_on_min":{
"min": {
"field": "sys_updated_on"
}
}
}
Meaning that I want to perform, in the same statement:
Max and Min aggregated value for "sys_created_on" field
and also
Max and Min aggregated value for "sys_updated_on" field
Thanks!

What you want is Stats Aggregation.
Here is an example input/output
INPUT
GET devdev/redemption/_search
{
"size": 0,
"aggs": {
"a1": {
"stats": {
"field": "reporting.campaign.endDate"
}
}
}
}
Result
{
"took": 97,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 146,
"max_score": 0,
"hits": []
},
"aggregations": {
"a1": {
"count": 11,
"min": 1443675599999,
"max": 1446353999999,
"avg": 1445607818180.818,
"sum": 15901685999989,
"min_as_string": "1443675599999",
"max_as_string": "1446353999999",
"avg_as_string": "1445607818180",
"sum_as_string": "15901685999989"
}
}
}

I've figured it out. In case of someone have the same doubt:
1) create a AggregationContainerDescriptor:
Func<AggregationContainerDescriptor<dynamic>, IAggregationContainer> aggregationsSelector = null;
2) Fill it up:
foreach (var field in requestList)
{
aggregationsSelector += ms => ms.Max(field.MaxAggregationAlias, mx => mx.Field(field.Name))
.Min(field.MinAggregationAlias, mx => mx.Field(field.Name));
}
3) Query it:
var esResponse = _esClient.Raw.Search<dynamic>(indexName.ToLower(), new PostData<dynamic>(jsonStr), null);
Cheers!