I have the following query in NEST (ElasticSearch C# client), note the nested aggregation:
var query = _elasticClient.Search<Auth5209>(s => s
.Size(0)
.Aggregations(a=> a
.Terms("incidentID", t=> t
.Field(f=>f.IncidentID)
.Size(5)
.Aggregations(a2 => a2
.Stats("authDateStats", s1=>s1.Field(f=>f.AuthEventDate))
)
)
)
);
This correctly generates the following query:
{
"size": 0,
"aggs": {
"incidentID": {
"terms": {
"field": "incidentID",
"size": 5
},
"aggs": {
"authDateStats": {
"stats": {
"field": "authEventDate"
}
}
}
}
}
}
Which gives me the following results:
"aggregations" : {
"incidentID" : {
"buckets" : [{
"key" : "0A631EB1-01EF-DC28-9503-FC28FE695C6D",
"doc_count" : 233,
"authDateStats" : {
"count" : 233,
"min" : 1401167036075,
"max" : 1401168969907,
"avg" : 1401167885682.6782,
"sum" : 326472117364064
}
}
]
}
}
What I can't figure out is how I access the "authDateStats" section. When I debug I don't see any way to access the data.
Neither the official documentation nor the answers here fully work for nest 2.0+. Although the answer from jhilden did get started me down the right path.
Here is a working example of a similar query which can be used with nest 2.0+:
const string termsAggregation = "device_number";
const string topHitsAggregation = "top_hits";
var response = await _elasticsearchClient.Client.SearchAsync<CustomerDeviceModel>(s => s
.Aggregations(a => a
.Terms(termsAggregation, ta => ta
.Field(o => o.DeviceNumber)
.Size(int.MaxValue)
.Aggregations(sa => sa
.TopHits(topHitsAggregation, th => th
.Size(1)
.Sort(x => x.Field(f => f.Modified).Descending())
)
)
)
)
);
if (!response.IsValid)
{
throw new ElasticsearchException(response.DebugInformation);
}
var results = new List<CustomerDeviceModel>();
var terms = response.Aggs.Terms(termsAggregation);
foreach (var bucket in terms.Buckets)
{
var hit = bucket.TopHits(topHitsAggregation);
var device = hit.Documents<CustomerDeviceModel>().First();
results.Add(device);
}
I'm guessing you already figured this out but you can access the nested aggregations, it's just in a base class, you can see it in Nest.KeyItem.base.base.Aggregations in the debugger.
Here is a full working sample of accessing an inner aggregation:
const string aggName = "LocationIDAgg";
const string aggNameTopHits = "LatestForLoc";
var response = await ElasticClient.SearchAsync<PlacementVerificationES>(s => s
.Query(BuildQuery(filter, null))
.Size(int.MaxValue)
.Aggregations(a=> a
.Terms(aggName, t=> t
.Field(f=>f.LocationID)
.Size(100)
.Aggregations(innerAgg => innerAgg
.TopHits(aggNameTopHits, th=> th
.Size(1)
.Sort(x=>x.OnField(f=> f.Date).Descending())
)
)
)
)
).VerifySuccessfulResponse();
//var debug = response.GetRequestString();
var agBucket = (Bucket)response.Aggregations[aggName];
var output = new List<PlacementVerificationForReporting>();
// ReSharper disable once LoopCanBeConvertedToQuery
// ReSharper disable once PossibleInvalidCastExceptionInForeachLoop
foreach (KeyItem i in agBucket.Items)
{
var topHits = (TopHitsMetric)i.Aggregations[aggNameTopHits];
var top1 = topHits.Hits<PlacementVerificationES>().Single();
var reportingObject = RepoToReporting(top1);
output.Add(reportingObject);
}
return output;
Related
I need to return JUST all categories DISTINCTS (without repeat any) from a document using NEST.
In SQL it looks like this:
SELECT DISTINCT Category
FROM Log
ORDER BY Category ASC
Inside ElasticSearch I do this way:
GET log/_search
{
"size":"0",
"aggs" : {
"alias_category" : {
"terms" : { "field" : "category.keyword" }
}
}
}
How can I do that using NEST?
public ICollection<string> SelectAllCategoriesDistinct(ElasticClient client)
{
var searchResponse = client.Search<LogElasticSearch>(s => s
.Query(q => q
.Terms(t => t
.Field(f => f.Category)
)
)
);
return (ICollection<string>)searchResponse;
}
I found a way to do it. Is not a elegant way, but I found in the elastico site (https://discuss.elastic.co/t/c-nest-best-way-of-accessing-properties-of-iaggregate-object/85384/2) and I did this way:
public ICollection<string> SelectAllCategoriesDistinct(ElasticClient client)
{
var searchResponse =
client.Search<LogElasticSearch>(s => s
.Size(0)
.Aggregations(agg => agg
.Terms("categories", t => t
.Field("category.keyword")
)
)
);
var aggregation = searchResponse.Aggregations.Values;
var listOfCategories = new List<string>();
if (searchResponse.Aggregations.Values.FirstOrDefault().GetType() == typeof(BucketAggregate))
{
foreach (IBucket bucket in ((BucketAggregate)aggregation.FirstOrDefault()).Items)
{
if (bucket.GetType() == typeof(KeyedBucket<object>))
{
var valueKey = ((KeyedBucket<object>)bucket).Key;
listOfCategories.Add(valueKey.ToString());
}
}
}
return listOfCategories.OrderBy(c => c).ToList();
}
If someone knows a better way to do it, help me to improve, but that way it reaches the goal.
I would use a composite aggregation to fetch all the terms. In contrast to the terms aggregation, the composite aggregation supports pagination, so multiple requests can be made if there are a lot of terms to fetch
private static void Main()
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(pool);
var client = new ElasticClient(settings);
var categories = new List<string>();
GetAllCategories(client, categories);
// do something with the categories
foreach(var category in categories)
Console.WriteLine(category);
}
private static void GetAllCategories(IElasticClient client, List<string> categories, CompositeKey after = null)
{
// number of terms to fetch in each request
var size = 10_000;
var response = client.Search<LogElasticSearch>(s => s
.Size(0)
.Aggregations(a => a
.Composite("categories", c => c
.After(after)
.Size(size)
.Sources(so => so
.Terms("category", t => t
.Field("category.keyword")
.Order(SortOrder.Ascending)
)
)
)
)
);
var compositeAgg = response.Aggregations.Composite("categories");
var buckets = compositeAgg.Buckets;
foreach (var bucket in buckets)
{
if (bucket.Key.TryGetValue("category", out string category))
{
categories.Add(category);
}
}
// there may be more
if (buckets.Count == size)
{
GetAllCategories(client, categories, compositeAgg.AfterKey);
}
}
I need to convert the following query into c# in using NEST.
"aggs": {
"number_of_weeks": {
"date_histogram": {
"field": "#timestamp",
"interval": "week"
}
}
}
in Kibana the output is
I wrote the following query but it give me zero bucket while in Kibana it return many result in buckets
var query3 = EsClient.Search<doc>(q => q
.Index("SomeIndex")
.Size(0)
.Aggregations(agg => agg.DateHistogram("group_by_week", e => e.Field(p => p.timestamp) .Interval(DateInterval.Week)
)) ;
var resultquery3 = query3.Aggregations.DateHistogram("group_by_week");
in vs studio the output is
The problem is likely that
e => e.Field(p => p.timestamp)
does not serialize to the "#timestamp" field in Elasticsearch. For this to work, you would need to either map it with an attribute on the POCO
public class Doc
{
[Date(Name = "#timestamp")]
public DateTime timestamp { get; set; }
}
or map it on ConnectionSettings
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(pool)
.DefaultMappingFor<Doc>(m => m
.PropertyName(e => e.timestamp, "#timestamp")
);
var client = new ElasticClient(settings);
Alternatively, you can simply pass a string to .Field(), which implicitly converts
.Field("#timestamp")
Can someone please help me translate the following into code for the mongodb 2.4 C# driver? I have been trying to get this to work but I'm having a hard time figuring it out. When I run the following mongo query through the C# drive, I get no records. When I run through the mongo command line I get back 2 records.
I just need to understand what I am doing that is causing the difference in behavior when using the C# driver.
So far I have the following for the C# driver:
public IEnumerable<ComponentRecordDataModel> GetComponentRecords(
)
{
var componentTypeFilter = Builders<ComponentRecordDataModel>.Filter.Eq(x => x.ComponentType, "Investment");
var authorizedUserFilter =
Builders<ComponentRecordDataModel>.Filter.AnyEq(x => x.AuthorizedUserIds, "59e8c1d35e13de1494887658");
var componentFieldFilters = new List<FieldValueDataModel>
{
new FieldValueDataModel()
{
FieldId = "59d664d1c153f67518f98888",
Value = "gov"
},
new FieldValueDataModel()
{
FieldId = "59d664d1c153f67518f98889",
Value = "azure"
}
};
var componentFilters = new List<FilterDefinition<FieldValueDataModel>>();
foreach (var componentFieldFilter in componentFieldFilters)
{
var bsonRegex = new BsonRegularExpression(Regex.Escape(componentFieldFilter.Value), "i");
componentFilters.Add(Builders<FieldValueDataModel>.Filter.And(new[]
{
Builders<FieldValueDataModel>.Filter.Eq(x => x.FieldId, componentFieldFilter.FieldId),
Builders<FieldValueDataModel>.Filter.Eq(x => x.Value, bsonRegex)
}));
}
var fieldFilter = Builders<ComponentRecordDataModel>.Filter.ElemMatch(
x => x.Fields, Builders<FieldValueDataModel>.Filter.Or(componentFilters)
);
var filter = componentTypeFilter & authorizedUserFilter & fieldFilter;
var renderedFilter = filter.ToJson();
return MongoContext.Find(filter).ToList();
}
/The mongodb query I am trying to replication is shown below/
db.getCollection('componentRecords').aggregate(
[{
$match: {
"componentType": "Investment"
}
},
{
$match: {
"authorizedUserIds": {
$elemMatch: {
$in: ["59e8c1d35e13de1494887658"]
}
}
}
},
{
$match: {
"fields": {
$elemMatch: { $or:
[{ $and : [
{ fieldId: ObjectId("59d664d1c153f67518f98888") },
{ value: { "$regex": "gov", '$options' : 'i' } }
]},
{ $and : [
{ fieldId: ObjectId("59d664d1c153f67518f98889") },
{ value: { "$regex": "azure", '$options' : 'i' } }
]}
]
}
}
}
}
])
Have you tried using linq ? it maybe easier to understand and write:
var arr = new [] { "59e8c1d35e13de1494887658" };
var id1 = ObjectId("59d664d1c153f67518f98888");
var gov = "gov";
var id2 = ObjectId("59d664d1c153f67518f98889");
var azure = "azure";
collection.AsQuerable().Where(w =>
w.componentType == "Investment" &&
w.authorizedUserIds.Any(a => arr.Contains(a.X)) &&
(
(w.fieldId == id1 && w.value.ToLower() == go".ToLower()) ||
(w.fieldId == id2 && w.value.ToLower() == azure.ToLower()) ||
)
).ToList();
Actually I didn't try this code in real example, but worked with linq in mongodb.
I setting up an index as below. But now I have a requirement because of which i need to tweak the indexing style. (i have to add analyzer field in the below code).
Reference[My previous question and its answer]: Elastic Search using NEST - Results different in debug and browser mode
How can I rewrite
var connectionSettings = new ConnectionSettings(pool)
.DefaultIndex(defaultIndex)
.MapDefaultTypeNames(m => m.Add(typeof(Class1), "omg"))
.PrettyJson()
.DisableDirectStreaming());
with the mapping settings like below.
{
"mappings": {
"Class1": {
"properties": {
"Answer": {
"type": "string",
"analyzer": "english"
}
}
}
}
}
This is my take on answer:
settings = new ConnectionSettings(pool)
.DefaultIndex(defaultIndex)
.MapDefaultTypeNames(m => m.Add(typeof(Class1), "omg"))
.PrettyJson()
.DisableDirectStreaming();
var descriptor = new CreateIndexDescriptor(defaultIndex)
.Mappings(ms => ms
.Map<Class1>(m => m
.Properties(ps => ps
.String(s=>s
.Name(n=>n.Ans)
.Analyzer("english")))));
I think I'm missing a link somewhere between the index creation and mappings. Though it didnt show an error while coding, the output is not as expected.
TIA
A CreateIndexDecriptor<T> is a descriptor for creating an index, but you need to pass it to the IElasticClient.CreateIndex() method in order to create the index in Elasticsearch.
void Main()
{
var defaultIndex = "default-index";
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(pool, new InMemoryConnection())
.DefaultIndex(defaultIndex)
.MapDefaultTypeNames(m => m.Add(typeof(Class1), "omg"))
.PrettyJson()
.DisableDirectStreaming();
var client = new ElasticClient(settings);
client.CreateIndex("new-index", c => c
.Mappings(ms => ms
.Map<Class1>(m => m
.Properties(ps => ps
.String(s => s
.Name(n => n.Ans)
.Analyzer("english")
)
)
)
)
);
}
public class Class1
{
public string Ans { get; set;}
}
The request to Elasticsearch looks like
{
"mappings": {
"omg": {
"properties": {
"ans": {
"type": "string",
"analyzer": "english"
}
}
}
}
}
I would like to implement the following command through c#. I've seen the Update.PushAll command but I'm not sure it's the right way. Any suggestion?
db.students.update(
{ name: "joe" },
{ $push: { scores: { $each: [ 90, 92, 85 ] } } }
, upsert = true
)
You can use PushAllWrapped to add an array of scores to your existing document:
var collection = db.GetCollection<Student>("students");
var query = Query<Product>.EQ(p => p.Name, "joe");
var push = Update<Student>.PushAllWrapped<int>(p => p.Scores, newScores);
collection.Update(query, push, UpdateFlags.Multi);
Using the new syntax, you can achieve using PushEach:
var collection = db.GetCollection<Student>("students");
var filter = Builders<Students>.Filter.Eq("name", "joe");
var update = Builders<Students>.Update.PushEach<Score>(x=>x.Scores, scores);
await collection.UpdateOneAsync(filter, update);