Develop Reference

Is there a way to list all Kibana dashboards via Nest in c#?

I am able to get a list of dashboards using a simple Get request in something like Postman:
http://localhost:9200/.kibana/_search?q=type:dashboard&size=20
However I need to get the list using the Nest client in C# and I have no idea how to form that sort of search request with the ElasticClient DSL. For more "traditional" searches I would use something like:
var settings = new ConnectionSettings(new Uri("http://localhost:9200/"))
.DefaultMappingFor<ElasticCustomerDTO>(i => i
.IndexName("customer")
.IdProperty(p => p.Identifier)
);
var client = new ElasticClient(settings);
ISearchResponse<ElasticCustomerDTO> searchResponse = client.Search<ElasticCustomerDTO>(s => s
.Query(q => q
.SimpleQueryString(f => f
.Query(filter.Name)
)
)
);
I'm struggling to create anything like the dashboard search query in that form (at least anything that returns results). Any help would be greatly appreciated.

You can use below NEST query to get Kibana dashboards
var searchResponse = await client.SearchAsync<Dictionary<string, object>>(s => s
.Index(".kibana")
.QueryOnQueryString("type:dashboard")
.Size(20));
foreach (var document in searchResponse.Documents)
{
Console.WriteLine(((Dictionary<string, object>)document["dashboard"])["title"]);
}
I'm using Dictionary<string, object> as returned type as I don't think there is any type in NEST representing Kibana DTOs.

Mongo C# filter on integer contains

I've a collection with orders, each with an OrderId field of type int32. I would like to filter the collecting and get all orders where the OrderId contains e.g. 123. For instance the following OrderIds must match:
123
9912399
99123
The following raw mongo filter works:
{ $where: "/^.*123.*/.test(this.OrderId)" }
How can this be expressed in the C# Mongo client or something different that does the job?
I'm not able to use the below code, as it seems to only work in string fields:
Builders<Order>.Filter.Regex(x => x.OrderId, new BsonRegularExpression("123"))

ideally you should store a string version of the order id in your documents. in case that's not an option you can do the following aggregation pipeline:
var res = await collection
.Aggregate()
.AppendStage<object>("{ $set : { orderIdString : { $toString : '$OrderId' } } }")
.Match(Builders<object>.Filter.Regex("orderIdString", "/123/"))
.AppendStage<object>("{ $unset : 'orderIdString' }")
.As<Order>()
.ToListAsync();

I don't think you can generate $where via typed builders. So, as it was mentioned above, the only option you have is to create a filter from a raw MQL like below:
var coll = db.GetCollection<Order>("coll");
var result = coll.Find("{ $where: '/^.*123.*/.test(this.OrderId)' }").ToList();

Did you try using:
x => x.OrderId.ToString()
instead of
x => x.Orderid

You can use Filter.Regex to achieve the desired behavior:
var valueToFilterBy = "123"
var filter = Builders<Order>.Filter.Regex(nameof(Order.Id), new BsonRegularExpression($".*{valueToFilterBy}.*"));
var data = await collection.FindAsync(filter).ToListAsync();

Using PIT (Point in Time) ID and Search_After API -NEST ElasticSearch

I have been reading up on Point in time API and wanted to implement it using NEST in my .net application. However, when reading that article (.net application hyperlink), I saw the Fluent DSL example as shown below. Is there a way to find that ID without having to go to kibana on the console and doing an query search to get the id to then later place that id inside of "a-point-in-time-id"? or does "a-point-in-time-id" do that for you like is that mapped to the ID?
s => s
.PointInTime("a-point-in-time-id", p => p
.KeepAlive("1m"))
I know in the kibana cli console if you do:
POST /app-logs*/_pit?keep_alive=5m it will give you a PIT (point in time) id. How do you go about retrieving that in NEST?
and when reading up on search_after and attempting to implement it using the search after usage for the .net client using the Fluent DSL Example. I noticed that they had they word "Project" but it does not say what "Project is in the example. What would that be exactly?
s => s
.Sort(srt => srt
.Descending(p => p.NumberOfCommits)
.Descending(p => p.Name)
)
.SearchAfter(
Project.First.NumberOfCommits,
Project.First.Name
)
Here I attempted to implement .PointInTime() and .Sort() and .SearchAfter() but got stuck.
var response = await _elasticClient.SearchAsync<Source>(s => s
.Size(3000) // must see about this
.Source(src => src.Includes(i => i
.Fields(f => f.timestamp,
fields => fields.messageTemplate,
fields => fields.message)))
.Index("app-logs*"
.Query(q => +q
.DateRange(dr => dr
.Field("#timestamp")
.GreaterThanOrEquals("2021-06-12T16:39:32.727-05:00")
.LessThanOrEquals(DateTime.Now))))
.PointInTime("", p => p
.KeepAlive("5m"))
.Sort(srt => srt
.Ascending(p => p.timestamp))
.SearchAfter()
I know when you are using PIT ID you do not need the index in the search, but in the hyperlink example it does not show how you would go about implementing that. So just a bit lost on how to do so. Any pointers/guidance/tutorials would be great!
But just trying to see how I can do this in NEST but if you are saying its apart of the XPACK, I would understand.

I was able to combine PIT and SearchAfter with the code sample below:
var pit = string.IsNullOrEmpty(pitId) ? client.OpenPointInTime(new OpenPointInTimeDescriptor().KeepAlive(ServerDetail.TTL).Index(Indices.Index(Name))) : null;
var request = new SearchRequest(Name)
{
SearchAfter = string.IsNullOrEmpty(pitId) ? null : new string[] { last },
Size = ServerDetail.PageSize,
PointInTime = new PointInTime(pit == null ? pitId : pit.Id)
};
List<FieldSort> sorts = new List<FieldSort>();
foreach (var item in sortBy)
{
sorts.Add(new FieldSort() { Field = item.Field, Order = item.Direction == SortDirection.Asc ? SortOrder.Ascending : SortOrder.Descending });
}
request.Sort = sorts.ToArray();
Your SearchAfter value should be values from your sort fields for the last object of your previous search result.
For the 1st search pitId is null so a new one is created, for subsequent requests, pitId is passed.
Hope that works for you?

Elasticsearch query with NEST don't work

I'm using Microsoft SQL Server Management Studio and ElasticSearch 2.3.4 with ElasticSearch-jdbc-2.3.4.1, and i linked ES with my mssql server. Everything works fine, but when i make a query using NEST on my MVC program the result is empty. When i put an empty string inside my search attribute i get the elements, but when i try to fill it with some filter i get an empty result. Can someone help me out please? Thanks in advance.
C#:
const string ESServer = "http://localhost:9200";
ConnectionSettings settings = new ConnectionSettings(new Uri(ESServer));
settings.DefaultIndex("tiky");
settings.MapDefaultTypeNames(map => map.Add(typeof(DAL.Faq), "faq"));
ElasticClient client = new ElasticClient(settings);
var response = client.Search<DAL.Faq>(s => s.Query(q => q.Term(x => x.Question, search)));
var result = response.Documents.ToList();
DAL:
Postman:
PS: i followed this guide to create it
EDIT:
Index mapping:

There's a couple of things that I can see that may help here:
By default, NEST camel cases POCO property names when serializing them as part of the query JSON in the request, so x => x.Question will serialize to "question". Looking at your mapping however, field names in Elasticsearch are Pascal cased, so what the client is doing will not match what's in Elasticsearch.
You can change how NEST serializes POCO property names by using .DefaultFieldNameInferrer(Func<string, string>) on ConnectionSettings
const string ESServer = "http://localhost:9200";
ConnectionSettings settings = new ConnectionSettings(new Uri(ESServer))
.DefaultIndex("tiky");
.MapDefaultTypeNames(map => map.Add(typeof(DAL.Faq), "faq"))
// pass POCO property names through verbatim
.DefaultFieldNameInferrer(s => s);
ElasticClient client = new ElasticClient(settings);
As Rob mentioned in the comments, a term query does not analyze the query input. When executing a term query against a field that is analyzed at index time then, in order to get matches, the query text that you pass to the term query would need to take the analysis that is applied at index time into account. For example,
Question is analyzed with the Standard Analyzer
A Question value of "What's the Question?" will be analyzed and indexed as tokens "what's", "the" and "question"
A term query would need to have a query input of "what's", "the" or "question" to be a match
A match query, unlike a term query, does analyze the query input, so the output of the search analysis will be used to find matches. In conjunction with Pascal casing highlighted in 1., you should now get documents returned.
You can also have the best of both worlds in Elasticsearch i.e. analyze input at index time for full-text search functionality as well as index input without analysis to get exact matches. This is done with multi-fields and here is an example of creating a mapping that indexes Question properties as both analyzed and not analyzed
public class Faq
{
public string Question { get; set; }
}
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var defaultIndex = "default-index";
var connectionSettings = new ConnectionSettings(pool)
.DefaultIndex(defaultIndex)
.DefaultFieldNameInferrer(s => s);
var client = new ElasticClient(connectionSettings);
if (client.IndexExists(defaultIndex).Exists)
client.DeleteIndex(defaultIndex);
client.CreateIndex(defaultIndex, c => c
.Mappings(m => m
.Map<Faq>(mm => mm
// let NEST infer mapping from the POCO
.AutoMap()
// override any inferred mappings explicitly
.Properties(p => p
.String(s => s
.Name(n => n.Question)
.Fields(f => f
.String(ss => ss
.Name("raw")
.NotAnalyzed()
)
)
)
)
)
)
);
The mapping for this looks like
{
"mappings": {
"faq": {
"properties": {
"Question": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}
The "raw" sub field under the "Question" field will index the value of the Question property without any analysis i.e. verbatim. This sub field can now be used in a term query to find exact matches
client.Search<Faq>(s => s
.Query(q => q
.Term(f => f.Question.Suffix("raw"), "What's the Question?")
)
);
which find matches for the previous example.

Nest SuggestCompletion usage, throws 'is not a completion suggest field' exception

I'm a complete beginner to elasticsearch and I have been trying to use elasticsearch's completion suggester using Nest for auto-complete on a property.
Here is my mapping (as mentioned here: ):
var createResult = client.CreateIndex(indexName, index => index
.AddMapping<Contact>(tmd => tmd
.Properties(props => props
.Completion(s =>
s.Name(p => p.CompanyName.Suffix("completion"))
.IndexAnalyzer("standard")
.SearchAnalyzer("standard")
.MaxInputLength(20)
.Payloads()
.PreservePositionIncrements()
.PreserveSeparators())
)
)
);
var resultPerson = client.IndexMany(documents.OfType<Person>(), new SimpleBulkParameters { Refresh = true });
var resultCompany = client.IndexMany(documents.OfType<Company>(), new SimpleBulkParameters { Refresh = true });
And while indexing I'm just making use of IndexMany method and passing the IEnumberable<Contact> (Contact has a property by name CompanyName, Contact is an abstract class, both Person and Company are concrete implementations of it). The search throws an exception saying ElasticSearchException[Field [companyName] is not a completion suggest field]. And the query looks like below:
SearchDescriptor<Contact> descriptor = new SearchDescriptor<Contact>();
descriptor = descriptor.SuggestCompletion("suggest", c => c.OnField(f => f.CompanyName).Text(q));
var result = getElasticClientInstance("contacts").Search<Contact>(body => descriptor);
string qe = result.ConnectionStatus.ToString();
What am I doing wrong here, I looked into Nest's tests on SuggestCompletion but not much help, meaning the test only depict on how to get suggestions but not on how to set index mappings for SuggestCompletion.
I also tried setting up edgeNgram tokenizer as mentioned in this post but, couldn't proceed there as well.
Any direction or an example on how to proceed would greatly help.

UPDATE
You are try to create a property with the name "companyName.completion" but at that position its not valid and it will use the last token "completion". So its actually mapping a field called completion.... try changing the call to: .Name(p => p.CompanyName)
Other observations
You specify a mapping for the Contact but while indexing you use the Person and Company types.
In elasticsearch terms you mapped:
/index/contact/
but your documents are going into:
/index/person/ and /index/company
NEST won't automatically map all implementation of a specific class and elasticsearch has no way of knowing the three are related.
I would refactor the mapping to a method and call it for all the types involved.
var createResult = client.CreateIndex(indexName, index => index
.AddMapping<Contact>(tmd => MapContactCompletionFields(tmd))
.AddMapping<Person>(tmd => MapContactCompletionFields(tmd))
.AddMapping<Company>(tmd => MapContactCompletionFields(tmd))
);
private RootObjectMappingDescriptor<TContact> MapContactCompletionFields<TContact>(
RootObjectMappingDescriptor<TContact> tmd)
where TContact : Contact
{
return tmd.Properties(props => props
.Completion(s => s
.Name(p => p.CompanyName.Suffix("completion"))
.IndexAnalyzer("standard")
.SearchAnalyzer("standard")
.MaxInputLength(20)
.Payloads()
.PreservePositionIncrements()
.PreserveSeparators()
)
);
}
That method returns the descriptor so you can further chain on it.
Then when you do a search for contacts:
var result = getElasticClientInstance("contacts").Search<Contact>(
body => descriptor
.Types(typeof(Person), typeof(Company))
);
That types hint will cause the search to looking /index/person and /index/company and will know how to give you back a covariant list of documents.
So you can do result.Documents.OfType<Person>() after the previous call.