Develop Reference

Turkish character problem in elasticsearch

When I search with Turkish characters in elasticsearch, it does not match. For example, when I type "yazilim", the result comes, but when I type "Yazılım", no result. The correct one is "Yazılım".
My index code.
var createIndexDescriptor = new CreateIndexDescriptor(INDEX_NAME).Mappings(ms => ms.Map<T>(m => m.AutoMap()
.Properties(pprops => pprops
.Text(ps => ps
.Name("Title")
.Fielddata(true)
.Fields(f => f
.Keyword(k => k
.Name("keyword")))))
)).Settings(st => st
.Analysis(an => an
.Analyzers(anz => anz
.Custom("tab_delim_analyzer", td => td
.Filters("lowercase", "asciifolding")
.Tokenizer("standard")
)
)
)
);
my search query code.
var searchResponse = eClient.Search<GlobalCompany>(s => s.Index(INDEX_NAME).From(0).Size(10)
.Query(q => q
.MultiMatch(m => m
.Fields(f => f
.Field(u => u.Title)
.Field(u => u.RegisterNumber))
.Type(TextQueryType.PhrasePrefix)
.Query(value))));

You are using an asciifolding filter, it makes sure ASCII characters are used (see docs).

You need to configure your field Title as a text field instead of a keyword field and set the analyzer for this field to tab_delim_analyzer.
I don't know how to translate this in dotNet world but here is what I mean in pure Kibana Dev Console script (curl):
DELETE deneme
PUT deneme
{
"settings": {
"analysis": {
"analyzer": {
"tab_delim_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"asciifolding"
]
}
}
}
},
"mappings": {
"properties": {
"Title": {
"type": "text",
"analyzer": "tab_delim_analyzer"
}
}
}
}

Nest Elasticsearch wildcard query works as querystring but not with fluent API

I have about a hundred test documents in my index, built using NBuilder:
[
{
"title" : "Title1",
"text" : "Text1"
},
{
"title" : "Title2",
"text" : "Text2"
},
{
"title" : "Title3",
"text" : "Text3"
}
]
I want to query them with a wildcard to find all items with "text" starts with "Text". But when I use two wildcard methods in Nest I get two different results.
var response = await client.SearchAsync<FakeFile>(s => s.Query(q => q
.QueryString(d => d.Query("text:Text*")))
.From((page - 1) * pageSize)
.Size(pageSize));
This returns 100 results. But I'm trying to use a fluent API rather than querystring.
var response = await client.SearchAsync<FakeFile>(s => s
.Query(q => q
.Wildcard(c => c
.Field(f => f.Text)
.Value("Text*"))));
This returns 0 results. I'm new to Elasticsearch. I've tried to make the example as simple as possible to make sure I understand it piece-by-piece. I don't know why nothing is returning from the second query. Please help.

Assuming your text field is of type text, then during indexing elasticsearch will store Text1 as text1 internally in the inverted index. Exactly the same analysis will happen when using query string query, but not when you are using wildcard query.
.QueryString(d => d.Query("text:Text*"))) looks for text* and .Wildcard(c => c.Field(f => f.Text).Value("Text*"))) looks for Text* but elasticsearch stores internally only first one.
Hope that helps.

Supposed your mapping looks like that:
{
"mappings": {
"doc": {
"properties": {
"title": {
"type": "text"
},
"text":{
"type": "text"
}
}
}
}
}
Try this (Value should be in lowercase):
var response = await client.SearchAsync<FakeFile>(s => s
.Query(q => q
.Wildcard(c => c
.Field(f => f.Text)
.Value("text*"))));
Or this (don't know if f.Text has text property on it):
var response = await client.SearchAsync<FakeFile>(s => s
.Query(q => q
.Wildcard(c => c
.Field("text")
.Value("text*"))));
Kibana syntax:
GET index/_search
{
"query": {
"wildcard": {
"text": {
"value": "text*"
}
}
}
}

Using .Suffix in SortFieldDescriptor

My ASP.NET Web API has a Search method to search for a specific query in the Elasticsearch database. Also the user can set sort parameters like the property to sort by or if it should be ordered ascending or descending.
http://localhost/api/search?query=jon&sortBy=lastname&sortOrder=desc
The controller passes the request to Elasticsearch using NEST.
var sortProperty = typeof(T).GetProperty(options.SortBy, BindingFlags.IgnoreCase | BindingFlags.Instance | BindingFlags.Public);
var sortOrder = options.SortOrder.IndexOf("desc", System.StringComparison.OrdinalIgnoreCase) >= 0
? SortOrder.Descending
: SortOrder.Ascending;
var result = await this.elasticClient.SearchAsync<Person>(search => search
.Query(q => q.MultiMatch(m => m
.Query(query)))
.Sort(sort => sort.Field(sfd => sfd
.Field(new Field(sortProperty))
.Order(sortOrder)));
The sortProperty can be a text field, like firstname and lastname in this sample. To be able to sort by this text fields, I've added the "raw" keyword fields.
{
"people": {
"mappings": {
"person": {
"properties": {
"birthdate": {
"type": "date"
},
"firstname": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"id": {
"type": "integer"
},
"lastname": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
}
}
}
}
}
}
Now I have to add the suffix "raw" to the properties firstname and lastname.
.Sort(sort => sort.Descending(p => p.Firstname.Suffix("raw")));
But how do I add this to the more generic version I used above, where sortProperty and sortOrder are used to create the SortFieldDescriptor?
Something like the following does not work:
.Sort(sort => sort.Field(sfd => sfd
.Field(new Field(sortProperty).Suffix("raw"))
.Order(sortOrder)));

.Suffix() is intended to be used with member expressions only.
I think you can make this easier by simply using a string to represent the field.
So, instead of
var sortProperty = typeof(T).GetProperty(options.SortBy, BindingFlags.IgnoreCase | BindingFlags.Instance | BindingFlags.Public);
and using PropertyInfo to create an instance of Field, you could use
var sortOrder = options.SortOrder.IndexOf("desc", System.StringComparison.OrdinalIgnoreCase) >= 0
? SortOrder.Descending
: SortOrder.Ascending;
var result = await this.elasticClient.SearchAsync<Person>(search => search
.Query(q => q
.MultiMatch(m => m
.Query(query)
)
)
.Sort(sort => sort
.Field(sfd => sfd
.Field($"{options.SortBy}.raw")
.Order(sortOrder)
)
);
Note that when instantiating a Field from a string, the string value is taken verbatim, so it needs to match the casing of the field in Elasticsearch. In contrast, expressions and PropertyInfo are transformed according to .DefaultFieldNameInferrer() on ConnectionSettings, which will camel case by default.
There are implicit conversions from string, PropertyInfo and Expression<Func<T, object>> to an instance of Field. Take a look at the Field inference documentation.

Elasticsearch can't read ScriptedMetric from BucketScript?

I should work since the ScriptedMetric is a metric, and it does return a single numeric value, but I can't get it to work.
I'm using NEST (5.5.0 via NuGet, I'm targeting Elasticsearch 6.0.0) in C# to get it to work, I did however also try building the same query in Kibana to rule out an issue with NEST. And in Kibana I'm getting exactly the same error.
The error:
buckets_path must reference either a number value or a single value numeric metric aggregation, got: org.elasticsearch.search.aggregations.metrics.scripted.InternalScriptedMetric
My code:
ISearchResponse<LogItem> aggregationResponse = await client.SearchAsync<LogItem>(s => s
.Size(0)
.Type("errordoc")
.Query(...)
.Aggregations(a => a
.Terms("Hash", st => st
.Field(o => o.messageHash.Suffix("keyword")).OrderDescending("Avg-Score")
.Aggregations(aa => aa
.Terms("Friendly", ff => ff
.Field(oo => oo.friendly.Suffix("keyword"))
)
.Max("Max-DateTime", ff => ff
.Field(oo => oo.dateTimeStamp)
)
.Average("Avg-Score", sc => sc
.Script("_score")
)
.ScriptedMetric("Urgency-Level", sm => sm
.InitScript(i => i.Inline("params._agg.data = []").Lang("painless"))
.MapScript(i => i.Inline("params._agg.data.add(doc.urgency.value)").Lang("painless"))
.CombineScript(i => i.Inline("int urgency = 0; for (u in params._agg.data) { urgency += u } return urgency").Lang("painless"))
.ReduceScript(i =>i.Inline("int urgency = 0; for (a in params._aggs) { urgency += a } return urgency").Lang("painless"))
)
.BucketScript("finalScore", scb => scb
.BucketsPath(bp => bp
.Add("maxDateTime", "Max-DateTime")
.Add("avgScore", "Avg-Score")
.Add("urgencyLevel", "Urgency-Level")
)
.Script(i => i.Inline("params.avgScore").Lang("painless"))
)
)
)
)
);
The ScriptedMetric aggregation is returning an 11 with my data set.
Am I doing something wrong? Or Is this not possible? If not possible, what would be an alternative?
Also, I know this ScriptedMetric does pretty much do what a Sum would do, but that's going to change of course...
Index mapping:
PUT /live
{
"mappings": {
"errordoc": {
"properties": {
"urgency": {
"type": "integer"
},
"dateTimeStamp": {
"type": "date",
"format": "MM/dd/yyyy hh:mm:ss a"
} }
}
}
}
Test data:
POST /live/errordoc
{ "messageID": "M111111", "messageHash": "1463454562\/-1210136287\/-1885530817\/-275007043\/-57589585", "friendly": "0", "urgency": "1", "organisation": "Organisation Name", "Environment": "ENV02", "Task": "TASK01", "Action": "A12", "dateTimeStamp": "11\/29\/2017 10:24:21 AM", "machineName": "DESKTOP-SMOM9R9", "parameters": "{ " }
Copy this document a couple of times, maybe changing the urgency/dateTimeStamp, as long as the Hash stays the same it should reproduce my environment...

Somebody on the Elasticsearch forums replied after I created 2 GitHub issues...
I did try everything, including this solution, but I must have done something else wrong during that try... Well whatever...
The solution
Change:
.Add("urgencyLevel", "Urgency-Level")
to:
.Add("urgencyLevel", "Urgency-Level.value")

Searching for an input keyword in all fields of an elasticsearch document using c# nest client

I have a nested elasticsearch document and I want to search within all the fields of that document i.e I want to search in both the top-level and the nested fields. My index name is people and my type name is person.
My documents look like this :
{
"id": 1,
"fname": "elizabeth",
"mname": "nicolas",
"lname": "thomas",
"houseno": "beijing",
"car": [
{
"carname": "audi",
"carno": 4444,
"color": "black"
},
{
"carname": "mercedez",
"carno": 5555,
"color": "pink"
}
]
}
Then i have the following query in .net which actually searches for an user input keyword in the elasticsearch documents. Basically, I want to search in each and every field of a document. And I use inner_hits in my query so that i can return only the matching nested document.
I have designed my query as :
var result = client.Search<person>
(s => s
.From(from)
.Size(size)
.Source(false)
.Query(query => query.Filtered(filtered => filtered
.Query(q => q.MatchAll())
.Filter(f => f.Nested(nf => nf
.InnerHits()
.Path(p => p.car)
.Query(qq => qq.Match(m => m.OnField(g => g.car.First().carname).Query(searchKeyword))))))));
And my corresponding JSON query which i use in the head plugin is :
POST-people/person/_search:
{
"_source":false,
"query": {
"filtered": {
"query": {"match_all": {}},
"filter": {
"nested": {
"path": "car",
"filter": {
"term": {
"car.carname": "searchKeyword"
}
},
"inner_hits" : {}
}
}
}
}
}
But i wanted to search in all the fields(id,fname,mname,lname,houseno,carname,carno,color) and not just in a single field e.g. in carname as i have done in my above query.
Also, i want to do partial searching like %xyz%.
How can i do these ?
Can anyone help me modify this query so that i can use this single query to search within all the fields as well as do partial searching?
I'm new to elasticsearch as well as .net,so I would be thankful for any help.

Did you try using Query instead of OnField method?
I mean, having your query this way:
var result = client.Search<person>
(s => s
.From(from)
.Size(size)
.Source(false)
.Query(query => query.Filtered(filtered => filtered
.Query(q => q.MatchAll())
.Filter(f => f.Nested(nf => nf
.InnerHits()
.Path(p => p.car)
.Query(qq => qq.Match(m => m.Query(searchKeyword))))))));