In my application I pass a boolean parameter to a function that searches certain documents in my elastic index via a HasChildQuery.
If this boolean is set to false I want to exclude documents with a specific field set, when the boolean is set to true I do not want this second condition.
This is my approach so far:
Query = new HasChildQuery
{
// ...
Query = new CommonTermsQuery
{
// This Query always needs to be there
Field = Nest.Infer.Field<FaqQuestion>(q => q.Content),
Query = content
}
&& (includeAutoLearnedData ? null : +new TermQuery
{
// I only want this Query if includeAutoLearnedData is false
Field = Nest.Infer.Field<FaqQuestion>(q => q.AutoLearned),
Value = false
})
}
My idea behind this is to always generate a request like this
has_child
|
|__ ...
|
|__ common_terms
and expand this to
has_child
|
|__ ...
|
|__ bool
|
|__must
| |
| |__common_terms
|
|__filter
|
|__term
if includeAutoLearnedData is false.
But the query for the case when it is true seems to not work.
I hoped that && (includeAutoLearnedData ? null : +new TermQuery will add the filter only when the boolean is false and leave the query unmodified when it is true
So what is the correct way of including an additional filter query under a certain condition in NEST?
EDIT:
I set a breakpoint when I get the result from my ElasticClient and expected it to have something like
Valid NEST response built from a successful low level call on POST: /faq/_search
# Audit trail of this API call:
- [1] HealthyResponse: Node: http://localhost:9200/ Took: 00:00:00.0770000
# Request:
{
"query": {
"has_child": {
"bool": {
"must": [{
"common_terms": { ... }
}],
"filter": [{
"term": { ... }
}]
}
}
}
}
but actual result had:
# Request:
{}
What you have is correct and your approach is sound, but the reason you're seeing {} in the output is because of conditionless queries in NEST; Essentially, if a query does not have certain properties set (or they are assigned null or empty string), then the query is considered conditionless and not serialized as part of the request. For example, for a term query, if
the field has an empty string assigned to it, or a null string, expression or property
the value is null or an empty string
then the term query is considered conditionless. You can change this behaviour using verbatim and strict
Verbatim
Individual queries can be marked as verbatim meaning that the query should be sent to Elasticsearch as is, even if it is conditionless.
Strict
Individual queries can be marked as strict meaning that if they are conditionless, an exception is thrown. This is useful for when a query must have an input value.
To demonstrate that your approach works
void Main()
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var defaultIndex = "default-index";
var connectionSettings = new ConnectionSettings(pool, new InMemoryConnection())
.DefaultIndex(defaultIndex)
.PrettyJson()
.DisableDirectStreaming()
.OnRequestCompleted(response =>
{
if (response.RequestBodyInBytes != null)
{
Console.WriteLine(
$"{response.HttpMethod} {response.Uri} \n" +
$"{Encoding.UTF8.GetString(response.RequestBodyInBytes)}");
}
else
{
Console.WriteLine($"{response.HttpMethod} {response.Uri}");
}
Console.WriteLine();
if (response.ResponseBodyInBytes != null)
{
Console.WriteLine($"Status: {response.HttpStatusCode}\n" +
$"{Encoding.UTF8.GetString(response.ResponseBodyInBytes)}\n" +
$"{new string('-', 30)}\n");
}
else
{
Console.WriteLine($"Status: {response.HttpStatusCode}\n" +
$"{new string('-', 30)}\n");
}
});
var client = new ElasticClient(connectionSettings);
var includeAutoLearnedData = false;
var request = new SearchRequest<Message>
{
Query = new HasChildQuery
{
Type = "child",
Query = new CommonTermsQuery
{
Field = Infer.Field<Message>(m => m.Content),
Query = "commonterms"
}
&& (includeAutoLearnedData ? null : +new TermQuery
{
Field = Infer.Field<Message>(m => m.Content),
Value = "term"
})
}
};
client.Search<Message>(request);
}
public class Message
{
public string Content { get; set; }
}
produces the following query when includeAutoLearnedData is false
{
"query": {
"has_child": {
"type": "child",
"query": {
"bool": {
"must": [
{
"common": {
"content": {
"query": "commonterms"
}
}
}
],
"filter": [
{
"term": {
"content": {
"value": "term"
}
}
}
]
}
}
}
}
}
and when it's true
{
"query": {
"has_child": {
"type": "child",
"query": {
"common": {
"content": {
"query": "commonterms"
}
}
}
}
}
}
(I noticed that we are missing a section on conditionless queries in the latest documentation. Will add one!)
Related
I have data in provinces collection like this:
{
"_id": {
"$oid": "63dc7ff82e7e5e91c0f1cd87"
},
"province": "province1",
"districts": [
{
"district": "district1",
"sub_districts": [
{
"sub_district": "sub_district1",
"zip_codes": [
"zip_code1"
]
},
{
"sub_district": "sub_district2",
"zip_codes": [
"zip_code2"
]
},
],
},
],
}
This is how I get a list of sub_district for now:
- I search for province using Builders.Filter.
- Use foreach to get districts array (In province collection) and use if-statement to check if district equal searchDistrict.
- Get sub_districts array in that distric.
Source code:
public static List<string> MongoDbSelectSubDistrict(string searchProvince, string searchDistrict)
{
List<string> subDistrictList = new List<string>();
try
{
var provincesCollection = _db.GetCollection<BsonDocument>("provinces");
var builder = Builders<BsonDocument>.Filter;
var filter = builder.Empty;
if (searchProvince != "")
{
var provinceFilter = Builders<BsonDocument>.Filter.Eq("province", searchProvince);
filter &= provinceFilter;
}
/*
//***Need to be revised***
if (searchDistrict != "")
{
var districtFilter = Builders<BsonDocument>.Filter.Eq("provinces.district", searchDistrict);
filter &= districtFilter;
}
*/
var queryProvinces = provincesCollection.Find(filter).ToList();
foreach (BsonDocument queryProvince in queryProvinces)
{
BsonArray districtArray = queryProvince.GetValue("districts").AsBsonArray;
foreach (BsonDocument districtDocument in districtArray)
{
string district = districtDocument.GetValue("district").ToString();
if (district == searchDistrict) //***Need to be revised***
{
BsonArray subDistrictArray = districtDocument.GetValue("sub_districts").AsBsonArray;
foreach (BsonDocument subDistrictDocument in subDistrictArray)
{
string subDistrict = subDistrictDocument.GetValue("sub_district").ToString();
subDistrictList.Add(subDistrict);
}
}
}
}
}
catch (TimeoutException ex)
{
}
return subDistrictList;
}
Is there any efficient way to get this?
This is what I want:
[
{
"sub_district": "sub_district1",
"zip_codes": [
"zip_code1"
]
},
{
"sub_district": "sub_district2",
"zip_codes": [
"zip_code2"
]
},
]
And one more question: if I want to search for sub_district in the collection, how do I get this without looping in sub_districts array?
I am trying to write a C# method that get the queryString of a controller and converts it into an ElasticSearch query, like below:
public QueryContainerDescriptor<T> Convert<T> (IQueryCollection query) where T: class
{
var selector = new QueryContainerDescriptor<T>();
List<QueryContainer> Must = new List<QueryContainer>();
foreach(var key in query.keys)
{
string value = query[key];
var match = new MatchQuery { Field = $"{key}.keyword", Query = value };
list.Add(match)
}
selector.Bool(q => q.Must(Must.ToArray()));
return selector;
}
It works as expected, but if I pass a queryString value with a backslash, like:
http://localhost:5000/api/indexData?user=ESKA\\USER
It should be converted into this query:
{ "bool": { "must": [ { "match" : { "user.keyword": "ESKA\\USER" } } ] }
But ElasticSearch will return nothing because the value will of the query be: ESKA\\\\USER with 4 backslashes, like:
{ "bool": { "must": [ { "match" : { "user.keyword": "ESKA\\\\USER" } } ] }
how can I solve this issue?
I don't think that Nest is performing any escaping of backslashes. Here's an example that writes out the requests (and responses, if using an IConnection that sends the request)
private static void Main()
{
var pool = new SingleNodeConnectionPool(new Uri($"http://localhost:9200"));
var settings = new ConnectionSettings(pool, new InMemoryConnection())
.DefaultIndex("default_index")
.DisableDirectStreaming()
.PrettyJson()
.OnRequestCompleted(callDetails =>
{
if (callDetails.RequestBodyInBytes != null)
{
var json = JObject.Parse(Encoding.UTF8.GetString(callDetails.RequestBodyInBytes));
Console.WriteLine(
$"{callDetails.HttpMethod} {callDetails.Uri} \n" +
$"{json.ToString(Newtonsoft.Json.Formatting.Indented)}");
}
else
{
Console.WriteLine($"{callDetails.HttpMethod} {callDetails.Uri}");
}
Console.WriteLine();
if (callDetails.ResponseBodyInBytes != null)
{
Console.WriteLine($"Status: {callDetails.HttpStatusCode}\n" +
$"{Encoding.UTF8.GetString(callDetails.ResponseBodyInBytes)}\n" +
$"{new string('-', 30)}\n");
}
else
{
Console.WriteLine($"Status: {callDetails.HttpStatusCode}\n" +
$"{new string('-', 30)}\n");
}
});
var client = new ElasticClient(settings);
var collection = new QueryCollection(new Dictionary<string, StringValues>
{
{ "user", "ESKA\\USER" }
});
var response = client.Search<object>(s => s
.Query(q => Convert<object>(q, collection))
);
}
public static QueryContainerDescriptor<T> Convert<T>(QueryContainerDescriptor<T> selector, IQueryCollection query) where T : class
{
List<QueryContainer> Must = new List<QueryContainer>();
foreach (var key in query.Keys)
{
string value = query[key];
var match = new MatchQuery { Field = $"{key}.keyword", Query = value };
Must.Add(match);
}
selector.Bool(q => q.Must(Must.ToArray()));
return selector;
}
the resulting query is
POST http://localhost:9200/default_index/_search?pretty=true&typed_keys=true
{
"query": {
"bool": {
"must": [
{
"match": {
"user.keyword": {
"query": "ESKA\\USER"
}
}
}
]
}
}
}
If the user value were to be the verbatim string literal #"ESKA\\USER", then the resulting query would be
"user.keyword": {
"query": "ESKA\\\\USER"
}
because each \ in the verbatim string literal needs to be escaped.
in NEST 2.x, I wrote code to query data like below:
var query = new QueryContainer();
query = query && new TermQuery { Field = "catId", Value = catId };
query = query && new NumericRangeQuery { Field ="price", GreaterThan = 10 };
var request =new SearchRequest<Project>
{
From = 0,
Size = 100,
Query = query,
Sort = new List<ISort>
{
new SortField { Field = "field", Order = SortOrder.Descending },
...
},
Type?? //problem comes here, how to specify type??
}
var response = _client.Search<Project>(request);
There are more than one type in my index, I want to query data in one of type.(just like query one of table data in a database), I hope in the SearchRequest object initializer have a "Type" parameter.
You can specify the indices and types in the constructor for SearchRequest<T>()
var catId = 1;
var query = new QueryContainer(new TermQuery { Field = "catId", Value = catId });
query = query && new NumericRangeQuery { Field = "price", GreaterThan = 10 };
var request = new SearchRequest<Project>("index-name", Types.Type(typeof(Project), typeof(AnotherProject)))
{
From = 0,
Size = 100,
Query = query,
Sort = new List<ISort>
{
new SortField { Field = "field", Order = Nest.SortOrder.Descending },
}
};
var response = client.Search<Project>(request);
would generate the following query
POST http://localhost:9200/index-name/project%2Canotherproject/_search?pretty=true
{
"from": 0,
"size": 100,
"sort": [
{
"field": {
"order": "desc"
}
}
],
"query": {
"bool": {
"must": [
{
"term": {
"catId": {
"value": 1
}
}
},
{
"range": {
"price": {
"gt": 10.0
}
}
}
]
}
}
}
I'm trying to combine multiple queries into a single boolean query but the query generated is empty for the queries that are raw.
This is my method call
ConvertQueryToESD("{\"term\":\"fieldname\":\"value\"}"
,null
,a range filter)
.From(0)
.Take(50)
.SortMulti(a sort)
This is the method
internal SearchDescriptor<T> ConvertQueryToESD<T>(string queryString, string filterId, List<QueryContainer> filters) where T : class
{
var filterQueryString = GetFilter(filterId);
if (!string.IsNullOrWhiteSpace(filterQueryString))
{
var qd = new QueryDescriptor<T>();
filters.Add(qd.Raw(filterQueryString));
}
if (!string.IsNullOrWhiteSpace(queryString))
{
var qd = new QueryDescriptor<T>();
filters.Add(qd.Raw(queryString));
}
var sd = new SearchDescriptor<T>();
sd.Query(q => q.Bool(b => b.Must(filters.ToArray())));
return sd;
}
My resulting query is
{
"from": 0,
"size": 50,
"sort": [
{
"startTimeLocal": {
"order": "desc"
}
}
],
"query": {
"bool": {
"must": [
{
"range": {
"startTimeLocal": {
"gte": "0001-01-01T00:00:00",
"lte": "2015-12-22T16:28:22"
}
}
},
{}
]
}
}
}
The {} is where I would expect to see the raw term query. Why is the raw query not being serialized in to the SearchDescriptor? I have inspected the SearchDescriptor and the field is populated before the search is ran. I have also tested only adding the raw string term query without the range filter and the search descriptor serializes correctly.
Addendum
private string GetFilter(string filterId)
{
if (string.IsNullOrWhiteSpace(filterId))
{
return null;
}
var response = _elasticClient.Get<FilterMapping>(filterId);
ThrowIfNotSuccessfull(response.ConnectionStatus);
return response.Source?.QueryString;
}
I am trying to add a new record to the var "issue". I get a list of XXX from a SQL server DB and return in as follows for a jTable grid:
public dynamic XXXList(int CCC)
{
try
{
var issue = db.XXX.ToList().
Select(c => new { DisplayText = c.AAA, Value = c.BBB, c.CCC}).
Where(h => h.HHH == JJJ);
return (new { Result = "OK", Options = issue });
}
catch (Exception ex)
{
return (new { Result = "ERROR", Message = ex.Message });
}
}
The function returns:
{
"$id": "1",
"Result": "OK",
"Options": [
{
"$id": "2",
"DisplayText": "Food and Beverages",
"Value": 4,
"CCC": 4
},
{
"$id": "3",
"DisplayText": "Wrong software versions",
"Value": 5,
"CCC": 4
}
]
}
How can I add another record to the issue var before returning it?
Example:
{
"DisplayText": "new display text",
"Value": 5,
"CCC": 4
}
EDIT:
Here is my function after applying the answers:
public dynamic XXXList(int CCC)
{
try
{
var newRecord = new[] { new { DisplayText = "None", Value = -1, CCC = -1} };
var issue = db.ProjectXXXs.Where(h => h.CCC == JJJ).Select(c => new { DisplayText = c.AAA, Value = c.BBB, c.CCC }).ToList().Concat(newRecord);
return (new { Result = "OK", Options = issue });
}
catch (Exception ex)
{
return (new { Result = "ERROR", Message = ex.Message });
}
}
Thank you very much for all the help.
Concatenate any additional items:
var additional = new[] {new { DisplayText = ..., Value = ..., CCC = ... },
new { DisplayText = ..., Value = ..., CCC = ... }};
var issue = db.XXX.
Where(h => h.HHH == JJJ).
Select(c => new { DisplayText = c.AAA, Value = c.BBB, c.CCC}).
ToList().
Concat(additional);
(EDIT: Credit for the reordering of Where, Select and ToList goes to James Curran.)
Well first thing you want to do is move the ToList() to the end. With the ToList where it was, you'd read every column from every record from the Database, build a list from that, and then search it. With the ToList at the end, you send a query for just those columns of that record to the database, and then build a list from what comes back. You'll also need to move the Where before the Select, so it applies to the XXX records and not the output from the select.
var issue = db.XXX
.Where(h => h.HHH == JJJ)
.Select(c => new { DisplayText = c.AAA, Value = c.BBB, c.CCC})
.ToList();
From here the Add or Concat options suggested by other should work, however, you may have to use a named class instead of an anonomous one.
class OptionValues
{
public string DisplayText {get; set;}
public int Value {get; set;}
public int CCC {get; set;}
}
// :
// :
.Select(c => new OptionValues {DisplayText = c.AAA, Value = c.BBB, CCC= c.CCC})
I'm not sure, but can't you just write a line under var issue:
issue.Add(new { });
with the values that you want of course.