I'm having problem to reproduce search result in OpenSearch / ElasticSearch from DSL query to .NET C# code base.
Index was populated from .NET Core application based on this model:
public class Car
{
public string Producer { get; set; }
public string Model { get; set; }
public string Vin { get; set; }
public ColorEnum Color { get; set; }
public TransmissionType Transmission { get; set; }
public CategoryType Category { get; set; }
}
Index mapping for model above looks like this:
"mappings": {
"_doc": {
"properties": {
"transmission": {
"type": "long"
},
"color": {
"type": "long"
},
"producer": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"model": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"vin": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"category": {
"type": "long"
}
}
}
Below is working as expected DSL query
{
"query": {
"bool": {
"must": [
{ "term": { "color": 0 }},
{ "term": { "transmission": 0 }},
{ "term": { "producer": "volvo" } }
]}}
}
The C# query that uses OpenSearch or ElasticSearch .NET Client and should produce that same outcome as DSL query is not respecting all required terms.
var documents = await _openSearchClient.SearchAsync<Car>(s => s
.Index(_indexName)
.Query(q => q
.Bool(b => b
.Must(m => m.Term("category", 0))
.Must(m => m.Term("transmission", 0))
.Must(m => m.Term("producer", producer)))));
return documents?.Documents;
The outcome of C# API call contain also documents that have category values set to 1 and transmission set to 1. It looks like only last term with producer name is respected.
Do i made mistake in c# query or there is something how to queries are executed ?
Thanks for all the help.
I found a solution based on input from this post
var documents = await _openSearchClient.SearchAsync<Car>(s => s
.Index(_indexName)
.Query(q => q
.Bool(bq => bq
.Must(mq => mq
.Term("transmission", 0)
&& mq.Term("color",0)
&& mq.Term("producer",producer)))));
Related
I have a model class that allows users to specify the template which is used to create content with specified properties.
public class Model
{
public string Id { get; set; }
public string Alias { get; set; }
public ICollection<Field> Fields { get; set; }
}
public class Field
{
public FieldType Type { get; set; }
public string Alias { get; set; }
}
public enum FieldType
{
String = 1,
Email = 2,
Password = 3,
DateTime = 4,
Integer = 5,
Decimal = 6,
...
}
The content that is created based on this template will have the structure as below:
public class Entry
{
public string _id { get; set; }
public List<FieldValue> FieldValues { get; set; }
}
public class FieldValue
{
public object Value { get; set; }
public string Alias { get; set; }
public FieldType Type { get; set; }
}
So, I for example, can have data such as:
PersonModel
{
"id": "34752b15",
"alias": "person",
"fields": [
{
"type": 1,
"alias": "name"
},
{
"type": 2,
"alias": "email"
}
]
}
PersonEntries:
[
{
"_id": "426e",
"fieldValues": [
{
"alias": "name",
"value": "John",
"type": 1
},
{
"alias": "email",
"value": "john#stark.com",
"type": 2
}
]
},
{
"_id": "c6110fc3cc0e",
"fieldValues": [
{
"alias": "name",
"value": "Arya",
"type": 1
},
{
"alias": "email",
"value": "arya#stark.com",
"type": 2
}
]
}
]
Or:
ShapeModel
{
"id": "afd69af4",
"alias": "shape",
"fields": [
{
"type": 1,
"alias": "name"
},
{
"type": 6,
"alias": "area"
},
{
"type": 6,
"alias": "perimeter"
}
]
}
ShapeEntries:
[
{
"_id": "426e",
"fieldValues": [
{
"alias": "name",
"value": "my-square",
"type": 1
},
{
"alias": "area",
"value": 1,
"type": 6
},
{
"alias": "perimeter",
"value": 4,
"type": 6
}
]
},
{
"_id": "426enbv",
"fieldValues": [
{
"alias": "name",
"value": "my-triangle",
"type": 1
},
{
"alias": "area",
"value": 0.5,
"type": 6
},
{
"alias": "perimeter",
"value": 3,
"type": 6
}
]
}
]
and so on.
The idea is to create graphql schemas dynamically (on runtime) when a new model is created and allow users to query entries based on the model alias.
What I tried is:
public ISchema CreateEntrySchema(List<Entry> entries, Model model, IServiceCollection services)
{
var entryType = new ObjectGraphType() { Name = model.Alias };
entryType.Field("_id", new IdGraphType());
foreach (var field in model.Fields)
{
if (entryType.Fields.FirstOrDefault(f => f.Name == field.Alias) != null)
{
continue;
}
Type clrGraphType = field.Type.GetStandardType().GetGraphTypeFromType(isNullable: true);
IGraphType graphType = (IGraphType)Activator.CreateInstance(clrGraphType);
entryType.Field(field.Alias,
graphType,
resolve: context =>
{
var entry = (Entry)context.Source;
object firstOrDefault = entry?.FieldValues.FirstOrDefault(value => value.Alias == context.FieldDefinition.Name)?.Value;
return firstOrDefault;
});
}
var queryType = new ObjectGraphType();
queryType.Field(
model.Alias,
type: new ListGraphType(entryType),
arguments: new QueryArguments(ConstructArguments(model.Fields)),
resolve: context =>
{
List<KeyValuePair<string, ArgumentValue>> args = context.Arguments?.Where(pair => pair.Value.Value != null).ToList();
if (!args?.Any() ?? true)
{
return entries;
}
return args.Aggregate(entries, (current, pair) =>
current.Where(entry => entry.FieldValues.FirstOrDefault(field => field.Alias == pair.Key) != null && entry.FieldValues.FirstOrDefault(field => field.Alias == pair.Key).Value.Equals(pair.Value.Value))
.ToList());
});
ISchema schema = new Schema { Query = queryType };
schema.RegisterType(entryType);
services.AddGraphQL(a =>
{
a.AddSchema(schema).AddClrTypeMappings();
});
return schema;
}
The problem is that it is registering only the last schema. So, for example, if I will call this method like:
CreateEntrySchema(personEntries, personModel, services);
CreateEntrySchema(shapeEntries, shapeModel, services);
And query like this:
{
shape{
name
}
person{
name
}
}
The result is:
"GraphQL.Validation.Errors.FieldsOnCorrectTypeError: Cannot query field 'person' on type 'Object'. Did you mean 'shape'?",
I want to count and sum all posts in Items. When i query my collection with:
GetCollection().Find(p => p.Type == "Test") i receive this:
[
{
"Type": "Test",
"Items": [
{ "Name": "123", "Id":"123" },
{ "Name": "123", "Id":"123" }
]
},
{
"Type": "Test",
"Items": [
{ "Name": "123", "Id":"123" },
{ "Name": "123", "Id":"123" }
]
}
]
But in this case i want to count all posts in items and the result i want to get is: 4. How can i write a query using MongoDB C# driver to get this?
This is how you can do in mongoDB via aggregation:
db.collection.aggregate([
{
$match: {
Type: "Test"
}
},
{
"$addFields": {
"Items": {
$size: "$Items"
}
}
},
{
$group: {
_id: "Sum",
Total: {
$sum: "$Items"
}
}
}
])
Explained:
Match all documents where Type:"Test"
AddFields/$size to count the Items array elements per document.
Group/Sum to count total Items elements.
With small modification you can adapt to C#
Playground
Assume that the result returns the array/list of Entity, you can work with System.Linq by flattening the list and getting the count of Items.
using System.Linq;
List<Entity> records = GetCollection()
.Find(p => p.Type == "Test")
.ToList();
int count = records
.SelectMany(x => x.Items)
.Count();
public class Entity
{
public string Type { get; set; }
public List<Item> Items { get; set; }
}
public class Item
{
public string Name { get; set; }
public string Id { get; set; }
}
Given 2 JSON strings:
[
{
"id":"BA",
"description":"BrandA",
"values":[
{
"id":"CategoryA",
"description":"CategoryA"
},
{
"id":"CategoryB",
"description":"CategoryB"
},
{
"id":"CategoryC",
"description":"CategoryC"
},
{
"id":"CategoryD",
"description":"CategoryD"
},
{
"id":"CategoryE",
"description":"CategoryE"
},
{
"id":"CategoryF",
"description":"CategoryF"
},
{
"id":"CategoryG",
"description":"CategoryG"
},
{
"id":"CategoryH",
"description":"CategoryH"
}
]
},
{
"id":"BB",
"description":"BrandB",
"values":[
{
"id":"CategoryA",
"description":"CategoryA"
},
{
"id":"CategoryB",
"description":"CategoryB"
},
{
"id":"CategoryC",
"description":"CategoryC"
}
]
}
]
AND
[
{
"id":"BA",
"description":"BrandA",
"values":[
{
"id":"CategoryA",
"description":"CategoryA"
},
{
"id":"CategoryC",
"description":"CategoryC"
}
]
},
{
"id":"BB",
"description":"BrandB",
"values":[
{
"id":"CategoryB",
"description":"CategoryB"
}
]
}
]
First one is the original. The second are the values that I want to remove from the original. So basically, if there is a match on brand and category between first and second JSON, regardless of the order of the elements, I want that match to be removed.
The expected result would be someting like this:
[
{
"id":"BA",
"description":"BrandA",
"values":[
{
"id":"CategoryB",
"description":"CategoryB"
},
{
"id":"CategoryD",
"description":"CategoryD"
},
{
"id":"CategoryE",
"description":"CategoryE"
},
{
"id":"CategoryF",
"description":"CategoryF"
},
{
"id":"CategoryG",
"description":"CategoryG"
},
{
"id":"CategoryH",
"description":"CategoryH"
}
]
},
{
"id":"BB",
"description":"BrandB",
"values":[
{
"id":"CategoryA",
"description":"CategoryA"
},
{
"id":"CategoryC",
"description":"CategoryC"
}
]
}
]
Catagory A and C in Brand A were removed as well as Category B in Brand B.
Based in some research, I was using https://github.com/wbish/jsondiffpatch.net, tried to work with it's functions, but so far I didn't manage to achieve the result I want. Also, to solve this by processing JSON direcly is not a must. If there is a simpler solution to achieve that by converting them to lists and use something like LINQ for example, it works for me as well (tried that, but didn't manage to find a way to do this comparison).
Thanks in advance.
If performance does not matter, the JsonSerializer and LINQ can be used:
Model
public class JsonModel
{
public class ValueModel
{
public string Id { get; set; }
public string Description { get; set; }
}
public string Id { get; set; }
public string Description { get; set; }
public IEnumerable<ValueModel> Values { get; set; }
}
Deserialize and LINQ
string json1Str = #"[...]";
string json2Str = #"[...]";
var opt = new System.Text.Json.JsonSerializerOptions { PropertyNameCaseInsensitive = true };
var json1 = System.Text.Json.JsonSerializer.Deserialize<List<JsonModel>>(json1Str, opt);
var json2 = System.Text.Json.JsonSerializer.Deserialize<List<JsonModel>>(json2Str, opt);
var result = json1.
Join(json2, j1 => j1.Id, j2 => j2.Id, (j1, j2) => new JsonModel
{
Id = j1.Id,
Description = j1.Description,
Values = j1.Values.Where(j1 => !j2.Values.Select(val => val.Id).Contains(j1.Id))
});
Console.WriteLine(System.Text.Json.JsonSerializer.Serialize(result, new System.Text.Json.JsonSerializerOptions { WriteIndented = true }));
}
Result
[
{
"Id": "BA",
"Description": "BrandA",
"Values": [
{
"Id": "CategoryB",
"Description": "CategoryB"
},
{
"Id": "CategoryD",
"Description": "CategoryD"
},
{
"Id": "CategoryE",
"Description": "CategoryE"
},
{
"Id": "CategoryF",
"Description": "CategoryF"
},
{
"Id": "CategoryG",
"Description": "CategoryG"
},
{
"Id": "CategoryH",
"Description": "CategoryH"
}
]
},
{
"Id": "BB",
"Description": "BrandB",
"Values": [
{
"Id": "CategoryA",
"Description": "CategoryA"
},
{
"Id": "CategoryC",
"Description": "CategoryC"
}
]
}
]
How to combine JSON objects in the same response that has the same key and value. like if I've two objects that have the same language: Python I want to combine them and List the remaining data under this language Python I don't want it being repeated
[
[
{
"language": "Python",
"id": 319029846,
"full_Name": "beurtschipper/Depix",
"name": "Depix"
},
{
"language": "Python",
"id": 319169382,
"full_Name": "benwilber/boltstream",
"name": "boltstream"
},
{
"language": "Python",
"id": 316899719,
"full_Name": "r0ysue/r0capture",
"name": "r0capture"
}
],
[
{
"language": "YARA",
"id": 318029147,
"full_Name": "fireeye/red_team_tool_countermeasures",
"name": "red_team_tool_countermeasures"
}
],
[
{
"language": "TypeScript",
"id": 313443335,
"full_Name": "pmndrs/valtio",
"name": "valtio"
}
]
]
what the form I want is
[
[
{
"language": "Python",
"id": [319029846, 319169382, 316899719],
"full_Name": ["beurtschipper/Depix", "benwilber/boltstream", "r0ysue/r0capture"],
"name": ["Depix", "boltstream", "r0capture"]
}
],
[
{
"language": "YARA",
"id": 318029147,
"full_Name": "fireeye/red_team_tool_countermeasures",
"name": "red_team_tool_countermeasures"
}
],
[
{
"language": "TypeScript",
"id": 313443335,
"full_Name": "pmndrs/valtio",
"name": "valtio"
}
]
]
And this is the code i'm using
public class Items
{
[JsonPropertyName("language")]
public string Language { get; set; }
[JsonPropertyName("id")]
public int Id { get; set; }
[JsonPropertyName("name")]
public string Name { get; set; }
[JsonPropertyName("full_name")]
public string Full_Name { get; set; }
public string total_count { get; set; }
}
public class Root
{
[JsonPropertyName("items")]
public List<Items> Items { get; set; }
}
Root jObj2 = JsonConvert.DeserializeObject<Root>(readerResult);
var result = jObj2.Items.Select(x => new
{
x.Language,
x.Id,
x.Full_Name,
x.Name
}).GroupBy(x => x.Language).ToArray();
return new JsonResult(result);
GroupBy is a good place to start. Once you have the groups, you need to select the individual properties of each group into a new list:
var result = jObj2.Items
.GroupBy(x => x.Language)
.Select(group => new
{
Language = group.Key,
Ids = group.Select(x => x.Id).ToList(),
FullNames = group.Select(x => x.Full_Name).ToList(),
Names = group.Select(x => x.Name).ToList()
})
.ToArray();
I'm new on elasticsearch world and I have a requirement to search for documents with a specific GUIDs. I've already figure out that to do that I need to map my GUID field as a keyword field.
There is a code already in production that is performing the indexing of the documents, but no mapping is being made. See below the actual code:
public class ProjectFileES
{
public Guid ClientId { get; set; }
public Guid ProjectId { get; set; }
public string DisplayName { get; set; }
public string FileName { get; set; }
public string Content { get; set; }
}
[...]
var _bulkDescriptor = new BulkDescriptor();
_bulkDescriptor.CreateMany<ProjectFileES>(list, (bd, q) =>
bd.Id(q.Id.ToString())
.Index($"{AppConstants.ES_IndexName}-{DateTime.UtcNow.ToString("yyyy-MM-dd")}")
.Type(AppConstants.ES_Type)
);
return await Client.BulkAsync(_bulkDescriptor);
As I read in the Elastic documentation, GUIDs are auto mapped to keyword type, as you can see below.
{
"mappings": {
"projectfilees": {
"properties": {
"clientId": {
"type": "keyword"
},
"projectId": {
"type": "keyword"
},
"displayName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"fileName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"content": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"id": {
"type": "keyword"
}
}
}
}
}
# Response:
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "roger-test3"
}
But now I'm trying to Search passing a projectId of a document which is indexed, as following:
var response = await _elastic.SearchAsync<ProjectFileES>(s => s.RequestConfiguration(c => c.DisableDirectStreaming())
.Index("*")
.Query(q =>
q.Bool(b => b
.Filter(f => f
.Term(t => t.ProjectId, projectId)
)
)
)
);
but my response object have no documents returned:
Response Object
Here is the JSON response inside the DebugInformation property:
{
"query": {
"bool": {
"filter": [
{
"term": {
"ProjectId": {
"value": "45efa5d0-8b41-11ea-871b-e3b070662e35"
}
}
}
]
}
}
}
# Response:
{
"took" : 48,
"timed_out" : false,
"_shards" : {
"total" : 373,
"successful" : 373,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}
Any help?