I've got data indexed using ElasticSearch, and I'm having trouble querying a particular field. A snippet of the JSON is as follows:
{
"_index": "indexName",
"_type": "type",
"_id": "00001",
"color": "red",
"place": "london",
"person": [
{
"name": "john",
"friends": [
"mary",
"jane"
]
}
{
"name": "jack",
"friends": [
"lisa",
"alex"
]
}
]
}
I need to query the index and pick out all records where one of the names inside person is "john".
I'm using Client.Search to do this, and I've had no trouble querying the fields that aren't nested (like color) by using:
var searchResults = client.Search<People>(s => s
.Index("indexName")
.Type("type")
.Query(q => q
.Bool(b => b
.Must(
x => x.Match(m => m.OnField(p => p.color).Query("red")),
x => x.Match(m => m.OnField(p => p.place).Query("london"))))));
I've got People defined as follows:
public class People
{
public string color {get; set; }
public string place {get; set; }
public List<Person> person {get; set; }
}
public class Person
{
public string name {get; set; }
// "friends" isn't here as I don't pull data from it
}
I'm unsure as to how to query on name as it's "inside" people - any help is greatly appreciated.
You need to wrap query in nested_query to have access to nested fields.
{
"nested" : {
"path" : "person",
"query" : {
"match" : {"person.name" : "john"}
}
}
}
Exceprt from documentation:
The query is executed against the nested objects / docs as if they
were indexed as separate docs (they are, internally) and resulting in
the root parent doc (or parent nested mapping).
Basically internally nested fields are stored as separate documents nearby (so they are quick to fetch) the original document. By default elastic doesn't load them, so you need to explicitly tell him that you want to access it. You could say nested fields are lazy ;)
Sorry It's been a long time since I worked on .Net and Linq. Don't know the API. But you need to create something like that.
Edit.
From github source and your code I think you need to:
var s = new SearchDescriptor<People>()
.Query(ff=>ff
.Nested(n=>n
.Path(f=>f.person[0])
.Query(q=>q.Term(f=>f.person[0].name,"john"))
)
);
Edit2.
Did you try direct curl to server? Or try query in head plugin? Something like:
curl -XPOST 'http://localhost:9202/indexName' -d '
{
"query": {
"nested": {
"path": "person",
"query": {
"query_string": {
"query": "person.name: john"
}
}
}
}
}'
This works on my cluster (with changed column names).
After a long while, I finally figured out that my data wasn't actually indexed as nested in the first place, and so simply adding
.Term("person.name", "john")
to my query worked perfectly.
Related
I have a Json file with following data:
[
{ "Id": 510, "Title": "Big Start", "Author": [{ "AuthId": 7777, "Name": "Tom" }] },
{ "Id": 511, "Title": "Rising Tide", "Author": [{ "AuthId": 6666, "Name": "Bob" }] }
]
For the above, I have defined the following classes:
public class Book {
public int Id { get; set; }
public string Title { get; set; }
public List<Author> Authors { get; set; }
}
public class Author {
public int AuthId { get; set; }
public string Name { get; set; }
}
I have deserialized the json data using the following code:
List<Book> books;
using (StreamReader sr = new StreamReader(Server.MapPath("...")))
{
books = JsonConvert.DeserializeObject<List<Item>>(sr.ReadToEnd()).ToList()
}
Now I want to sort the list of books using author Name. I have tried the following:
var list = books.OrderBy(b => b.Author.OrderBy(a => a.Name)).ToList();
But I get the error message 'At least one object must implement IComparable'.
Can someone help me sort the above list using the nested author Name.
Thanks in advance.
Thing is, Authors is a List, so which author name do we sort by? There might be many. "Book1, by Alfred and Zeus", "Book2 by Zigo, Mary and Ada" - how do these books sort? How do they sort if Book2 was by Ada, Mary and Zigo?
If there's only ever one author it's fairly simple
var list = books.OrderBy(b => b.Authors.First().Name).ToList();
As authors is a list you could also index it
var list = books.OrderBy(b => b.Authors[0].Name).ToList();
If there is ever more than one author you need to have a think about which author is used for sort.. If multiple authors contribute as tie breakers it gets more complex.. You could end up with a ThenBy that skips the first author etc, but then you need to consider if you're going to sort the authors too..
Remember that c# is case sens, so "bob" will sort after "Tom" and also the code above won't work if here are zero authors - you've implied there will always be one but if not, consider FirstOrDefault()?.Name??"" ..
..sometimes the simplest of questions leads to the most involved set of work! ;)
I know that Azure Cosmos DB SQL API allows using SQL syntax to query the documents from it and working for me.
For eg: SELECT * FROM c WHERE c.DynamicContent.MailingAddress.City LIKE '%PORT%' is working well.
The data I'm trying to save is like this:
{
"Id": 1001,
"AttentionName": "Keyword list",
"DynamicContent": {
"Agency": 2,
"EnteredOn": "2001-03-30T16:20:00+0000",
"UpdatedOn": "2001-03-30T16:20:00+0000",
"Name": "Sample Name",
"MailingAddress": {
"Address1": "501 Abbey St",
"City": "Portland",
"StateProvince": "OR",
"PostalCode": "97215",
"Country": "United States"
}
}
}
and as the content inside the DynamicContent is unstructured and dynamic, I'm using the below class to save the data:
public partial class CosmosDocument
{
[JsonProperty("Id")]
public long Id { get; set; }
[JsonProperty("AttentionName")]
public string AttentionName { get; set; }
[JsonProperty("DynamicContent")]
public dynamic DynamicContent { get; set; }
}
The problem I'm facing is not able to do something like this:
query.Where(c => c.DynamicContent.MailingAddress.City.Contains("PORT")).ToList();
because the properties(dynamic and UI controlled) that may come in the filter are inside a dynamic class and are not strongly typed.
Is there a way to do this properly or at least some other workaround to achieve this?
Please help with your thoughts.
Rather than using LINQ for queries, consider using QueryDefinition instead that allows constructing from an arbitrary string and parameter values. There are examples on the GetItemQueryIterator docs, for instance:
QueryDefinition queryDefinition = new QueryDefinition("select * from ToDos t where t.cost > #expensive")
.WithParameter("#expensive", 9000);
using (FeedIterator<ToDoActivity> feedIterator = this.Container.GetItemQueryIterator<ToDoActivity>(
queryDefinition,
null,
new QueryRequestOptions() { PartitionKey = new PartitionKey("Error")}))
{
while (feedIterator.HasMoreResults)
{
foreach(var item in await feedIterator.ReadNextAsync())
{
Console.WriteLine(item.cost);
}
}
}
I am using Nest to insert/update Documents in Elasticsearch. Here's a sample POCO class that I use to map the Document...
public class MyClass
{
public string Id { get; set; }
public decimal? MyField { get; set; }
}
This works as expected...When I add the Document, if the nullable field MyField has a value, it is returned in the JSON. If the nullable field doesn't have a value, it's not returned in the _source JSON because null values aren't stored.
However, there might be times where I need to update a single document and remove the value from a single field. That is, when I first insert the document, MyField has a value and is returned in the Elasticsearch JSON result. Then later, for whatever reason, I need to remove that value.
I am using partial document updates, and if possible, would prefer to keep it like that. (The full Document model I'm using will have 100+ fields, and my index will eventually have 100M+ records.) So, I'm looking for the most efficient way possible to partially update the documents.
Thanks in advance!
Think how would you do it just using api? there are several ways. for example like this:
// kibana version
POST my_index/_update_by_query
{
"query": {
"ids": {
"values": [
"12345"
]
}
},
"script": {
"source": "ctx._source.myField = null"
}
}
// nest version
var response = client.UpdateByQuery<CustomerDto>(u => u
.Query(q => q
.Ids(i => i.Values(12345))
)
.Script("ctx._source.flag = null")
//.Conflicts(Conflicts.Proceed)
.Refresh(true)
);
or this:
// kibana version
POST my_index/my_type/12345/_update
{
"script": {
"source": "ctx._source.remove('myField')"
}
}
// nest version
var response = client.Update<CustomerDto>(
new UpdateDescriptor<CustomerDto, CustomerDto>("index_name", "type", 12345)
.Script(s => s.Source("ctx._source.remove('middleName')")));
I have the following structure – for simplicity I abbrev. few properties.
The root document (ProjectSchema) as you can see has property rootFolder (FolderSchema) which can have n nested level. For deserialization I am using classes written in C# – ProjectSchema and FolderSchema, mentioned above.
Goal
Deserialize BsonDocument into my custom class ProjectSchema where I need to find specific folder by Id and in that folder insert new FolderSchema
Problem
Could you tell me please how to iterate, respectively find a specific document nested possibly n levels and after all, how can I insert an element into the found one.
Structure
{
"_id" : ObjectId("5abce10cb02728006f1460fd"),
"owner_ref" : ObjectId("5ababb7188f6ba0079199dd0"),
"description" : "",
"rootFolder" : [
{
"_id" : ObjectId("5abce10cb02728006f1460fc"),
"folders" : [
{
"_id" : ObjectId("5abce9b5b02728006f1460ff"),
"folders" : [
{
"_id" : ObjectId("5abd5775b02728006f146130"),
"folders" : [
{
"_id" : ObjectId("5abd5781b02728006f146131"),
"folders" : [
{
"_id" : ObjectId("5abd578ab02728006f146132"),
"folders" : [],
"fileRefs" : [],
"docs" : [],
"name" : "NSubFolder1"
}
],
"fileRefs" : [],
"docs" : [],
"name" : "SubSubFolder1"
}
],
"fileRefs" : [],
"docs" : [],
"name" : "SubFolder1"
}
],
"fileRefs" : [],
"docs" : [],
"name" : "Folder1"
},
{
"_id" : ObjectId("5abd576db02728006f14612f"),
"folders" : [],
"fileRefs" : [],
"docs" : [],
"name" : "Folder2"
}
],
"fileRefs" : [],
"docs" : [
{
"_id" : ObjectId("5abce10cb02728006f1460fe"),
"name" : "main.tex"
}
],
"name" : "rootFolder"
}
]
}
Project schema - this holds folders and other attributes. That's my main document.
public class ProjectSchema
{
[BsonId]
public BsonObjectId ProjectId { get; set; }
[BsonElement("description")]
public string Description { get; set; } = "some desc";
[BsonElement("owner_ref")]
public BsonObjectId OwnerRef { get; set; }
[BsonElement("rootFolder")]
public List<FolderSchema> RootFolder { get; set; } = new List<FolderSchema>();
public ProjectSchema()
{
ProjectId = new BsonObjectId(ObjectId.GenerateNewId());
}
}
Folder schema this holds folders
public class FolderSchema
{
[BsonId]
public BsonObjectId FolderId { get; set; }
[BsonElement("name")]
public string Name { get; set; } = "new folder";
[BsonElement("docs")]
public IEnumerable<DocSchema> Docs { get; set; } = new List<DocSchema>();
[BsonElement("fileRefs")]
public IEnumerable<FileSchema> FileRefs { get; set; } = new List<FileSchema>();
[BsonElement("folders")]
public IEnumerable<FolderSchema> Folders { get; set; } = new List<FolderSchema>();
public FolderSchema()
{
FolderId = new BsonObjectId(ObjectId.GenerateNewId());
}
}
My attempt
Unfortunately, this works without success, everything worked without error , but nothing has happened. I assume there is an error in that filter, because Any() targets only first level. I have no idea how to target FolderSchema nested n levels.
Initial call - in this example I am trying to add new folder within folder SubFolder1 named HEUREKA
var projects = database.GetCollection<ProjectSchema>("projects");
var folder = new FolderSchema() { Name = "HEUREKA" };
var filter = Builders<ProjectSchema>.Filter.Where(p => p.ProjectId == new BsonObjectId(new ObjectId("5abce10cb02728006f1460fd"))
&& p.RootFolder.Any(l => l.FolderId == new BsonObjectId(new ObjectId("5abd5775b02728006f146130"))));
var update = Builders<ProjectSchema>.Update.Push(p => p.RootFolder, folder);
await projects.FindOneAndUpdateAsync(filter, update);
#Edit:
It's a good point, to replace whole document with the updated one, the simplest one – and I can admit, it works. However, my documents can be pretty big, So I would rather to update a piece rather then whole document. However, If I pick the option of partial update, I am still not able to update certain part of my document, I do not know how to put it together. So I tried the following one:
For clarification, I know ProjectId ProjectSchema - respectively for which project I want to update a Folder, as well as I know the parent FolderId FolderSchema to which I want to add a new folder.
//**Schema simplification:**
Project Schema
Folder Schema
Folder Schema
Folder Schema
[n level folder schema]
Folder Schema
[n level]
Written filter, to receive Parent folder named "NSubFolder1", or I would use Folder id instead of folder name, because I know it.
var eq = Builders<FolderSchema>.Filter.Eq(f => f.Name, "NSubFolder1");
var emN = Builders<FolderSchema>.Filter.ElemMatch(_ => _.Folders, eq);
Written filter to receive project where I want to add folder.
var eqProj = Builders<ProjectSchema>.Filter.Eq(p => p.ProjectId, "project id here");
var emP = Builders<ProjectSchema>.Filter.ElemMatch(_ => _.ProjectId, eqProj);
How these two filters combine together in order to receive specified FolderSchema under ProjectSchema and then push new FolderSchema to parent array?
Conceptually what you're trying to do is construct an arbitrarily deep recursive $elemMatch query. I don't think this can be done with a single find/update command with your current database design.
It seems as though you you know the ProjectId of your top level ProjectSchema beforehand - if this is the case, it would be possible to retrieve that document, traverse it, and construct an n-level deep FilterDefinition which would be described like so (though written differently)
var eq = Builders<FolderSchema>.Filter.Eq(f => f.Name, "HEUREKA");
var emN = Builders<FolderSchema>.Filter.ElemMatch(_ => _.Folders, eq);
...
var em2 = Builders<FolderSchema>.Filter.ElemMatch(_ => _.RootFolder, em3);
var em1 = Builders<FolderSchema>.Filter.ElemMatch(_ => _.RootFolder, em2);
var emRoot = Builders<ProjectSchema>.Filter.ElemMatch(_ => _.RootFolder, em1);
Though at this point, it would seem easiest to simply add your new FolderSchema in the deserialized object and replace the document in the database. Still, you could construct an Update query in much the same way you'd generate the ElemMatch query if you didn't want to replace the entire document.
If you were able to change your schema such that you insert multiple FolderSchema objects into the collection instead of one grand arbitrarily deep ProjectSchema document, it seems you could use the MongoDb Aggregate framework $graphLookup as described at https://docs.mongodb.com/manual/reference/operator/aggregation/graphLookup/ with more examples at https://docs.mongodb.com/manual/tutorial/model-tree-structures/
Without knowing more about your situation, it's hard to say if this would be a better option, or simply an alternative. The ability to use the MongoDB Aggregate framework should definitely be considered a plus.
I'm looking for a clean way to wrap my JSON responses to a list of objects within an JSON object using AutoMapper(Sorry if I'm unclear, I'm not sure what the correct name of the action I want to do is)
This is what my endpoint looks like:
[HttpGet]
public IHttpActionResult GetDevices()
{
var deviceDtos = _context.Devices.ToList()
.Select(Mapper.Map<Device, DeviceDto>);
return Ok(deviceDtos);
}
This is what my Mappingprofile for device looks like:
public MappingProfile()
{
Mapper.CreateMap<Device, DeviceDto>();
Mapper.CreateMap<DeviceDto, Device>();
}
This is what I currently get:
[
{
"$id": "1",
"id": "abc",
"name": "abc1",
"deviceTypeName": "abc",
"lastSeen": abc,
"contactLost": abc,
"contactLostTime": abc,
"isRegistered": abc,
"dataCollectorId": "abc",
"dataCollectorName": "abc",
"locationId": "abc",
"locationName": "abc"
},
]
But what I want is:
{
"Devices" :
[
{
"$id": "1",
"id": "abc",
"name": "abc1",
"deviceTypeName": "abc",
"lastSeen": abc,
"contactLost": abc,
"contactLostTime": abc,
"isRegistered": abc,
"dataCollectorId": "abc",
"dataCollectorName": "abc",
"locationId": "abc",
"locationName": "abc"
},
]
}
I have looked through similar posts, and they suggest using JSONConvert.Serialize(new{Device = device}), but when I do this the format either dissapears or I get \n \r, or if I use Formatting.None the object is no longer wrapped inside a Device JSON object.
I've also looked through: Automapper:Converting JSON to list of objects
But I'm not able to make this work with what I'm doing.
I've tried using JsonObject(Title = "Devices") but that doesn't show.
Can I add something to my CreateMap() that makes this possible?
If you feel like you need additional code/information I'd be happy to supply it.
Thank you in advance
There is nothing to do with AutoMapper configuration. You just need to wrap your list inside an object. You can use anonymous class for that
return Ok(new {Devices = deviceDtos});
or you can create wrapper class and return instance of that class
public class DevicesReponse
{
public List<DeviceDto> Devices { get; set; }
}
and use your wrapper class as response
var response = new DevicesResponse { Devices = devicesDtos };
return Ok(response);