Develop Reference

How to update filed type from string to array of strings in elasticsearch?

I have an index filled by data with this schema:
{
"name" : "string",
"book": "string"
}
And I am working with ElasticSearch with Nest and C#.
Now because of the business requirement book field must be an array and I Update my query model C# POCO class but when I run my query I am getting this error:
expected:'[', actual:'"test"', at offset:2248
Now I want to Update my schema to convert the book to array type. I try to use script and mapping but could not find a good solution? is there any working method to perform this change in ElasticSearch?

ES has built in functionally for docs update by query https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html.
In case above query will look like this:
POST my-index-000001/_update_by_query
{
"script": {
"source": " ctx._source.book = [ctx._source.book] ",
"lang": "painless"
},
"query": {
"match_all":{}
}
}

you cannot update or change field type in mapping(schema) of indices that has been created.
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html#updating-field-mappings
if you need change type, you should create new indices and use Reindex API.

How to filter a JSON by a string value contained in an array inside JSON using JSONPath

I am listing down my sample JSON below,
I want to write a JSONPath that selects element when subjects has (or contains) "Maths".
NOTE: I am using Goessner's JSONPath using Newtonsoft library (C#)**
{
"class":{
"type":"group",
"identifier":"MyFavClass",
"subs":[
{
"type":"individual",
"identifier":"Rambo"
},
{
"type":"individual",
"identifier":"Rancho"
},
{
"type":"biggroup",
"identifier":"village",
"subjects":[
"Maths",
"English"
]
},
{
"type":"biggroup",
"identifier":"newvillage",
"subjects":[
"Maths"
]
}
]
}
}

Assuming you are using Json.NET's SelectTokens() method for your JSONPath queries, you may find all math subjects using the following query:
var value = "Maths";
var subjects = root.SelectTokens($"class.subs[*].subjects[?(# == '{value}')]");
Of course this will just return a sequence of JValue tokens with value "Maths". If you want the class.subs[*] objects whose subjects[*] arrays contains "Maths", you may do:
var subs = root.SelectTokens($"class.subs[?(#.subjects[?(# == '{value}')])]");
Notes:
[?(...)] applies a filter expression ... to an array.
In a filter expression, array contents that are primitive values may be matched using the # symbol for the current object.
Json.NET's implementation of JSONPath supports nested queries inside array filter expressions. Thus [?(#.subjects[?(# == '{value}')])] matches array items that have an item in their "subjects" array that matches the value parameter.
Note that search values that contain a single quote character ' must be escaped as shown in Querying JSON with JSON Path and escaped properties. Thus if the string to be matched was Math's you would need to escape it like so:
var value = "Math's".Replace("'", "\\\'");
Demo fiddle here.

MongoDB C# Case Insensitive Sort and Index

So far I've been using this code to find my documents and then sort them:
var options = new FindOptions
{
Modifiers = new BsonDocument("$hint", "PathTypeFilenameIndex")
};
return await Collection
.Find(f => f.Metadata["path"] == path, options)
.SortBy(f => f.Metadata["type"])
.ThenBy(f => f.Filename)
.ToListAsync();
I have a class that has Metadata field with path and type fields, also the class has a Filename field. I want all documents with a given path inside the metadata sorted by type and then by Filename.
An example result would be a list of documents ordered by the Name field like this:
a, Ab, B, c, D
Unfortunately, I get something like this:
Ab, B, D, a, c
And that's because MongoDB sorts the data with a simple binary comparison, where 'A' < 'a' because of their ASCII codes.
So my question is: Is there a way to make a case insensitive sort and keep using the "$hint"?
That options I pass to the Find method should tell MongoDB which index to use. I found this post: MongoDB and C#: Case insensitive search but the method here doesn't work for sorting and I couldn't tell MongoDB which index to use.

I think you can use aggregation pipeline with $addFields, $toLower (to convert filename to lowercase in temporary field), and $sort to sort them irrespective of the case
In mongodb shell you would write something like this :
db.collection.aggregate([{
$addFields : {
"lowercaseFileName" : {
$loLower : "$fileName"
}
},{
$sort : {
"metadata.type" : 1,
lowercaseFileName : 1
}
}
}])
Please write the similar code in c#, and see if it works. I dont know c#, otherwise i would have given you the exact query, but i cant.
The idea is to transform the filename to lowercase, save it in temporary field, using addFields and sort by that field.
Hope this helps you out.
Read more about $addFields, $toLower here.
Update
For whoever wants a working code in C# , thanks to #kaloyan-manev
You can use this :
return await Collection.Aggregate()
.Match(f => f.Metadata["path"] == path)
.AppendStage<BsonDocument>(new BsonDocument("$addFields", new BsonDocument("lowercaseFileName", new BsonDocument("$toLower", "$filename"))))
.AppendStage<GridFSFileInfo>(new BsonDocument("$sort", new BsonDocument { {"metadata.type", 1}, {"lowercaseFileName", 1} }))
.ToListAsync();

Did you try to set the CollationStrenght = 2?
Your code would be similar all you need is to set the Collation in the FindObject:
var options = new FindOptions
{
Modifiers = new BsonDocument("$hint", "PathTypeFilenameIndex"),
Collation = new Collation("en", strength: CollationStrength.Secondary)
};

Iterate over 2 nested lists

I am writing a weather app and need to go through 2 nested loops. For the return value I want to iterate over the first list, looking at the corresponding second list data. When the data in the second list matches the bool, I need to get data from the corresponding first list. Now I think that my code works... but would like to ask if this is a good way to do this. I am also not sure if this LINQ query will work in general, with even more nested lists. Here's my approach in LINQ:
public static async Task<string> UpdateWeather(string lat, string lon)
{
WeatherObject weather = await WeatherAPI.GetWeatherAsync(lat, lon);
var first = (from l in weather.list
from w in l.weather
where w.id == 800
select l.city.name).First();
return first;
}

Your code is OK, it is a LINQ query.But one more thing. Use FirstOrDefault() instead of First(). First() will throw an exception if no matched element is found, but FirstOrDefault() will return the element or the default value.
You can also write in LINQ Method syntax if you prefer this.
public static async Task<string> UpdateWeather(string lat, string lon)
{
WeatherObject weather = await WeatherAPI.GetWeatherAsync(lat, lon);
var first = weather.list.Where(l => l.weather.Any(w => w.id == 800))
.Select(l => l.city.name)
.FirstOrDefault();
return first;
}

I believe your query should work, and it should generally work with more nested lists following a similar structure. As to if it is a good way to do this - it depends on the data structure and any data constraints.
For example, if two elements in weather.list have an element in their nested weather list might have the same id, then your code will only return the first one - which may not be correct.
e.g. in json:
[
{
city : {
name : "Chicago"
},
weather : [
{
id = 799
},
{
id = 800
}
]
},
{
city : {
name : "New York"
},
weather : [
{
id = 800
},
{
id = 801
}
]
}
}
For this dataset, your code will return "Chicago", but "New York" also matches. This may not be possible with the data API you are accessing, but given that there are no data constraints to ensure exclusivity of the nested lists, you might want to defensively check that there is only 0 or 1 elements in the returned list that match the expected criteria.
Another suggestion
On another note, not strictly an answer to your question - if you think your code will work but aren't sure, write a unit test. In this case, you'd wrap the call to WeatherAPI in a class that implements an interface you define. Update your method to call the method on a reference to the interface.
For your real application, ensure that an instance of the wrapper/proxy class is set on the reference.
For the unit test, use a framework like Moq to create a mock implementation of the interface that returns a known set of data and use that instead. You can then define a suite of unit tests that use mocks that return different data structures and ensure your code works under all expected structures.
This will be a lot easier if your class is not a static method as well, and if you can use dependency injection (Ninject, Autofac or one of many others...) to manage injecting the appropriate implementation of the service.
Further explanations of unit testing, dependency injection and mocking will take more than I can write in this answer, but I recommend reading up on it - you'll never find yourself thinking "I think this code works" again!

Sorting by aggregate of two fields

I have a mongo database with documents that look like this:
{
PublishedDate: [date],
PublishedDateOverride: [NullableDate],
...
}
The reason I have the override as a separate field is that it is important to know the original published date as well as the overridden one.
When I get these documents back I want to sort them by their "apparent" published date. That is if there is an override it should use that, otherwise use the original.
Our current system just sorts by PublishedDateOverride and then by PublishedDate which of course groups all of those with a null override together.
For a concrete example take the following four documents:
A = {
PublishedDate: 2014-03-14,
PublishedDateOverride: 2014-03-24,
...
}
B = {
PublishedDate: 2014-01-21,
PublishedDateOverride: 2014-02-02,
...
}
C = {
PublishedDate: 2014-03-01,
PublishedDateOverride: null,
...
}
D = {
PublishedDate: 2014-03-27,
PublishedDateOverride: null,
...
}
The desired sort order would be D (2014-03-27), A (2014-03-14), C (2014-03-01), B (2014-02-02).
I need to be able to do this in the database since I am also paging this data so I can't just sort after getting it out of the database.
So the question:
What is the best way to achieve this goal? Is there a way to sort by an expression? Is there a way to have a calculated field such that whenever I update a document it will put the appropriate date in there to sort on?
I'm doing this in C# in case that is relevant but I would assume any solution would be a mongo one, not in my client code.

If you want a projection of only the valid and greater date then use aggregate with the $cond operator and the $gt operator. A basic shell example for translation (which is not hard) :
db.collection.aggregate([
{ "$project": {
"date": { "$cond": [
{ "$gt": [
"$PublishedDate",
"$PublishedDateOverride"
]},
"$PublishedDate",
"$PublishedDateOverride"
]}
}},
{ "$sort": { "date": 1 } }
])
So that basically breaks down your documents to having the "date" field set to which ever of those two fields had the greater value. Then you can sort on the result. All processed server side.

try
datesCollection.OrderBy(d => d.PublishedDateOverride!= null? d.PublishedDateOverride: d.PublishedDate)

Use Indexes to Sort Query Results
To sort on multiple fields, create a compound index.