Develop Reference

ElasticSearch - Nest - How do I get it to show results with hits score above 90%?

C# ElasticSearch How can I make a search on PredictText return results with a hit score above 90%?
var settings = new ConnectionSettings(new Uri("http://localhost:9200")).DefaultIndex("test1");
settings.BasicAuthentication("*****", "******");
var client = new ElasticClient(settings);
var text = "example text";
var searchResponse = client.Search<Flowi>(s => s
.From(0)
.Size(5)
.Query(q => q
.Match(m => m
.Field(f => f.PredictText)
.Query(text)
)
)
.Sort(sort => sort.Descending(SortSpecialField.Score))
);
foreach (var hit in searchResponse.Hits)
{
Console.WriteLine( "Id:{0} QuestionPath:{1} Filename:{2} Score:{3}",
hit.Source.Id, hit.Source.QuestionPath, hit.Source.Filename, hit.Score );
}

How to use GroupBy() in subqueries with LINQ?

I have pretty simple LINQ expression
IQueryable<FreeBetDTO> records = UnitOfWork.FreeBets
.Include(f => f.FreeBetCategories)
.Include(f => f.FreeBetCards)
.Where(f => f.FreeBetCards.Any(cards => cards.UserId == request.UserId))
.Select(f => new FreeBetDTO
{
FreeBetId = f.FreeBetId
LineCategories = f.FreeBetCategories
.GroupBy(g => new { g.LineCategoryID, g.Title })
.Select(c =>
new LineCategoryDTO
{
LineCategoryID = c.Key.LineCategoryID,
Title = c.Key.Title
}).AsEnumerable()
});
When I am executing it I catch the error:
System.InvalidOperationException: Unable to translate collection subquery in projection since it uses 'Distinct' or 'Group By' operations and doesn't project key columns of all of it's tables which are required to generate results on client side. Missing column: t.ID. Either add column(s) to the projection or rewrite query to not use 'GroupBy'/'Distinct' operation.
at Microsoft.EntityFrameworkCore.Query.SqlExpressions.SelectExpression.
....
The problem is here .GroupBy(g => new { g.LineCategoryID, g.Title }). If I don't group records, the error disappears.
I was trying a lot of cases with GroupBy() and Distinct(). But can't understand why this is happening. Because I just need grouping like this.

Error message says that you have to include Id column in projection. But you can't do that with GroupBy. So rewrite query into two steps (removed not needed includes):
var rawRecords = UnitOfWork.FreeBets
.Where(f => f.FreeBetCards.Any(cards => cards.UserId == request.UserId))
.Select(f => new
{
FreeBetId = f.FreeBetId
LineCategories = f.FreeBetCategories.Select(c => new { c.Id, c.LineCategoryID, c.Title })
.ToList()
})
.AsEnumerable();
var records = rawRecords
.Select(f => new FreeBetDTO
{
FreeBetId = f.FreeBetId
LineCategories = f.LineCategories.GroupBy(g => new { g.LineCategoryID, g.Title })
.Select(c =>
new LineCategoryDTO
{
LineCategoryID = c.Key.LineCategoryID,
Title = c.Key.Title
})
});
Similar query, but more optimal:
var query =
from f in UnitOfWork.FreeBets
from c in f.FreeBetCards
where f.FreeBetCards.Any(cards => cards.UserId == request.UserId)
select new { f.FreeBetId, c.LineCategoryID, c.Title };
query = query.Distinct();
var records = query.AsEnumerable()
.GroupBy(f => f.FreeBetId)
.Select(g => new FreeBetDTO
{
FreeBetId = g.Key
LineCategories = g.Select(c =>
new LineCategoryDTO
{
LineCategoryID = c.LineCategoryID,
Title = c.Title
})
.AsEnumerable()
});

Make Elasticsearch diacritics insensitive

I am using Elasticsearch 6.6.0 and NEST in a .NET MVC project.
I am indexing some products using this code:
var esSettings = new ConnectionSettings(node);
esSettings = esSettings.DefaultIndex(IndexInstanceName);
esSettings = esSettings
.DefaultMappingFor<SearchableProduct>(s => s.IdProperty("Id").IndexName(IndexInstanceName + "-products-" + ConfigurationManager.AppSettings["DefaultCulture"]));
var elastic = new ElasticClient(esSettings);
var mapResponse = elastic.Map<SearchableProduct>(x => x.AutoMap().Index(IndexInstanceName + "-products-" + culture));
var indexState = new IndexState
{
Settings = new IndexSettings()
};
indexState.Settings.Analysis = new Analysis
{
Analyzers = new Analyzers()
};
indexState.Settings.Analysis.Analyzers.Add("nospecialchars", new CustomAnalyzer
{
Tokenizer = "standard",
Filter = new List<string> { "standard", "lowercase", "stop", "asciifolding" }
});
//products
if (!elastic.IndexExists(IndexInstanceName + "-products-" + culture).Exists)
{
var response = elastic.CreateIndex(
IndexInstanceName + "-products-" + culture,
s => s.InitializeUsing(indexState)
.Mappings(m => m.Map<SearchableProduct>(sc => sc.AutoMap())));
}
await this.IndexProductsAsync(context, products, elastic, culture);
await elastic.RefreshAsync(new RefreshRequest(IndexInstanceName + "-products-" + culture));
and for the search I use the below code:
ISearchResponse<SearchableProduct> result = await elastic.SearchAsync<SearchableProduct>(s => s
.Index(elasticIndexName + "-products-" + culture)
.Take(DefaultPageSize)
.Source(src => src.IncludeAll())
.Query(query =>
query.QueryString(qs =>
qs.Query(q).DefaultOperator(Operator.And).Fuzziness(Fuzziness.EditDistance(0)).Fields(x => x.Field(d => d.Name, 2)
.Field(d => d.MetaTitle, 1)
.Field(d => d.Image, 1)
.Field(d => d.SystemId, 2)
.Field(d => d.Manufacturer, 1)
)
))
.Sort(d => d.Ascending(SortSpecialField.Score))
);
When i search for a word with accent in greek (eg παγωτό) I get results (Because in my index the product is indexed with accent), but when i use the same word without accent (eg παγωτο) i get no results.
Is anything wrong with the indexing settings or the search code?
Can I index my data without accents or alternatively index them as is but make the search or index accent insensitive?

Creating a field with a greek analyzer will make sure indexed text and query string pass the same analysis path. For παγωτό that means, during indexing, the text will be tokenized to παγωτ as well as during making the query request.
Please check my example which creates a field with greek analyzer and the example outputs both documents with παγωτό and παγωτο when looking for παγωτό or παγωτο.
class Program
{
static async Task Main(string[] args)
{
var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(connectionPool)
.DefaultIndex("index_name")
.DisableDirectStreaming()
.PrettyJson();
var client = new ElasticClient(settings);
await client.Indices.DeleteAsync("index_name");
var createIndexResponse = await client.Indices.CreateAsync("index_name",
c => c
.Map(map => map.AutoMap<Document>()));
await client.IndexManyAsync(new []
{new Document {Id = 1, Text = "παγωτό"}, new Document {Id = 2, Text = "παγωτο"},});
await client.Indices.RefreshAsync();
var query = "παγωτό";
var searchResponse = await client.SearchAsync<Document>(s => s
.Query(q => q.Match(m => m.Field(f => f.Text).Query(query))));
Console.OutputEncoding = Encoding.UTF8;
Print(query, searchResponse);
query = "παγωτο";
var searchResponse2 = await client.SearchAsync<Document>(s => s
.Query(q => q.Match(m => m.Field(f => f.Text).Query(query))));
Print(query, searchResponse2);
}
private static void Print(string query, ISearchResponse<Document> searchResponse)
{
Console.WriteLine($"For {query} found:");
foreach (var document in searchResponse.Documents)
{
Console.WriteLine($"Document {document.Id} {document.Text}");
}
}
}
public class Document
{
public int Id { get; set; }
[Text(Analyzer = "greek")]
public string Text { get; set; }
}
Prints:
For παγωτό found:
Document 1 παγωτό
Document 2 παγωτο
For παγωτο found:
Document 1 παγωτό
Document 2 παγωτο
Hope that helps.

Lambda Join Group by where clause issue

Why am I getting only one entry in DownTimeDetails list even though in Data we have 3 entries.
VehicleEventDetails Res = dbEntity.DownTimeHeaders
.Join(dbEntity.DownTimeDetails, dth => dth.DownTimeHeaderID, dtd => dtd.DownTimeHeaderID, (dth, dtd) => new { dth, dtd })
.Where(x => x.dth.DownTimeHeaderID == 42)
.GroupBy(gx => gx.dtd.DownTimeDetailID)
.Select(t => new VehicleEventDetails()
{
BookingId = t.Select(a => a.dth.BookingId).FirstOrDefault(),
DownTimeDetails = t.Select(ab => new DownTimeDetails
{
LocalDTStartTime = (DateTime)ab.dtd.LocalDTStartTime,
LocalDTEndTime = (DateTime)ab.dtd.LocalDTEndTime,
CalculatedEventDTReason = ab.dtd.CalculatedEventDTReason,
CalculatedEventDTInMinutes = (int)ab.dtd.CalculatedEventDT,
}).ToList()
}).FirstOrDefault();

You are looking for something like this:
VehicleEventDetails Res = dbEntity.DownTimeHeaders
.Where(x => x.DownTimeHeaderID == 42)
.Select(x => new VehicleEventDetails
{
BookingId = x.BookingId,
DownTimeDetails = x.DownTimeDetails
.Select(dtd=> new DownTimeDetails
{
LocalDTStartTime = (DateTime)dtd.LocalDTStartTime,
LocalDTEndTime = (DateTime)dtd.LocalDTEndTime,
CalculatedEventDTReason = dtd.CalculatedEventDTReason,
CalculatedEventDTInMinutes = (int)dtd.CalculatedEventDT,
})
.ToList()
})
.FirstOrDefault();
Notes:
Using .Join is an anti-Entity Framework pattern. Always try to use navigation properties, they exist for a reason.
Don't use .GroupBy unless you actually need a group. You don't want any grouping in this query.
As a general note, try not to make the expression variable names so confusing.

LINQ Method - Optimization

I'm reading a CSV file splitting it into cols, then grouping into a new class.
It looks clunky just wondering is there is a more simple method for instance like not selecting them into the class first:
EDIT: so to clarify I'm trying to get the TimesheetHours grouped by all the other columns.
var rowList = csvFile.Rows.Select(row => row.Split(','))
.Select(cols => new UtilisationRow {
UploadId = savedUpload.Id,
FullName = cols[0],
TimesheetWorkDateMonthYear = Convert.ToDateTime(cols[1]),
TimesheetTaskJobnumber = cols[2],
TimesheetWorktype = cols[3],
TimesheetHours = Convert.ToDouble(cols[4]),
TimesheetOverhead = cols[5]
})
.GroupBy(d => new {
d.FullName,
d.TimesheetWorkDateMonthYear,
d.TimesheetTaskJobnumber,
d.TimesheetWorktype,
d.TimesheetOverhead
})
.Select(g => new UtilisationRow {
FullName = g.First().FullName,
TimesheetWorkDateMonthYear = g.First().TimesheetWorkDateMonthYear,
TimesheetTaskJobnumber = g.First().TimesheetTaskJobnumber,
TimesheetWorktype = g.First().TimesheetWorktype,
TimesheetHours = g.Sum(s => s.TimesheetHours),
TimesheetOverhead = g.First().TimesheetOverhead
})
.ToList();
Many thanks,
Lee.

The two problems in your code are that you call First() repeatedly on a group, while you should retrieve that same data from group's key, and that you are using UtilisationRow in the first Select, which should use an anonymous type instead:
var rowList = csvFile.Rows.Select(row => row.Split(','))
.Select(cols => new {
UploadId = savedUpload.Id,
FullName = cols[0],
TimesheetWorkDateMonthYear = Convert.ToDateTime(cols[1]),
TimesheetTaskJobnumber = cols[2],
TimesheetWorktype = cols[3],
TimesheetHours = Convert.ToDouble(cols[4]),
TimesheetOverhead = cols[5]
})
.GroupBy(d => new {
d.FullName,
d.TimesheetWorkDateMonthYear,
d.TimesheetTaskJobnumber,
d.TimesheetWorktype,
d.TimesheetOverhead
})
.Select(g => new UtilisationRow {
FullName = g.Key.FullName,
TimesheetWorkDateMonthYear = g.Key.TimesheetWorkDateMonthYear,
TimesheetTaskJobnumber = g.Key.TimesheetTaskJobnumber,
TimesheetWorktype = g.Key.TimesheetWorktype,
TimesheetHours = g.Sum(s => s.TimesheetHours),
TimesheetOverhead = g.Key.TimesheetOverhead
})
.ToList();
Now the "pipeline" of your method looks pretty clean:
The first Select does the initial parsing into a temporary record
GroupBy bundles matching records into a group
The final Select produces records of the required type.