Using PIT (Point in Time) ID and Search_After API -NEST ElasticSearch - c#

I have been reading up on Point in time API and wanted to implement it using NEST in my .net application. However, when reading that article (.net application hyperlink), I saw the Fluent DSL example as shown below. Is there a way to find that ID without having to go to kibana on the console and doing an query search to get the id to then later place that id inside of "a-point-in-time-id"? or does "a-point-in-time-id" do that for you like is that mapped to the ID?
s => s
.PointInTime("a-point-in-time-id", p => p
.KeepAlive("1m"))
I know in the kibana cli console if you do:
POST /app-logs*/_pit?keep_alive=5m it will give you a PIT (point in time) id. How do you go about retrieving that in NEST?
and when reading up on search_after and attempting to implement it using the search after usage for the .net client using the Fluent DSL Example. I noticed that they had they word "Project" but it does not say what "Project is in the example. What would that be exactly?
s => s
.Sort(srt => srt
.Descending(p => p.NumberOfCommits)
.Descending(p => p.Name)
)
.SearchAfter(
Project.First.NumberOfCommits,
Project.First.Name
)
Here I attempted to implement .PointInTime() and .Sort() and .SearchAfter() but got stuck.
var response = await _elasticClient.SearchAsync<Source>(s => s
.Size(3000) // must see about this
.Source(src => src.Includes(i => i
.Fields(f => f.timestamp,
fields => fields.messageTemplate,
fields => fields.message)))
.Index("app-logs*"
.Query(q => +q
.DateRange(dr => dr
.Field("#timestamp")
.GreaterThanOrEquals("2021-06-12T16:39:32.727-05:00")
.LessThanOrEquals(DateTime.Now))))
.PointInTime("", p => p
.KeepAlive("5m"))
.Sort(srt => srt
.Ascending(p => p.timestamp))
.SearchAfter()
I know when you are using PIT ID you do not need the index in the search, but in the hyperlink example it does not show how you would go about implementing that. So just a bit lost on how to do so. Any pointers/guidance/tutorials would be great!
But just trying to see how I can do this in NEST but if you are saying its apart of the XPACK, I would understand.

I was able to combine PIT and SearchAfter with the code sample below:
var pit = string.IsNullOrEmpty(pitId) ? client.OpenPointInTime(new OpenPointInTimeDescriptor().KeepAlive(ServerDetail.TTL).Index(Indices.Index(Name))) : null;
var request = new SearchRequest(Name)
{
SearchAfter = string.IsNullOrEmpty(pitId) ? null : new string[] { last },
Size = ServerDetail.PageSize,
PointInTime = new PointInTime(pit == null ? pitId : pit.Id)
};
List<FieldSort> sorts = new List<FieldSort>();
foreach (var item in sortBy)
{
sorts.Add(new FieldSort() { Field = item.Field, Order = item.Direction == SortDirection.Asc ? SortOrder.Ascending : SortOrder.Descending });
}
request.Sort = sorts.ToArray();
Your SearchAfter value should be values from your sort fields for the last object of your previous search result.
For the 1st search pitId is null so a new one is created, for subsequent requests, pitId is passed.
Hope that works for you?

Related

OrderByDescending not working in my LINQ query

I have come across a problem with a query where I'm trying to sort the result by descending but it doesn't do it. I have gone through few posts on stackoverflow with similar issue but none of the solutions seemed to work. I was wondering if someone could point out what is wrong with my code. Heres part of the query relevant to a problem:
from followup in ctx.FollowUps
.Where(f => f.DestEntityId == user.Id
&& f.DestEntityType == (int)ContactEntityTypeEnum.User)
.OrderByDescending(x => x.AddDate)
select new
{
user = new { Firstname = user.Firstname },
followup = new { followup.NextFollowUpDate }
});
I also tried adding .ToList() before .OrderByDescending() but that didn't solve the problem either.
Following up the comments from #SILENT & #D.Shih your Linq expression is looking a bit odd.
var followUps = ctx.FollowUps
.Where(f => f.DestEntityId == user.Id
&& f.DestEntityType == (int)ContactEntityTypeEnum.User)
.OrderByDescending(x => x.AddDate)
.Select(x => new
{
UserFirstname = user.Firstname,
AddDate = x.AddDate,
FollowupDate = x.NextFollowUpDate
}).ToList();
This will order the results by AddDate, not NextFollowUpDate, so I've added AddDate to the output just to avoid confusion between the two, since I don't know how you'd have assessed that they weren't in order based on only seeing the NextFollowUpDate. I don't recommend mixing the LinQL (from/where/select) with the Fluent methods. (above)
There doesn't look to be a point in returning each individual property in a separate anonymous type, just flatten it into a single package since anonymous types are intended for short lives and immediate consumption.

mongo Db unsupported filter type when using Any() or Contains() - Are there any workarounds / fixes?

i have seen other threads for this kind of question but none of the highlighted answers work for me.
The piece of code that throws the error is this
List<Distribution> distributionPublishQueueList = (mongoDb.GetCollection<Distribution>().Find(Builders<Distribution>.Filter.And(
Builders<Distribution>.Filter.Where(x => x.Status == EntityStatus.Ok),
Builders<Distribution>.Filter.Where(x => x.IsActive),
Builders<Distribution>.Filter.Where(x => distinctDistributionIdInPublishQueueList.Contains(x.Id))))).ToList();
The original code was this:
List<Distribution> distributionPublishQueueList =
(await mongoDb.GetCollection<Distribution>()
.FindAsync(x => x.Status == EntityStatus.Ok
&& x.IsActive
&& distinctDistributionIdInPublishQueueList.Contains(x.Id)))
.ToList();
but i tried to make it more mongo friendly. Both pieces of code above are the same. The list distinctDistributionIdInPublishQueueList is a list of distribution Id's which are strings. So i am trying to find all distributions which Id is inside of that list + the other 2 filters. When i use the contains inside of the filter definition it throws an Unsupported filter exception. But the following code works because i bring the list into local memory and use LINQ against it:
List<Distribution> distributionPublishQueueList = (await mongoDb.GetCollection<Distribution>().FindAsync(x => x.Status == EntityStatus.Ok && x.IsActive)).ToList();
distributionPublishQueueList = distributionPublishQueueList.Where(x => distinctDistributionIdInPublishQueueList.Contains(x.Id)).ToList();
I need to be able to not do this in local memory due to the amount of Distributions that are present in the database. Is there a workaround to using Contains and Any. I have also tried using MongoCSharpDriver In statement and Builders.Filter.In and other variations.
An example error would be as follows. This is the code that is used.
List<Asset> assetList = (await mongoDb.GetCollection<Asset>().FindAsync(
asset => extractAssetsFromContentService.ExtractAssetFromDraftContent(contentAsMarkdown)
.Any(extractedAsset => extractedAsset.AssetId == asset.Id))).ToList();
System.ArgumentException : Unsupported filter: Any(value(System.Collections.Generic.List`1[DocWorks.Common.Transformation.Model.ExtractedAssetModel]).Where(({document}{AssetId} == {document}{_id}))).
this would be the same error except the Any would be 'Contains' when using contains instead of any. Similar to a distribution i cannot bring the assets into local memory. All entities share the same base class which Stores the Ids as follows:
[BsonRepresentation(BsonType.ObjectId)]
public string Id { get; set; }
You can use any of the following solutions;
var filter = Builders<Distribution>.Filter.ElemMatch(x => x.Distincts, i => distinctIdList.Contains(i.Id));
var results = await (await _collection.FindAsync(filter)).ToListAsync();
or
Expression<Func<Distribution, bool>> filter = x => districtIdList.Contains(x.Id);
var results = await (await _collection.FindAsync(filter)).ToListAsync();
These two solutions work fine in my projects. I have a library where you can apply these usages. When you send an expression to any of the get methods here, it will work. For example;
var distributionPublishQueueList = await _distributionRepository.GetAllAsync(x => distinctDistributionIdInPublishQueueList.Contains(x.Id));
You can see the documentation from here.

NetSuite SuiteTalk - Retrieve Value String From "SearchColumnSelectCustomField"

I have a small application that iterates over the results of a "Saved Search" retrieving the values from several Custom Columns(simplified example):
var results = searchResults.Select(a => new
{
X = ((SearchColumnBooleanCustomField)a.basic.customFieldList
.First(b => b.scriptId == "custentityX")).searchValue
Y = ((SearchColumnDateCustomField)a.basic.customFieldList
.First(b => b.scriptId == "custentityY")).searchValue
Z = ((SearchColumnSelectCustomField)a.basic.customFieldList
.First(b => b.scriptId == "custentityZ")).searchValue.name
}
For most returned column types I get a value consistent with the type(bool/date/etc...) but with the "SearchColumnSelectCustomField" I don't get any value in the returned "searchValue.name", it's always null, however the "searchValue.internalID" column is always populated. So for some reason its getting the selection but not returning the value from that selection.
How do I access the text value that I can see from the NetSuite interface from SuiteTalk("searchValue.name")? Do I have to execute another query to retrieve all value key pairs related to that internalID? For every custom field? And if so then what is the purpose of the name field in the first place?
I've tried searching around for this, but there's not really allot of documentation on the subject(or on SuiteTalk in general), in other languages(PHP/Java) people mention "getSelectValue"(here, and briefly here), I could try this in C#, but I'm not sure if these apply or if that can be done for custom value selections.
Then there's some references for determining the values BEFORE the search, this seems like overkill to me, is it really that hard? I have dozens of CustomFields I would like to access in my final application. Seems like there should be an easier way...
As far as I know, the web service response will only contain the internalId and typeId for a SearchColumnSelectCustomField. In order to get the name, you'll have to first query NetSuite to find all custom lists and their values.
You can do this using a CustomListSearch and set the bodyFieldsOnly search preference to false. Pass in no criteria to the CustomListSearch and you'll be returned every custom list and their values. Just store there results in memory and reference it when reading the column values from your saved search.
I tried the answer posted by #Adud123 and it works great, Here's what the code looks like:
public Dictionary<string, Dictionary<long, string>> getCustomFieldLists()
{
return
nsService.search(new CustomListSearch())
.recordList.Select(a => (CustomList) a)
.ToDictionary(a => a.internalId,
a => a.customValueList.customValue
.ToDictionary(b => b.valueId, c => c.value));
}
var valueLookup = getCustomFieldLists();
var results = searchResults.Select(a => new
{
Z = (a.basic.customFieldList.Where(b => b.scriptId == "custentityZ")
.Select(a => (SearchColumnSelectCustomField)a)
.Select(a => valueLookup[a.searchValue.typeId][a.searchValue.internalId])
.First()
}

asp.net MVC Where in List

I building my first application with c# and sp.net MVC 5, so far so good :)
Now I have a problem, we using 2 User Tables, first one contains the username, other the user data.
string user = User.Identity.Name;
var data = db.FE_Benutzer;
var collection = data.Where(o => o.Benutzername == user).Select(x => new
{
id = x.ID,
name = x.Name,
hauptbereiche = x.Hauptbereich.ToList()
});
var dataHauptbereich = db.Hauptbereich;
var collectionHauptbereich = dataHauptbereich.Where(o => collection.ElementAt(0).hauptbereiche.Contains(o)).Select(x => new
{
id = x.ID,
name = x.Name
});
return Json(collectionHauptbereich, JsonRequestBehavior.AllowGet);
I getting this error
LINQ to Entities does not recognize the method '<>f__AnonymousType63[System.Int32,System.String,System.Collections.Generic.List1[scorring.Models.Hauptbereich]] ElementAt[<>f__AnonymousType63](System.Linq.IQueryable1[<>f__AnonymousType63[System.Int32,System.String,System.Collections.Generic.List1[scorring.Models.Hauptbereich]]], Int32)' method, and this method cannot be translated into a store expression.
hauptbereiche = x.Hauptbereich.ToList()
contains a list of ids where the user have premission to.
When I fetching the data
dataHauptbereich.Where
I wont to include only the ids I have in the list
how is this possible?
Entity Framework doesn't know how to turn ElementAt into SQL. See this answer for more information: Getting the first result from a LINQ query - why does ElementAt<T>(0) fails when First<T>() succeeds?
Try
dataHauptbereich.Where(o => collection.ElementAt(0).hauptbereiche.Any(h => h.ID == o.ID))
Or
dataHauptbereich.Where(o => collection.Any(c => c.hauptbereiche.Any(h => h.ID == o.ID)))
I'm having a bit of a time deciphering exactly what you're trying to achieve with your code here, but it looks to me like your simply querying Hauptbereichs that belong to a particular user. Your first query selects an anonymous object composed of id, name and hauptbereiche, but of these you only ever use the hauptbereiche property. Then, in your second query, you merely selecting Hauptbereichs that match an item in this hauptbereiche property's collection. Actually, here, you're only comparing values from the first item in the original collection, which begs the question of why you're selecting anything other than the first item. That, and this second query is entirely redundant because if the items match that means you already had the items in the first place. You could get the same info directly from collection.ElementAt(0).hauptbereiche without issuing the second query.
So, here's a couple of simpler options:
If you're trying to get all the Hauptbereichs that belong to all the FE_Benutzers where Benutzername == user then just do:
var collectionHauptbereich = db.FE_Benutzer.Where(m => m.Benutzername == user)
.Include(m => m.Hauptbereich)
.SelectMany(m => m.Hauptbereich);
If you want just the first FE_Benutzer item's Hauptbereichs, then do:
var benutzer = db.FE_Benutzer.Where(m => m.Benutzername == user)
.Include(m => m.Hauptbereich)
.FirstOrDefault();
var collectionHauptbereich = benutzer != null
? benutzer.Hauptbereich.ToList()
: new List<Hauptbereich>();

Nest SuggestCompletion usage, throws 'is not a completion suggest field' exception

I'm a complete beginner to elasticsearch and I have been trying to use elasticsearch's completion suggester using Nest for auto-complete on a property.
Here is my mapping (as mentioned here: ):
var createResult = client.CreateIndex(indexName, index => index
.AddMapping<Contact>(tmd => tmd
.Properties(props => props
.Completion(s =>
s.Name(p => p.CompanyName.Suffix("completion"))
.IndexAnalyzer("standard")
.SearchAnalyzer("standard")
.MaxInputLength(20)
.Payloads()
.PreservePositionIncrements()
.PreserveSeparators())
)
)
);
var resultPerson = client.IndexMany(documents.OfType<Person>(), new SimpleBulkParameters { Refresh = true });
var resultCompany = client.IndexMany(documents.OfType<Company>(), new SimpleBulkParameters { Refresh = true });
And while indexing I'm just making use of IndexMany method and passing the IEnumberable<Contact> (Contact has a property by name CompanyName, Contact is an abstract class, both Person and Company are concrete implementations of it). The search throws an exception saying ElasticSearchException[Field [companyName] is not a completion suggest field]. And the query looks like below:
SearchDescriptor<Contact> descriptor = new SearchDescriptor<Contact>();
descriptor = descriptor.SuggestCompletion("suggest", c => c.OnField(f => f.CompanyName).Text(q));
var result = getElasticClientInstance("contacts").Search<Contact>(body => descriptor);
string qe = result.ConnectionStatus.ToString();
What am I doing wrong here, I looked into Nest's tests on SuggestCompletion but not much help, meaning the test only depict on how to get suggestions but not on how to set index mappings for SuggestCompletion.
I also tried setting up edgeNgram tokenizer as mentioned in this post but, couldn't proceed there as well.
Any direction or an example on how to proceed would greatly help.
UPDATE
You are try to create a property with the name "companyName.completion" but at that position its not valid and it will use the last token "completion". So its actually mapping a field called completion.... try changing the call to: .Name(p => p.CompanyName)
Other observations
You specify a mapping for the Contact but while indexing you use the Person and Company types.
In elasticsearch terms you mapped:
/index/contact/
but your documents are going into:
/index/person/ and /index/company
NEST won't automatically map all implementation of a specific class and elasticsearch has no way of knowing the three are related.
I would refactor the mapping to a method and call it for all the types involved.
var createResult = client.CreateIndex(indexName, index => index
.AddMapping<Contact>(tmd => MapContactCompletionFields(tmd))
.AddMapping<Person>(tmd => MapContactCompletionFields(tmd))
.AddMapping<Company>(tmd => MapContactCompletionFields(tmd))
);
private RootObjectMappingDescriptor<TContact> MapContactCompletionFields<TContact>(
RootObjectMappingDescriptor<TContact> tmd)
where TContact : Contact
{
return tmd.Properties(props => props
.Completion(s => s
.Name(p => p.CompanyName.Suffix("completion"))
.IndexAnalyzer("standard")
.SearchAnalyzer("standard")
.MaxInputLength(20)
.Payloads()
.PreservePositionIncrements()
.PreserveSeparators()
)
);
}
That method returns the descriptor so you can further chain on it.
Then when you do a search for contacts:
var result = getElasticClientInstance("contacts").Search<Contact>(
body => descriptor
.Types(typeof(Person), typeof(Company))
);
That types hint will cause the search to looking /index/person and /index/company and will know how to give you back a covariant list of documents.
So you can do result.Documents.OfType<Person>() after the previous call.

Categories

Resources