Azure Cognitive Search - match exact phrase for some fields - c#

I have different fields in my Azure Cognitive Search but let me show you some with which I have problems.
{
"name": "Name",
"type": "Edm.String",
"searchable": true,
"filterable": false,
"retrievable": true,
"sortable": true,
"facetable": false,
"key": false,
"indexAnalyzer": null,
"searchAnalyzer": null,
"analyzer": "standard.lucene",
"synonymMaps": []
}
and
{
"name": "Code",
"type": "Edm.String",
"searchable": true,
"filterable": false,
"retrievable": true,
"sortable": false,
"facetable": false,
"key": false,
"indexAnalyzer": null,
"searchAnalyzer": null,
"analyzer": "keyword",
"synonymMaps": []
}
As you can see above, for Name I set analyzer standard.lucene (I have language-specific for other fields like NameEn) and keyword analyzer for Code field.
For example, when I search by 1-1 it looks for 1 instead of 1-1. I try with double quotes but it seems I also don't work ("1-1").
The issue is that as a result I get Name with the number 1 instead of Code which have 1-1.
Do you have any idea how can I do it? I suppose I should search by the whole phrase like: "1-1" rest part of the query.


When you send query it is analyzed by analyzers of all searchable fields and then tokenized query (different for each field) will be executed against all of them.
You can send queries to analyze endpoint to debug how each analyzer is working with your query -
https://serviceName.search.windows.net/indexes/index-name/analyze?api-version=2020-06-30
In your case:
{
"text": "1-1",
"analyzer": "standard"
}
returns these tokens for Name field
"tokens": [
{
"token": "1",
"startOffset": 0,
"endOffset": 1,
"position": 0
},
{
"token": "1",
"startOffset": 2,
"endOffset": 3,
"position": 1
}
]
and for Code field
{
"text": "1-1",
"analyzer": "keyword"
}
you get
"tokens": [
{
"token": "1-1",
"startOffset": 0,
"endOffset": 3,
"position": 0
}
]
So with such query you are really looking for documents with
Name=1 | Code=1-1
If you want to search only in selected fields you can specify them using searchFields parameter.

Related

Solr .Net Facet Term for obtaining numBuckets

Trying to implement with the following Facet Term which works when querying through postman but I cannot find any equivalent in Solr.NET as there are not facet term queries. I can see by field, range, and pivots. I would need to get the numBuckets and I do not see how this can me achieved in with the Solr.NET libraries.
Postman body example.
{
"query": "*:*",
"fields": [
"user_id"
],
"filter": [
"",
],
"facet": {
"user_id": {
"mincount": 1,
"numBuckets": true,
"allBuckets": false,
"offset": 0,
"type": "terms",
"field": "user_id",
"limit": 1
}
},
"params": {
"echoParams": "none"
},
"limit": 100,
"offset": 0,
"sort": "ismail_bius asc,createdon_zius desc"
}
I have tried to pass it in the ExtraParams property of the query options but no luck on it.
Thanks.

How to skip fields with default values in JSON response in HotChocolate/GraphQl

I want to remove fields from output JSON with default values (0, false) to reduce size of response, but can't find a way how to do it in HotChocolate/GraphQl.
I didn't find a settings to control it (JsonFormatter), "Optional" also not for this purposes, and there is not options to do it via query syntax.
After query to graphQl I receive a long list, with many default values in JSON output, simple example, pay attention to "sortOrder" and "isDeleted":
"values": [{
"id": 448,
"name": "Dr.",
"isDeleted": false,
"sortOrder": 0
}, {
"id": 449,
"name": "Mr.",
"isDeleted": true,
"sortOrder": 0
}, {
"id": 450,
"name": "Mrs.",
"isDeleted": false,
"sortOrder": 1
}]
Expecting result:
"values": [{
"id": 448,
"name": "Dr.",
}, {
"id": 449,
"name": "Mr.",
"isDeleted": true,
}, {
"id": 450,
"name": "Mrs.",
"sortOrder": 1
}]
Query:
getLookups(withDeleted: true) {
id
name
isDeleted
values {
id
name
isDeleted
sortOrder
}
}
Type description:
descriptor.BindFields(BindingBehavior.Explicit);
descriptor.Interface<ILookupValueType>();
descriptor.Field(f => f.Id).Type<NonNullType<IntType>>();
descriptor.Field(f => f.Name).Type<StringType>();
descriptor.Field(f => f.Image).Type<StringType>();
descriptor.Field(f => f.IsDeleted).Type<BooleanType>();
descriptor.Field(f => f.IsDefault).Type<BooleanType>();
descriptor.Field(f => f.SortOrder).Type<IntType>();
P.S. Please, don't offer pagination, it is not what I need.
And thank you for your help.

Azure Search - Accent insensitive analyzer not working when sorting

I'm using Azure Search. I have a model with a property with this attributes
[IsRetrievable(true), IsSearchable, IsSortable, Analyzer("standardasciifolding.lucene")]
public string Title { get; set; }
I want the search to be accent insensitive. Although it is working when searching/filtering, it is not working when sorting the results. So, If I have words that start with an accent and I sort alphabetically, those results appear at the end of the list.

I verified your use case by creating an index with Id and a Title field that uses the standardasciifolding.lucene analyzer. I then submitted the 4 sample records via the REST API:
{
"value": [
{
"#search.action": "mergeOrUpload",
"Id": "1",
"Title" : "øks"
},
{
"#search.action": "mergeOrUpload",
"Id": "2",
"Title": "aks"
},
{
"#search.action": "mergeOrUpload",
"Id": "3",
"Title": "áks"
},
{
"#search.action": "mergeOrUpload",
"Id": "4",
"Title": "oks"
}
]}
I then ran a query with $orderby specified. I used Postman with variables wrapped in double curly braces. Replace with relevant values for your environment.
https://{{SEARCH_SVC}}.{{DNS_SUFFIX}}/indexes/{{INDEX_NAME}}/docs?search=*&$count=true&$select=Id,Title&searchMode=all&queryType=full&api-version={{API-VERSION}}&$orderby=Title asc
The results were returned as
{
"#odata.context": "https://<my-search-service>.search.windows.net/indexes('dg-test-65224345')/$metadata#docs(*)",
"#odata.count": 4,
"value": [
{
"#search.score": 1.0,
"Id": "2",
"Title": "aks"
},
{
"#search.score": 1.0,
"Id": "4",
"Title": "oks"
},
{
"#search.score": 1.0,
"Id": "3",
"Title": "áks"
},
{
"#search.score": 1.0,
"Id": "1",
"Title": "øks"
}
]
}
So, the sort order is indeed a, o, á, ø which confirms what you find. The order is inversed if I change to $orderby=Title desc. Thus, the sorting appears to be done by the original value and not the normalized value. We can check how the analyzer works, by posting a sample title to the analyzer with a POST request to
https://{{SEARCH_SVC}}.{{DNS_SUFFIX}}/indexes/{{INDEX_NAME}}/docs?search=*&$count=true&$select=Id,Title&searchMode=all&queryType=full&api-version={{API-VERSION}}&$orderby=Title asc
{ "text": "øks", "analyzer": "standardasciifolding.lucene" }
Which produces the following tokens
{
"#odata.context": "https://<my-search-service>.search.windows.net/$metadata#Microsoft.Azure.Search.V2020_06_30_Preview.AnalyzeResult",
"tokens": [
{
"token": "oks",
"startOffset": 0,
"endOffset": 3,
"position": 0
},
{
"token": "øks",
"startOffset": 0,
"endOffset": 3,
"position": 0
}
]
}
You could try to define a custom analyzer which produces a normalized version, but I am not sure it will work. For example, the sorting does not appear to support case-insensitive sorting which would be related to this use case where multiple characters should be sorted as if they were a normalized version. E.g. a and A cannot be sorted as the same character according to this user voice entry (feel free to vote for it).
WORKAROUND
The best workaround I can think of is to process the data yourself. Let Title contain the original title, and then create another field called TitleNormalized where you store the normalized version. In your application you would then query with $orderby on the TitleNormalized field.

There is a new feature that allows you to enable normalization while filtering. Please check out the Text normalization for case-insensitive filtering, faceting and sorting feature that's in Preview.
You can update your index to use this "normalizer" feature for the fields in which you'd like case-insensitive order-by operations.
You don't need a separate field TitleNormalized anymore. You can add "normalizer": "standard" to the Title field, and $orderBy=Title will sort in the order you'd like, ignoring casing and accents.
The "standard" normalizer is pre-defined. If you'd like other filters to be applied, please look at predefined and custom normalizers
"index": {
"name": "your-name-here",
"fields": [
{"name": "Title", "type": "Edm.String", "filterable": true, "sortable": true, "facetable": false, "searchable": true, "normalizer":"standard"}
]
}

Azure Cosmos DB Add Composite Index for Array of String

I am trying to add a new composite index to do multiple fields search.
I would like to know thing to consider while adding a new Composite index and will it work for array string?
Sample Cosmos Document
{
"id": "ed78b9b5-764b-4ebc-a4f2-6b764679",
"OrderReference": "X000011380",
"SetReferences": [
"000066474884"
],
"TransactionReference": "ed78b9b5-764b-4ebc-6b7644f06679",
"TransactionType": "Debit",
"Amount": 73.65,
"Currency": "USD",
"BrandCode": "TestBrand",
"PartitionKey": "Test-21052020-255",
"SettlementDateTime": "2020-05-21T04:35:35.133Z",
"ReasonCode": "TestReason",
"IsProcessed": true,
}
My Existing index policy
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/PartitionKey/?"
},
{
"path": "/BrandCode/?"
}
],
"excludedPaths": [
{
"path": "/*"
},
{
"path": "/\"_etag\"/?"
}
],
"compositeIndexes": [
[
{
"path": "/PartitionKey",
"order": "ascending"
},
{
"path": "/IsProcessed",
"order": "ascending"
}
]
]
}
To fetch data from Array of string SettlementReferences, IsProcessed, ReasonCode.
SELECT * FROM c WHERE ARRAY_CONTAINS(c.SettlementReferences, '00884') and c.IsProcessed = true and c.ReasonCode = 'TestReason'
I am planning to add the following policy
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/PartitionKey/?"
},
{
"path": "/BrandCode/?"
}
],
"excludedPaths": [
{
"path": "/*"
},
{
"path": "/\"_etag\"/?"
}
],
"compositeIndexes": [
[
{
"path": "/PartitionKey",
"order": "ascending"
},
{
"path": "/IsProcessed",
"order": "ascending"
}
],
[
{
"path": "/SettlementReferences",
"order": "ascending"
},
{
"path": "/IsProcessed",
"order": "ascending"
},
{
"path": "/ReasonCode",
"order": "ascending"
}
]
]
}
Please let me know if this change is sufficient?
Moreover, I tried to compare RU's before and after the change. I don't see any massive difference, both coming around 133.56 Rus.
Is there anything more I need to consider for optimized performance?

Composite Indexes will not help with this query and overall don't have any impact on equality statements. They are useful when doing order by's in your queries. This is why you don't see any RU/s reduction in your query. However, you will notice increased RU/s on writes.
If you want to improve your query performance you should add any properties in your where clauses into the "includedPaths" in your index policy.
Another thing to point out too is generally it is a best practice to by default index everything and selectively add properties to excludedPaths. This way, if your schema changes it will be indexed automatically without having to rebuilt your index.

As mark mentioned we need to add include path for Array "/SettlementReferences /[]/?". After adding my number of Ru's reduced from 115 to 5 ru's.

I am trying to deserialize JSON data to a dataset

I thought JSON was supposed to be easy to use.
I have a C# .NET Windows form program in Visual Studio 2015 being used as a client consuming a Web API 2 remote server with RestSharp as the HTTP client.
the data returned from the API call is in Json format. However, when I try to deserialize the returned Json into a dataset using this statement:
DataSet data = (DataSet)JsonConvert.DeserializeObject(response.Content, (typeof(DataSet))); ,
I get this error:
"Unexpected JSON token while reading DataTable: EndArray".
If I use the text from "response.Content" and validate it here: http://www.jsonlint.com/#, the results says "Valid JSON".
When I use the debugger to view the "response.Content" as Json, it shows the correct records that I am looking for.
I have tried different formats of JsonConvert, such as:
DataSet data = JsonConvert.DeserializeObject<DataSet>(response.Content);
DataTable datat = (DataTable)JsonConvert.DeserializeObject(response.Content, (typeof(DataTable)));
I also tried this:
XmlDocument xd1 = new XmlDocument();
xd1 = (XmlDocument)JsonConvert.DeserializeXmlNode(response.Content, "RootObject");
DataSet dataset = new DataSet();
dataset.ReadXml(new XmlNodeReader(xd1));
which I got from here: http://www.c-sharpcorner.com/blogs/how-to-parse-json-string-to-dataset-in-c-sharp1
That seems to work without giving an error, but the dataset contains four tables ( RootObject, data, CustomerRoles and ShoppingCartitems).
I tried removing three of the tables, but there are constraints and foreignkeys that I don't know how to remove and it won't allow me to remove the tables until I remove the constraints. If I try to remove the constraints, it gives this error: "Cannot remove unique constraint 'Constraint1'. Remove foreign key constraint 'RootObject_data' first."
Does anybody know how to use the "easy to use", "most popular" JSON format to deserialize my Json data into a dataset and come out with one table of data?
JSON data
{
"success": true,
"data": [{
"Id": 6,
"CustomerGuid": "70b390d8-82d5-4bba-aa68-fc8268a1b1ff",
"UserName": "victoria_victoria#nopCommerce.com",
"Email": "victoria_victoria#nopCommerce.com",
"CustomerRoles": [{
"Id": 3,
"Name": "Registered",
"SystemName": "Registered"
}],
"AdminComment": null,
"IsTaxExempt": false,
"AffiliateId": 0,
"VendorId": 0,
"HasShoppingCartItems": false,
"Active": false,
"Deleted": false,
"IsSystemAccount": false,
"SystemName": null,
"LastIpAddress": null,
"CreatedOnUtc": "\/Date(1472933472393)\/",
"LastLoginDateUtc": null,
"LastActivityDateUtc": "\/Date(1472933472393)\/",
"ExternalAuthenticationRecords": [],
"ShoppingCartItems": []
}, {
"Id": 5,
"CustomerGuid": "eb9e6f24-f362-4c10-942a-366e2919dc11",
"UserName": "brenda_lindgren#nopCommerce.com",
"Email": "brenda_lindgren#nopCommerce.com",
"CustomerRoles": [{
"Id": 3,
"Name": "Registered",
"SystemName": "Registered"
}],
"AdminComment": null,
"IsTaxExempt": false,
"AffiliateId": 0,
"VendorId": 0,
"HasShoppingCartItems": false,
"Active": false,
"Deleted": false,
"IsSystemAccount": false,
"SystemName": null,
"LastIpAddress": null,
"CreatedOnUtc": "\/Date(1472933472363)\/",
"LastLoginDateUtc": null,
"LastActivityDateUtc": "\/Date(1472933472363)\/",
"ExternalAuthenticationRecords": [],
"ShoppingCartItems": []
}, {
"Id": 4,
"CustomerGuid": "9f46dbae-6942-410c-90b8-9b38a0890064",
"UserName": "james_pan#nopCommerce.com",
"Email": "james_pan#nopCommerce.com",
"CustomerRoles": [{
"Id": 3,
"Name": "Registered",
"SystemName": "Registered"
}],
"AdminComment": null,
"IsTaxExempt": false,
"AffiliateId": 0,
"VendorId": 0,
"HasShoppingCartItems": false,
"Active": false,
"Deleted": false,
"IsSystemAccount": false,
"SystemName": null,
"LastIpAddress": null,
"CreatedOnUtc": "\/Date(1472933472317)\/",
"LastLoginDateUtc": null,
"LastActivityDateUtc": "\/Date(1472933472317)\/",
"ExternalAuthenticationRecords": [],
"ShoppingCartItems": []
}, {
"Id": 3,
"CustomerGuid": "6277386b-13ee-427b-9cfe-4ebfa487c340",
"UserName": "arthur_holmes#nopCommerce.com",
"Email": "arthur_holmes#nopCommerce.com",
"CustomerRoles": [{
"Id": 3,
"Name": "Registered",
"SystemName": "Registered"
}],
"AdminComment": null,
"IsTaxExempt": false,
"AffiliateId": 0,
"VendorId": 0,
"HasShoppingCartItems": false,
"Active": false,
"Deleted": false,
"IsSystemAccount": false,
"SystemName": null,
"LastIpAddress": null,
"CreatedOnUtc": "\/Date(1472933472253)\/",
"LastLoginDateUtc": null,
"LastActivityDateUtc": "\/Date(1472933472253)\/",
"ExternalAuthenticationRecords": [],
"ShoppingCartItems": []
}, {
"Id": 2,
"CustomerGuid": "241f45f1-b38c-4e22-8c5a-743fa3276620",
"UserName": "steve_gates#nopCommerce.com",
"Email": "steve_gates#nopCommerce.com",
"CustomerRoles": [{
"Id": 3,
"Name": "Registered",
"SystemName": "Registered"
}],
"AdminComment": null,
"IsTaxExempt": false,
"AffiliateId": 0,
"VendorId": 0,
"HasShoppingCartItems": false,
"Active": false,
"Deleted": false,
"IsSystemAccount": false,
"SystemName": null,
"LastIpAddress": null,
"CreatedOnUtc": "\/Date(1472933472207)\/",
"LastLoginDateUtc": null,
"LastActivityDateUtc": "\/Date(1472933472207)\/",
"ExternalAuthenticationRecords": [],
"ShoppingCartItems": []
}, {
"Id": 1,
"CustomerGuid": "a940dc03-5f52-47d2-9391-8597b3b31cf2",
"UserName": "tony#lakesideos.com",
"Email": "tony#lakesideos.com",
"CustomerRoles": [{
"Id": 1,
"Name": "Administrators",
"SystemName": "Administrators"
}, {
"Id": 2,
"Name": "Forum Moderators",
"SystemName": "ForumModerators"
}, {
"Id": 3,
"Name": "Registered",
"SystemName": "Registered"
}],
"AdminComment": null,
"IsTaxExempt": false,
"AffiliateId": 0,
"VendorId": 0,
"HasShoppingCartItems": true,
"Active": true,
"Deleted": false,
"IsSystemAccount": false,
"SystemName": null,
"LastIpAddress": "71.185.255.7",
"CreatedOnUtc": "\/Date(1472933470783)\/",
"LastLoginDateUtc": "\/Date(1477522483903)\/",
"LastActivityDateUtc": "\/Date(1477523996553)\/",
"ExternalAuthenticationRecords": [],
"ShoppingCartItems": [{
"Id": 1,
"StoreId": 1,
"ShoppingCartTypeId": 1,
"CustomerId": 1,
"ProductId": 18,
"AttributesXml": null,
"CustomerEnteredPrice": 0.0000,
"Quantity": 1,
"CreatedOnUtc": "\/Date(1473801903447)\/",
"UpdatedOnUtc": "\/Date(1473803336207)\/",
"IsFreeShipping": false,
"IsShipEnabled": true,
"AdditionalShippingCharge": 0.0000,
"IsTaxExempt": false
}]
}]
}

Here's a brief overview of what you need to do.
Taking from NewtonSoft - Deserialize Object example
Snippet of your data
{
"Id": 1,
"CustomerGuid": "a940dc03-5f52-47d2-9391-8597b3b31cf2",
"UserName": "tony#lakesideos.com",
"Email": "tony#lakesideos.com"
This is a Customer object. It needs it's own C# model class.
public class Customer {
int id;
string guid;
string username;
string email;
// etc...
// getters and setters...
}
This is stored in a list of data
"data": [ { ... }, { ... } ]
You need another Object to hold a list of those objects plus a boolean success value. Let's call this one Response.
{
"success": true,
"data": [{
That class starts with
public class Response {
bool success;
IList<Customer> data;
Then, you need other lists and object classes for something like this
"CustomerRoles": [{
"Id": 3,
"Name": "Registered",
"SystemName": "Registered"
}],
Once you've modeled the entire domain of classes, you should be able to have
Response res = JsonConvert.DeserializeObject<Response>(json);
Since you seem to have already tried that, and are getting an error about the END_ARRAY character, something within your DataSet class is not correctly mapped to a List
And then get res.data to access the list of data, then you can loop over those Customer objects to build your Dataset / table objects, I assume to display your data in some list of the GUI

Try starting from Response.Data instead of Response.Content, that will get you one layer in to expose the actual data to deserialize.
For example:
var response = _requestProcessor.Process<Request, Response>(
new Request(id)).dataList;
return new JsonNetResult() { Data = response, JsonRequestBehavior= JsonRequestBehavior.AllowGet };
Whatever the Request object is, it has an object that holds the responses' data in it; that is the actual item to deserialize.
Perhaps what you need to do is what Cricket_007 suggests, and make a class to encapsulate the Success object returned in the JSON, then deserialize to one Success object and a Data object, which is a list of the objects you're trying to get to.
The steps you need to follow are:
Convert the whole JSON string to an object that consists of the two objects within, Success and Data.
JSONObject content = new JSONObject(Response.Content); perhaps, this is untested.
From that object, deserialize just the Data object within it.
content.data -or- content["data"]
or some such should get you to the parts you want to work with.

Categories

Resources