I'm stuck at a problem, apparently I might have misunderstood something, but here is the problem.
I've got some testdata in a database, but when I run this:
Connection to elastic
private string uri = "http://localhost:9200/";
private string testindex = "testconnectiones";
private static ElasticClient GetClient(string testindex, string uri)
{
var pool = new SingleNodeConnectionPool(new Uri(uri));
var connectionSettings =
new ConnectionSettings(pool, sourceSerializer: JsonNetSerializer.Default).DefaultIndex(testindex);
return new ElasticClient(connectionSettings);
}
Then
public void TestCreateIndexBaseOnOrderId1()
{
var client = GetClient(testindex, uri);
var searchResponse = client.Search<TestLogs>(s => s
.AllTypes()
.From(0)
.Size(1000)
.Analyzer("standard")
.Query(q => q
.Match(m => m
.Field(f => f.OrderID)
.Query("")
)
)
);
var eventTestArray = searchResponse.Documents.Select(x => new {x.OrderID }).ToArray();
Console.WriteLine("searchResponse.Documents.Count: " + searchResponse.Documents.Count());
var i = 0;
var j = 0;
foreach (var s in eventTestArray)
{
Console.WriteLine($"{i}: " + s);
i++;
}
Assert.AreNotEqual(eventTestArray, null);
}
the output is only:
searchResponse.Documents.Count: 0.
There should be 10 orders in the search response.
I just wonder if I've misunderstood something.
The data in the database looks for example like this
{
"_index": "testconnectiones",
"_type": "logs",
"_id": "1",
"_version": 1,
"found": true,
"_source": {
"OrderId": 1,
"Event": "CreateOrder"
}
}
When using the type TestLogs, how does the client know to target testconnectiones index?
There are a few things to consider:
Ensuring that testconnectiones index is queried
Ensuring that f => f.OrderID will serialize to a string value to target a field that exists for the type and index in Elasticsearch. By default, NEST camel cases property names when serializing
Ensuring that the query has an input so that it is not considered "conditionless", or using .Verbatim() to serialize the query exactly as is
Ensuring that indexed documents are available for search
A part of the problem was now solved went by.
QueryContainer query = new TermQuery()
{
Field = "OrderId",
Value = "1"
};
var searchRequest = new SearchRequest(index: "testindex")
{
Query = query
};
var searchResult = client.Search<TestLogs>(searchRequest);
foreach (var s in orderIdArray)
{
Console.WriteLine($"{i}: OrderId:" + s.OrderID + " Event: " + s.Event + " Time: " + s.TimeStamp);
i++;
}
Now is the problem only that if i change
QueryContainer query = new TermQuery()
{
Field = "OrderId",
Value = "1"
};
to...
QueryContainer query = new TermQuery()
{
Field = "OrderId",
Value = "1"
};
It wont give any output is Test method ELK_algorithmsTests.TestIndexCreation.testToMakeSameSelectionButWithSelectedEvent threw exception:
System.IndexOutOfRangeException: ..
any suggestion where is should look? because im out of ideas right now..
Related
This bounty has ended. Answers to this question are eligible for a +50 reputation bounty. Bounty grace period ends in 9 hours.
Keppy is looking for an answer from a reputable source.
I have a Elastic Client from Elasticsearch.Net which fetching data from InMemoryConnection and add Query filter on the search but result is not filtering. Its returning entire data from responseBody as result.
Am I missing something or this is how InMemoryConnection is working?
CurrenciesDTO.cs
internal class CurrenciesDTO
{
[Keyword(Name = "CCY")]
public string CCY { get; set; }
}
Program.cs
using ConsoleApp_Elastic;
using Elasticsearch.Net;
using Nest;
using Newtonsoft.Json;
using System.Collections.Generic;
using System.Text;
using System.Threading;
List<CurrenciesDTO> listCurrencies = new List<CurrenciesDTO> { new CurrenciesDTO() { CCY = "GEL" }, new CurrenciesDTO() { CCY = "INR" }, new CurrenciesDTO() { CCY = "JPY" }, new CurrenciesDTO() { CCY = "USD" } };
var response = new
{
took = 1,
timed_out = false,
_shards = new
{
total = 1,
successful = 1,
skipped = 0,
failed = 0
},
hits = new
{
total = new
{
value = 193,
relation = "eq"
},
max_score = 1.0,
hits = Enumerable.Range(0, listCurrencies.Count).Select(i => (object)new
{
_index = "test.my.currencies",
_type = "_doc",
_id = listCurrencies[i].CCY,
_score = 1.0,
_source = new
{
CCY = listCurrencies[i].CCY,
}
})
}
};
string json = JsonConvert.SerializeObject(response);
var responseBody = Encoding.UTF8.GetBytes(json);
ConnectionSettings connectionSettings = new ConnectionSettings(new InMemoryConnection(responseBody, 200));
connectionSettings.OnRequestCompleted(apiCallDetails =>
{
if (apiCallDetails.RequestBodyInBytes != null)
{// not reaching here
Console.WriteLine(
$"{apiCallDetails.HttpMethod} {apiCallDetails.Uri} " +
$"{Encoding.UTF8.GetString(apiCallDetails.RequestBodyInBytes)}");
}
});
var client = new ElasticClient(connectionSettings);
var filterItems = new List<Func<QueryContainerDescriptor<CurrenciesDTO>, QueryContainer>>();
filterItems.Add(p => p.Term(v => v.Field(f=>f.CCY).Value("USD")));
var result = await client.SearchAsync<CurrenciesDTO>(s => s
.Index("test.my.currencies")
.Query(q => q.Bool(x => x.Filter(filterItems))), CancellationToken.None);
// .Query(q => q.Term(p => p.CCY, "USD")));
//expected 1 record but 4 records are returned.
foreach (var a in result.Documents.ToArray())
{
Console.WriteLine(a.CCY);
}
Console.ReadLine();
Yes, this is by design. InMemoryConnection was created to make unit testing easier and won't be much help with validating actual queries.
For making sure that Elasticsearch is configured the way you are expecting it to be and that queries sent to Elasticsearch are valid I would suggest using Testcontainers.
Simple test would look like:
spin up new docker instance of Elasticsearch with Testcontainers help
index some data
run you code against Elasticsearch running inside container
I am using Elasticsearch 6.6.0 and NEST in a .NET MVC project.
I am indexing some products using this code:
var esSettings = new ConnectionSettings(node);
esSettings = esSettings.DefaultIndex(IndexInstanceName);
esSettings = esSettings
.DefaultMappingFor<SearchableProduct>(s => s.IdProperty("Id").IndexName(IndexInstanceName + "-products-" + ConfigurationManager.AppSettings["DefaultCulture"]));
var elastic = new ElasticClient(esSettings);
var mapResponse = elastic.Map<SearchableProduct>(x => x.AutoMap().Index(IndexInstanceName + "-products-" + culture));
var indexState = new IndexState
{
Settings = new IndexSettings()
};
indexState.Settings.Analysis = new Analysis
{
Analyzers = new Analyzers()
};
indexState.Settings.Analysis.Analyzers.Add("nospecialchars", new CustomAnalyzer
{
Tokenizer = "standard",
Filter = new List<string> { "standard", "lowercase", "stop", "asciifolding" }
});
//products
if (!elastic.IndexExists(IndexInstanceName + "-products-" + culture).Exists)
{
var response = elastic.CreateIndex(
IndexInstanceName + "-products-" + culture,
s => s.InitializeUsing(indexState)
.Mappings(m => m.Map<SearchableProduct>(sc => sc.AutoMap())));
}
await this.IndexProductsAsync(context, products, elastic, culture);
await elastic.RefreshAsync(new RefreshRequest(IndexInstanceName + "-products-" + culture));
and for the search I use the below code:
ISearchResponse<SearchableProduct> result = await elastic.SearchAsync<SearchableProduct>(s => s
.Index(elasticIndexName + "-products-" + culture)
.Take(DefaultPageSize)
.Source(src => src.IncludeAll())
.Query(query =>
query.QueryString(qs =>
qs.Query(q).DefaultOperator(Operator.And).Fuzziness(Fuzziness.EditDistance(0)).Fields(x => x.Field(d => d.Name, 2)
.Field(d => d.MetaTitle, 1)
.Field(d => d.Image, 1)
.Field(d => d.SystemId, 2)
.Field(d => d.Manufacturer, 1)
)
))
.Sort(d => d.Ascending(SortSpecialField.Score))
);
When i search for a word with accent in greek (eg παγωτό) I get results (Because in my index the product is indexed with accent), but when i use the same word without accent (eg παγωτο) i get no results.
Is anything wrong with the indexing settings or the search code?
Can I index my data without accents or alternatively index them as is but make the search or index accent insensitive?
Creating a field with a greek analyzer will make sure indexed text and query string pass the same analysis path. For παγωτό that means, during indexing, the text will be tokenized to παγωτ as well as during making the query request.
Please check my example which creates a field with greek analyzer and the example outputs both documents with παγωτό and παγωτο when looking for παγωτό or παγωτο.
class Program
{
static async Task Main(string[] args)
{
var connectionPool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(connectionPool)
.DefaultIndex("index_name")
.DisableDirectStreaming()
.PrettyJson();
var client = new ElasticClient(settings);
await client.Indices.DeleteAsync("index_name");
var createIndexResponse = await client.Indices.CreateAsync("index_name",
c => c
.Map(map => map.AutoMap<Document>()));
await client.IndexManyAsync(new []
{new Document {Id = 1, Text = "παγωτό"}, new Document {Id = 2, Text = "παγωτο"},});
await client.Indices.RefreshAsync();
var query = "παγωτό";
var searchResponse = await client.SearchAsync<Document>(s => s
.Query(q => q.Match(m => m.Field(f => f.Text).Query(query))));
Console.OutputEncoding = Encoding.UTF8;
Print(query, searchResponse);
query = "παγωτο";
var searchResponse2 = await client.SearchAsync<Document>(s => s
.Query(q => q.Match(m => m.Field(f => f.Text).Query(query))));
Print(query, searchResponse2);
}
private static void Print(string query, ISearchResponse<Document> searchResponse)
{
Console.WriteLine($"For {query} found:");
foreach (var document in searchResponse.Documents)
{
Console.WriteLine($"Document {document.Id} {document.Text}");
}
}
}
public class Document
{
public int Id { get; set; }
[Text(Analyzer = "greek")]
public string Text { get; set; }
}
Prints:
For παγωτό found:
Document 1 παγωτό
Document 2 παγωτο
For παγωτο found:
Document 1 παγωτό
Document 2 παγωτο
Hope that helps.
I'm doing comparisons across databases (with about 20 fields) and I've defined a concrete type to handle the comparison.
If my comparison fails, I want to loop through the individual items in the catch block and provide the user a list errors. Below, I've done it manually through the first few variables. Is there a more efficient way to loop this through all 20 fields? I started with a foreach (Object objectItem .. but not sure if that's the right way to go.
Any thoughts or much needed guidance?
try {
CollectionAssert.IsSubsetOf(orgs, members, "Error Matching testlist Fields");
}
catch
{
//OrgID
var sourceOrgID = orgs.Select(o => o.OrgID);
var destOrgID = members.Select(o => o.OrgID);
var errorList1 = sourceOrgID.Except(destOrgID);
string failedTests = null;
failedTests = string.Join("\n", errorList1);
Assert.IsTrue(0 == failedTests.Length, "The following Org IDs are not contained in the source: \n" + failedTests);
//DealerCode
var sourceDealerCode = orgs.Select(o => o.DealerCode);
var destDealerCode = members.Select(o => o.DealerCode);
var errorList2 = sourceDealerCode.Except(destDealerCode);
failedTests = null;
failedTests = string.Join("\n", errorList2);
Assert.IsTrue(0 == failedTests.Length, "The following Dealer Codes are not contained in the source: \n" + failedTests);
//orgkey
var sourceOrgKey = orgs.Select(o => o.OrgKey);
var destOrgKey = members.Select(o => o.OrgKey);
var errorList3 = sourceOrgKey.Except(destOrgKey);
failedTests = null;
failedTests = string.Join("\n", errorList3);
Assert.IsTrue(0 == failedTests.Length, "The following Org Keys are not contained in the source: \n" + failedTests);
You need reflection to do this:
Type type = obj.GetType();
PropertyInfo[] properties = type.GetProperties();
foreach (PropertyInfo property in properties)
{
GetColumn(orgList,property.Name);
}
var names = items.Select(x => x.GetType().GetProperty("prpname").GetValue(x));
public IEnumerable<object> GetColumn(List<Item> items, string columnName)
{
var values = items.Select(x =>
x.GetType().GetProperty(columnName).GetValue(x));//u can put your test code heere and you can loop it through object properties
}
you can create a list to store rslt and add results to the list else you can write output in logs files
If I haven't misunderstood your question, you could do something like this using shouldly
[TestMethod]
public void UnitTestExample()
{
var orgs = new Organisation
{
Ids = new List<int>{ 1,3 },
DealerCodes = new List<string> { "foo","bar"}
};
var members = new Organisation
{
Ids = new List<int> { 1,2,3 },
DealerCodes = new List<string> { "foo", "bar", "buzz" }
};
orgs.ShouldSatisfyAllConditions(
() => orgs.Ids.ShouldBe(members.Ids, ignoreOrder: true),
() => orgs.DealerCodes.ShouldBe(members.DealerCodes, ignoreOrder: true)
);
}
The output produced is:
You would still have to specify every property you want to check inside ShouldSatisfyAllConditions but shouldly does all the heavy lifting around comparing the lists and output the differences.
I'm assume that:
orgs is a IQueryable<T1>
members is a IQueryable<T2>
typeof(T1) == typeof(T2)
If so, that function could help:
private static void CompareRecords<TEntity>(IQueryable<TEntity> orgs, IQueryable<TEntity> members)
{
var entityType = typeof (TEntity);
var selectMethod = typeof (Queryable).GetMethods().First(x => x.Name == "Select");
var exceptMethod = typeof(Queryable).GetMethods().First(x => x.Name == "Except");
var toArrayMethod = typeof (Enumerable).GetMethods().First(x => x.Name == "ToArray");
foreach (var property in entityType.GetProperties())
{
var paramExpr = Expression.Parameter(entityType, "x");
var propExpr = Expression.Property(paramExpr, property.Name);
var delegateType = typeof (Func<,>).MakeGenericType(entityType, property.PropertyType);
var lambdaExpr = Expression.Lambda(delegateType, propExpr, paramExpr);
var parameterizedSelectMethod = selectMethod.MakeGenericMethod(entityType, property.PropertyType);
var parameterizedExceptMethod = exceptMethod.MakeGenericMethod(property.PropertyType);
var source = parameterizedSelectMethod.Invoke(null, new object[] {orgs, lambdaExpr});
var dest = parameterizedSelectMethod.Invoke(null, new object[] {members, lambdaExpr});
var errorList = parameterizedExceptMethod.Invoke(null, new[] {source, dest});
var errorListArray = toArrayMethod.MakeGenericMethod(property.PropertyType).Invoke(null, new[] {errorList});
var failedTests = string.Join("\n", ((IEnumerable)errorListArray).Cast<object>().Select(x => x.ToString()));
Assert.IsTrue(0 == failedTests.Length, $"The following {property.Name} are not contained in the source: \n{failedTests}");
}
}
i have many objects like this:
var obj4 = new Data { name = "person", date = DateTime.Now.AddDays(1), data = "pr-214-2-20151224-word2-word3" };
i want to write a linq query that result be this:
result=[name=person, date=04/09/2016 12:00:00 AM, data=[2,3]]//2 and 3 are the numbers after word
so far i tried this:
var listak=new List<Data>{obj,obj2,obj3,obj4};
var u = listak.OrderByDescending(s => s.date).TakeWhile(s => s.date > DateTime.Now).Select(s=>new
{
name=s.name,
date=s.date,
data=s.data.Split(new []{"-"},StringSplitOptions.None).Select(m=>new
{
word = m.Where(c=>m.StartsWith("word")).Select(c=>m.Remove(0,4))//this line is incorrect, i dont know how to correct this part
})
});
but i cant get what i want, how can i achieve that?
You can try this :
var obj4 = new { name = "person", date = DateTime.Now.AddDays(1), data = "pr-214-2-20151224-word2-word3" };
var listak = new[] { obj4 }.ToList();
var u = listak.OrderByDescending(s => s.date).TakeWhile(s => s.date > DateTime.Now).Select(s => new
{
name = s.name,
date = s.date,
data = s.data.Split(new[] { "-" }, StringSplitOptions.None).Where(c => c.StartsWith("word")).Select(m => m.Remove(0, 4))
});
Console.Write(u);
Can somebody help me to get the custom JSON string from datatable using C#.
I need something like below.I am able to achive this by using for each loop.
[
{"message": {"alert": "Address Updated"},"target": {"userIds": ["BKAC7759"]}},
{"message": {"alert": "Payment Processed"},"target": {"userIds": ["BKAC7759"]}},
{"message": {"alert": "Notice Sent"},"target": {"userIds": ["BKAC7759"]}}
]
But is there any way i can make it in simple way. My datatable contains values for "alert" and "userIds"
private string GetJsonData(int numberofRecords)
{
// OleDbConnection conn = new OleDbConnection(connectionString);
try
{
DataTable Test = new DataTable("A");
Test.Columns.Add("alert");
Test.Columns.Add("userIds");
Test.Rows.Add("Address Updated", "BKAC7759");
Test.Rows.Add("Payment Made", "BKAC7759");
//Test.Rows.Add("Check Processed", "MAND1884");
//Test.Rows.Add("Notice Mailed", "JAID3869");
//Test.Rows.Add("DL Suspended", "AOQU4798");
string jo = string.Empty;
string com = ",";
int i = 0;
int count = Test.Rows.Count;
string bracketright = "]";
string bracketleft = "[";
foreach (DataRow row in Test.Rows)
{
if (i == 0)
{
jo = jo + bracketleft;
}
jo = jo + "{\"message\":{\"alert\":\"" + row[0].ToString() + "\"},\"target\":{\"userIds\":[\"" + row[1].ToString() + "\"]}}";
if (i != count - 1)
{
jo = jo + com;
}
else
{
jo = jo + bracketright;
}
i++;
}
return jo;
}
catch (Exception ex)
{
Logger.Error("GetJsonData(int numberofRecords): " + ex.Message);
return string.Empty;
}
finally
{
// always close the connection.
// conn.Close();
}
}
You can use Linq + DataTableExtensions (in namespace System.Data and system DLL System.Data.DataSetExtensions.dll) to transform your table into an enumerable of anonymous types, then serialize that to JSON with json.net.
I notice your "userIds" property is a JSON array. Do you want all the user userIds for a given alert to be combined? If so, you can use ToLookup to combine them:
var root = dataTable.AsEnumerable()
.ToLookup(r => r["alert"].ToString(), r => r["userIds"].ToString())
.Select(g => new { message = new { alert = g.Key }, target = new { userIds = g } });
var json = JsonConvert.SerializeObject(root);
If not, do:
var root = dataTable.AsEnumerable()
.Select(r => new { message = new { alert = r["alert"].ToString() }, target = new { userIds = new [] { r["userIds"].ToString() } } });
var json = JsonConvert.SerializeObject(root);
For the following table:
var dataTable = new DataTable("A");
dataTable.Columns.Add("alert");
dataTable.Columns.Add("userIds");
dataTable.Rows.Add("Address Updated", "BKAC7759");
dataTable.Rows.Add("Payment Made", "BKAC7759");
dataTable.Rows.Add("Address Updated", "MAND1884");
dataTable.Rows.Add("Payment Made", "MAND1884");
The first produces the following JSON:
[
{"message":{"alert":"Address Updated"},"target":{"userIds":["BKAC7759","MAND1884"]}},
{"message":{"alert":"Payment Made"},"target":{"userIds":["BKAC7759","MAND1884"]}}
]
And the second produces the following:
[
{"message":{"alert":"Address Updated"},"target":{"userIds":["BKAC7759"]}},
{"message":{"alert":"Payment Made"},"target":{"userIds":["BKAC7759"]}},
{"message":{"alert":"Address Updated"},"target":{"userIds":["MAND1884"]}},
{"message":{"alert":"Payment Made"},"target":{"userIds":["MAND1884"]}}
]