Develop Reference

How to make Filter Aggregations using NEST?

I have a requirement for filter aggregations using NEST. But since I don't know much about this, I have made the below:
class Program
{
static void Main(string[] args)
{
ISearchResponse<TestReportModel> searchResponse =
ConnectionToES.EsClient()
.Search<TestReportModel>
(s => s
.Index("feedbackdata")
.From(0)
.Size(50000)
.Query(q =>q.MatchAll())
);
var testRecords = searchResponse.Documents.ToList<TestReportModel>();
result = ComputeTrailGap(testRecords);
}
private static List<TestModel> ComputeTrailGap(List<TestReportModel> testRecords)
{
var objTestModel = new List<TestModel>();
var ptpDispositionCodes = new string[] { "PTP" };
var bptpDispositionCodes = new string[] { "BPTP","SBPTP" };
int gapResult = testRecords.Where(w => w.trailstatus == "Gap").Count();
var ptpResult = testRecords.Where(w => ptpDispositionCodes.Contains(w.lastdispositioncode)).ToList().Count();
var bptpResult = testRecords.Where(w => bptpDispositionCodes.Contains(w.lastdispositioncode)).ToList().Count();
objTestModel.Add(new TestModel { TrailStatus = "Gap", NoOfAccounts = gapResult });
objTestModel.Add(new TestModel { TrailStatus = "PTP", NoOfAccounts = ptpResult });
objTestModel.Add(new TestModel { TrailStatus = "BPTP", NoOfAccounts = bptpResult });
return objTestModel;
}
}
DTO
public class TestReportModel
{
public string trailstatus { get; set; }
public string lastdispositioncode { get; set; }
}
public class TestOutputAPIModel
{
public List<TestModel> TestModelDetail { get; set; }
}
public class TestModel
{
public string TrailStatus { get; set; }
public int NoOfAccounts { get; set; }
}
This program works but as can be figure out that we are only accessing the Elastic Search via NEST and the rest of the Aggregations /Filter are done using Lambda.
I would like to perform the entire operation (Aggregations /Filter etc) using NEST framework and put it in the TestModel using filter aggregation.
How can I construct the DSL query inside NEST?
Update
I have been able to make the below so far but the count is zero. What is wrong in my query construction?
var ptpDispositionCodes = new TermsQuery
{
IsVerbatim = true,
Field = "lastdispositioncode",
Terms = new string[] { "PTP" },
};
var bptpDispositionCodes = new TermsQuery
{
IsVerbatim = true,
Field = "lastdispositioncode",
Terms = new string[] { "BPTP" },
};
ISearchResponse<TestReportModel> searchResponse =
ConnectionToES.EsClient()
.Search<TestReportModel>
(s => s
.Index("feedbackdata")
.From(0)
.Size(50000)
.Query(q =>q.MatchAll())
.Aggregations(fa => fa
.Filter("ptp_aggs", f => f.Filter(fd => ptpDispositionCodes))
.Filter("bptp_aggs", f => f.Filter(fd => bptpDispositionCodes))
)
);
Result

I see that you are trying to perform a search on the type of TestReportModel. The overall structure of your approach seems good enough. However, there is a trouble with the queries that are being attached to your filter containers.
Your TestReportModel contains two properties trialStatus and lastdispositioncode. You are setting the Field property as description inside your terms query. This is the reason that you are seeing the counts as zero. The model on which you are performing the search on (in turn the index that you are performing the search on) does not have a property description and hence the difference. NEST or Elasticsearch, in this case does not throw any exception. Instead, it returns count zero. Field value should be modified to lastdispositioncode.
// Type on which the search is being performed.
// Response is of the type ISearchResponse<TestReportModel>
public class TestReportModel
{
public string trailstatus { get; set; }
public string lastdispositioncode { get; set; }
}
Modified terms queries are as follows
// Field is "lastdispositioncode" and not "description"
// You may amend the Terms field as applicable
var ptpDispositionCodes = new TermsQuery
{
IsVerbatim = true,
Field = "lastdispositioncode",
Terms = new string[] { "PTP" },
};
var bptpDispositionCodes = new TermsQuery
{
IsVerbatim = true,
Field = "lastdispositioncode",
Terms = new string[] { "BPTP", "SBPTP" },
};
Since it seems that the values to lastdispositioncode seem to take a single word value (PTP or BPTP from your examples), I believe, it is not going to matter if the field in the doc is analyzed or not. You can further obtain the counts from the ISearchResponse<T> type as shown below
var ptpDocCount = ((Nest.SingleBucketAggregate)response.Aggregations["ptp_aggs"]).DocCount;
var bptpDocCount = ((Nest.SingleBucketAggregate)response.Aggregations["bptp_aggs"]).DocCount;
Edit: Adding an approach for keyword search
QueryContainer qc1 = new QueryContainerDescriptor<TestReportModel>()
.Bool(b => b.Must(m => m.Terms(t => t.Field(f => f.lastdispositioncode.Suffix("keyword"))
.Terms(new string[]{"ptp"}))));
QueryContainer qc2 = new QueryContainerDescriptor<TestReportModel>()
.Bool(b => b.Must(m => m.Terms(t => t.Field(f => f.lastdispositioncode.Suffix("keyword"))
.Terms(new string[]{"bptp", "sbptp"}))));
Now these query containers can be hooked to your aggregation as shown below
.Aggregations(aggs => aggs
.Filter("f1", f => f.Filter(f => qc1))
.Filter("f2", f => f.Filter(f => qc2)))
The queries for aggregations that are generated by the NEST client in this case look like below
"f1": {
"filter": {
"bool": {
"must": [
{
"terms": {
"lastdispositioncode.keyword": [
"bptp"
]
}
}
]
}
}
}
Also, coming back to the case of search being case-insensitive, Elasticsearch deals with search in a case-insensitive fashion. However, it varies depending on analyzed vs non-analyzed fields. Analyzed fields are tokenized and text fields are by default tokenized. We use the suffix extension method of NEST on analyzed fields ideally and to get an exact match on the analyzed field. More about them here

Testing FluentValidation ChildRules

Given the following object:
public class PatchDTO
{
public PatchDTO()
{
Data = new List<Datum>();
}
public List<Datum> Data { get; set; }
public class Datum
{
public Datum()
{
Attributes = new Dictionary<string, object>();
}
public string Id { get; set; }
public Dictionary<string, object> Attributes { get; set; }
}
}
I have my validator set as follows:
RuleFor(oo => oo.Data)
.NotEmpty()
.WithMessage("One or more Data blocks must be provided");
RuleForEach(d => d.Data).ChildRules(datum =>
{
datum.RuleFor(d => d.Id)
.NotEmpty()
.WithMessage("Invalid 'Data.Id' value");
});
Which I'm trying to test using the test extensions as such:
[Theory]
[InlineData(null)]
[InlineData("")]
public void Id_Failure(string id)
{
dto.Data[0].Id = id;
var result = validator.TestValidate(dto);
result.ShouldHaveValidationErrorFor(oo => oo.Data[0].Id)
.WithErrorMessage("Invalid 'Data.Id' value");
}
But when I run the test it says:
FluentValidation.TestHelper.ValidationTestException
HResult=0x80131500
Message=Expected a validation error for property Id
----
Properties with Validation Errors:
[0]: Data[0].Id
But as you can see under the 'Validation Errors', it has actually picked up in the validation failure but isn't tying it to this test. So how do I test these ChildRules or tell the test extension method which property it should actually be checking?
(I also used the validator.ShouldHaveValidationErrorFor directly with the same results)

I've had this problem before and resorted to using the string overload for ShouldHaveValidationErrorFor
The following (nunit) test passes
[TestCase(null)]
[TestCase("")]
public void Id_InvalidValue_HasError(string id)
{
var fixture = new Fixture();
var datum = fixture.Build<PatchDTO.Datum>().With(x => x.Id, id).Create();
var dto = fixture.Build<PatchDTO>().With(x => x.Data, new List<PatchDTO.Datum> { datum }).Create();
var validator = new PatchDTOValidator();
var validationResult = validator.TestValidate(dto);
validationResult.ShouldHaveValidationErrorFor("Data[0].Id")
.WithErrorMessage("Invalid 'Data.Id' value");
}
It's been a while since I looked at it, but I believe the issue is in the extension ShouldHaveValidationErrorFor matching on property name and the property expression overload doesn't resolve the property name to 'Data[0].Id'. If you inspect the validation results you'll get a ValidationError object that looks something like this
{
"PropertyName":"Data[0].Id",
"ErrorMessage":"Invalid 'Data.Id' value",
"AttemptedValue":"",
"CustomState":null,
"Severity":0,
"ErrorCode":"NotEmptyValidator",
"FormattedMessageArguments":[
],
"FormattedMessagePlaceholderValues":{
"PropertyName":"Id",
"PropertyValue":""
},
"ResourceName":null
}
EDIT:
Had a quick peek into the property expression overload, as per below
public IEnumerable<ValidationFailure> ShouldHaveValidationErrorFor<TProperty>(Expression<Func<T, TProperty>> memberAccessor)
{
return ValidationTestExtension.ShouldHaveValidationError(this.Errors, ValidatorOptions.PropertyNameResolver(typeof (T), memberAccessor.GetMember<T, TProperty>(), (LambdaExpression) memberAccessor), true);
}
Presumably you could use another/write your own property name resolver to handle the case as it is settable. You'd probably have to dig into the expression to do it.

How to wrap bulk upsert in Mongo with Task in C#

I have some model
public partial class Option
{
[BsonId]
public int Id { get; set; }
public Quote Quote { get; set; }
public Contract Contract { get; set; }
}
Method that does bulk write
public Task SaveOptions(List<Option> contracts)
{
var context = new MongoContext();
var processes = new List<Task<UpdateResult>>();
var collection = context.Storage.GetCollection<Option>("options");
contracts.ForEach(contract =>
{
var item = Builders<Option>.Update.Set(o => o, contract);
var options = new UpdateOptions { IsUpsert = true };
processes.Add(collection.UpdateOneAsync(o => o.Id == contract.Id, item, options));
});
return Task.WhenAll(processes);
}
Call for the method above
Task.WhenAll(service.SaveOptions(contracts)) // also tried without Task.WhenAll
For some reason, it doesn't create any record in Mongo DB
Update
Tried to rewrite bulk write this way, still no changes.
public Task SaveOptions(List<Option> contracts)
{
var context = new MongoContext();
var records = new List<UpdateOneModel<Option>>();
var collection = context.Storage.GetCollection<Option>("options");
contracts.ForEach(contract =>
{
var record = new UpdateOneModel<Option>(
Builders<Option>.Filter.Where(o => o.Id == contract.Id),
Builders<Option>.Update.Set(o => o, contract))
{
IsUpsert = true
};
records.Add(record);
});
return collection.BulkWriteAsync(records);
}

I think you want to use ReplaceOneAsync:
processes.Add(collection.ReplaceOneAsync(o => o.Id == contract.Id, contract, options));
The problem with using UpdateOneAsync here is that you're supposed to specify a field to update, and o => o doesn't do that. Since you want to replace the entire object, you need to use ReplaceOneAsync.
** Note that you can also do this with BulkWriteAsync by creating a ReplaceOneModel instead of an UpdateOneModel.

Unit Testing issue with Moq and Include (EF6)

I have done fair bit research and tried all sorts of different ways of getting the Test to pass but now i need some help.
I am trying to test the following repository method:
public class PortalsRepository : BaseRepository<PortalDomainRole>, IPortalsRepository
{
public PortalsRepository(IAuthDbContext context) : base(context)
{
}
public IEnumerable<PortalRole> GetRoles(string domainName)
{
return Set.Include(x => x.PortalDomain)
.Include(x => x.PortalRole)
.Where(x => x.PortalDomain.Name.ToLower() == domainName)
.Select(x => x.PortalRole)
.ToList();
}
}
The Context looks like:
public interface IAuthDbContext : IDbContextBase
{
}
public interface IDbContextBase
{
IDbSet<T> Set<T>() where T : class;
IEnumerable<DbValidationError> GetEntityValidationErrors();
int SaveChanges();
Task<int> SaveChangesAsync();
Task<int> SaveChangesAsync(CancellationToken cancellationToken);
}
My Unit Test Set-up Looks like:
protected override void GivenThat()
{
var mockRolesSet = GetMockDbSet(PortalRoles().AsQueryable());
mockRolesSet.Setup(x => x.Include("PortalRole")).Returns(mockRolesSet.Object);
var mockDomainsSet = GetMockDbSet(PortalDomains().AsQueryable());
mockDomainsSet.Setup(x => x.Include("PortalDomain")).Returns(mockDomainsSet.Object);
var mockPortalDomanRolesSet = GetMockDbSet(PortalDomainRoles().AsQueryable());
mockPortalDomanRolesSet.Setup(x => x.Include("PortalRole")).Returns(mockPortalDomanRolesSet.Object);
mockPortalDomanRolesSet.Setup(x => x.Include("PortalDomain")).Returns(mockPortalDomanRolesSet.Object);
var customDbContextMock = new Mock<IAuthDbContext>();
customDbContextMock.Setup(x => x.Set<PortalRole>()).Returns(mockRolesSet.Object);
customDbContextMock.Setup(x => x.Set<PortalDomain>()).Returns(mockDomainsSet.Object);
customDbContextMock.Setup(x => x.Set<PortalDomainRole>()).Returns(mockPortalDomanRolesSet.Object);
ClassUnderTest = new PortalsRepository(customDbContextMock.Object);
}
My Unit Test Supporting Methods:
public List<PortalDomainRole> PortalDomainRoles()
{
var data = new List<PortalDomainRole>
{
new PortalDomainRole { PortalRoleId = 2, PortalDomainId = 1},
new PortalDomainRole { PortalRoleId = 1, PortalDomainId = 2},
new PortalDomainRole { PortalRoleId = 2, PortalDomainId = 2}
};
return data;
}
public List<PortalDomain> PortalDomains()
{
var data = new List<PortalDomain>
{
new PortalDomain { Name = "google.co.uk", PortalDomainId = 1 },
new PortalDomain { Name = "bbc.com", PortalDomainId = 2 }
};
return data;
}
public List<PortalRole> PortalRoles()
{
var data = new List<PortalRole>
{
new PortalRole {Name = "New Products", PortalRoleId = 1},
new PortalRole {Name = "Classic Products", PortalRoleId = 2}
};
return data;
}
When the unit test executes the method in question, i get:
System.NullReferenceException : Object reference not set to an instance of an object.
Most likely it does not know how to handle the nested include statements - i have followed many online questions and tutorials and now i am stuck.

My answer is probably a bit controversial, but in my experience, the best way to test your repository layer (or whatever you call the code that does the actual data access), is by having it actually call the database during testing.
When you are writing your unit test, you are assuming that Entity Framework works in a specific way. But sometimes it works in different ways than you expect, and thus a test may pass, even though the code doesn't work correctly.
Take this example, that illustrates the problem (the last EF version I worked with was version 4, but I assume that my statement is still true for EF6)
public class Foo {
public DateTime StartDate { get; set; }
public DateTime EndDate { get; set; }
public bool Active {
get { return StartDate < DateTime.Now && EndDate > DateTime.Now }
}
}
public class FooRepository {
public IEnumerable<Foo> ActiveFoos { get { return DataContext.Foos.Where(x => x.Active) } }
}
Testing this FooRepository against a mocked data access will pass, but executing against a real database will throw an exception. That is because EF will try to create an SQL clause for the Where(x => x.Active), but because Active is not a field in the database, EF will have no idea how to translate the query to SQL.
So your unit test provides a false positive. Executing the tests against the database will make it fail, as it should.

Why are no query parameters being passed to my NancyFX module?

I am running a self-hosted NancyFX web server inside of my application. Right now I have one module hosted:
public class MetricsModule : NancyModule
{
private IStorageEngine _storageEngine;
public MetricsModule(IStorageEngine storageEngine) : base("/metrics")
{
_storageEngine = storageEngine;
Get["/list"] = parameters =>
{
var metrics = _storageEngine.GetKnownMetrics();
return Response.AsJson(metrics.ToArray());
};
Get["/query"] = parameters =>
{
var rawStart = parameters.start;
var rawEnd = parameters.end;
var metrics = parameters.metrics;
return Response.AsJson(0);
};
}
}
My Bootstrapper class is:
public class OverlookBootStrapper : DefaultNancyBootstrapper
{
private readonly IStorageEngine _storageEngine;
public OverlookBootStrapper(IStorageEngine storageEngine)
{
_storageEngine = storageEngine;
}
protected override void ConfigureApplicationContainer(TinyIoCContainer container)
{
container.Register(_storageEngine);
}
}
I am trying to test it with the following test:
[TestInitialize]
public void Init()
{
_storageEngine = new Mock<IStorageEngine>();
var bootstrapper = new OverlookBootStrapper(_storageEngine.Object);
_browser = new Browser(bootstrapper);
}
[TestMethod]
public void Query_Builds_Correct_Query_From_Parameters()
{
var metric = new Metric("device", "category", "name", "suffix");
var startDate = DateTime.Now;
var endDate = DateTime.Now.AddMinutes(10);
var path = "/metrics/query";
var response = _browser.Get(path, with =>
{
with.HttpRequest();
with.Query("start", startDate.ToString());
with.Query("end", endDate.ToString());
with.Query("metrics", metric.ToParsableString());
});
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode, "Incorrect status code returned");
_storageEngine.Verify(x => x.ExecuteQuery(It.Is<Query>(y => y.StartDate == startDate)), Times.Once());
_storageEngine.Verify(x => x.ExecuteQuery(It.Is<Query>(y => y.EndDate == endDate)), Times.Once());
_storageEngine.Verify(x => x.ExecuteQuery(It.Is<Query>(y => y.Metrics.Contains(metric))), Times.Once());
}
When this test is debugged and a breakpoint is put on return Response.AsJson(0);, I inspected the parameters object and noticed that parameters.Count is zero, and all 3 values are null.
What am I doing incorrectly?
Edit: When I bring up this endpoint in the web browser, the same issue occurs. I get a result of 0 sent back to my browser, but when debugging I see that no query string parameters I specify have been recognized by NancyFX.

The parameters argument to your lambda contains the route parameters you captured in the in your Get["/query"]. In this case nothing. See #thecodejunkie's comment for an example where there is something.
To get to the query paramters use Request.Query. That's also a dynamic and will contain whatever query parameters was in the request. Like so:
Get["/query"] = parameters =>
{
var rawStart = Request.Query.start;
var rawEnd = Request.Query.end;
var metrics = Request.Query.metrics;
return Response.AsJson(0);
};
This should work with your tests too.

You can let NancyFx's model binding take care of the url query string.
public class RequestObject
{
public string Start { get; set; }
public string End { get; set; }
public string Metrics { get; set; }
}
/query?start=2015-09-27&end=2015-10-27&metrics=loadtime
Get["/query"] = x =>
{
var request = this.Bind<RequestObject>();
}