consider the following scenario:
public class DBEntry {
public string Id;
}
public class ComputedEntry {
public string Id;
public int ComputedIndex;
}
IQueryable<DBEntry> databaseQueryable; // Somewhere hidden behind the API
IQueryable<ComputedEntry> entryQueryable; // Usable with the API
Let's assume each DBEntry has a unique Id and not much else. A ComputedEntry has a 1:n relationship with DBEntry, meaning that a DBEntrycan be expanded into more than a single ComputedEntryduring execution of the query.
Now, I am trying to query entryQueryable to get a range of computed indices, e.g:
entryQueryable.Where(dto => dto.ComputedIndex < 10 && dto.Id == "some-id");
What I'm looking for is a way of separating the given query expression to only push down the relevant parts of the query to the databaseQueryable. In the example above something like this should happen (probably in the implementation of IQueryableProvider.Execute when using the entryQueryable):
var results = databaseQueryable.Where(e => e.Id == "some-id").ToList();
int i = 0;
return results.Select(e => new ComputedEntry(e.Id, i++));
So basically I'd like the query to be separated and the relevant/compatible parts should be pushed down to the databaseQueryable.
The obvious question would be: How should I approach this? I tried to figure out a way of separating the expression with an ExpressionVisitor, but haven't been very successful here and it seems like this is a rather complex task.
Any ideas? Maybe there is an already existing method of optimizing/translating the query I am not aware of? I have looked through the documentation but couldn't find anything useful here.
Many thanks for your suggestions!
Related
I'm trying to build a RESTful API using CQRS (with MediatR) in ASP.NET Core 6.
The issue I'm facing is when trying to expose child entities.
Consider a ItemProperty to have many ItemPropertyOptions.
The query I'm trying to expose is the following:
public record GetItemPropertyOptionsQuery(string PropertyReference, int PageNumber = 1, int PageSize= 10) : IRequest<PaginatedList<ItemPropertyOptionDto>>;
So in the controller my initial attempt was the following:
[HttpGet]
[Route("ItemProperties/{propertyReference}/Options")]
public async Task<ActionResult<PaginatedList<ItemPropertyOptionDto>>> GetAsync([FromQuery]GetItemPropertyOptionsQuery query, string propertyReference)
{
if (query.PropertyReference != propertyReference)
{
return BadRequest("You are querying for a different object.");
}
return Ok(await Mediator.Send(query));
}
As you can see, the main issue is that I need the property reference to process my query, but that is also part of the URL, so I have them duplicated. This means that the consumer of the API has to include them twice and I have to check for them to be equal.
The resulting call is something like:
/ItemProperties/test/Options?PropertyReference=test
This is a smell to me but I'm not sure how to solve it.
A first idea was to not expose directly the query but a different object that does not include this parameter and the create the Query myself, but I would need to create an extra record and match it to the query/command fields for every endpoint I create, and that doesn't sound very clean from a maintainability PoV since every time I change the Query to include a new parameter I need to do the same in the Request.
Or do you think it would be ok this way?
public record GetItemPropertyOptionsRequest(int PageNumber = 1, int PageSize= 10);
[HttpGet]
[Route("ItemProperties/{propertyReference}/Options")]
public async Task<ActionResult<PaginatedList<ItemPropertyOptionDto>>> GetAsync([FromQuery]GetItemPropertyOptionsRequest request, string propertyReference)
{
var query = new GetItemPropertyOptionsQuery(propertyReference, request.PageNumber, request.PageSize);
return Ok(await Mediator.Send(query));
}
What are you trying to achieve with this, return a list of
ItemPropertyOptions for a given ItemProperty?
#Peppermintology yes
In which case you do not require the additional (and superfluous second) propertyReferece parameter.
All you require is the first propertyReference parameter. That should operate as the lookup key for the relevant ItemProperty entity in your database and then all you need to do is retrieve the releated ItemPropertyOptions as a relationship.
So your URL would look as follows:
/item-properties/{propertyId}/options
Then you would use the propertyId parameter in your query handler to obtain your records and relation opptions. That might look something like:
var results = await _dbContext.PropertyItems
.Where(item => item.Id == propertyId)
.Include(item => item.Options)
.ToListAsync();
I tried my best to explain in the title, however I am trying to achieve giving linq statements an 'alias' and still use them in dot notation. Allow me to explain further.
below we have a list that has a linq statement applied:
private List<string> _matches;
var output = _matches.Where(x => x.EntityScore == 100).ToList();
I agree that this is simple to read. However I wish to simplify it further especially when the statements start to get bigger. This is an example of linq getting longer than I care for:
private List<string> _matches;
var matchAddressList = _matches.Where(x => x.EntityDetails.Addresses.Any(x => x.Street.Equals(inputObject.Address)
&& x.StateProvinceDistrict.Equals(inputObject.State)
&& x.City.Equals(inputObject.City))).ToList();
What I am trying to do is alias certain groups of LINQ and then call that linq as a dot operator
for example:
var finalOutput = _matches.perfectMatches().addressMatches(inputObject).someOtherMatchCondition(inputObject)
I think the above line is clear and easily readable. Future devs dont necessarily have to look into the logic. They can read the business domain name and understand what it does.
I want to avoid the following line, as I believe the previous code is more clean:
var finalOutput = someOtherMatchCondition(addressMatches(perfectMatches(_matches)));
the previous line is how I feel you would go about it using functions at a basic level. However I am struggling to find a way to create an alias or encapsulate the linq logic into a business domain name and then use that as a dot operator.
I have tried expression body definitions:
public List<string> perfectMatches => _matches.Where(x => x.EntityScore == 100).ToList();
is this going to require extensions of another class? or the writing of generics? or am I perhaps unaware of a standard way of doing this?
Update: maybe this is helpfull too:
How to add custom methods for LINQ queries (C#)
It has to be an extension method to make use of the dot notation.
Do you mean something like that. It is rather pseudo code than working. You may have to play around with the types or try out some kind of generic approach:
public class ProductionCode
{
public void MyMain()
{
var myList = new List<EntityThingType>() { .... };
var newList = myList.PerfectMatches().AddressMatches(myInputObject).ToList();
}
}
public static class test
{
public static IEnumerable<EntityThingType> PerfectMatches(this IEnumerable<EntityThingType> myList)
{
return myList.Where(x => x.EntityScore == 100);
}
public static IEnumerable<EntityThingType> AddressMatches(this IEnumerable<EntityThingType> myList, MyObjectType inputObject)
{
return myList.Where(x => x.EntityDetails.Addresses.Any(x => x.Street.Equals(inputObject.Address)
&& x.StateProvinceDistrict.Equals(inputObject.State)
&& x.City.Equals(inputObject.City)));
}
}
I think what you are looking for is Extension Methods. You can have the perfectMatches() method be an extension method that takes an IEnumerable<string> and return the same. Then you can chain those together.
First, let me say that I've researched this problem and read the following stack overflow articles, but none of them really address this situation.
How can I use Linq to join between objects and entities?
inner join in linq to entities
Situation
I have two classes
public class Section{
public string SchoolId{get;set;}
public string CourseId{get;set;}
public string SectionId{get;set;}
}
public class RelatedItem{
public string SchoolId{get;set;}
public string CourseId{get;set;}
public string SectionId{get;set;}
//..
}
I have an array of Section coming from one source and is an actual collection of Objects.
RelatedItem I'm getting via a LINQ to Entities call against a DbContext.
My goal is to get all of the RelatedItems based on the Sections I have from the other source.
I'm writing a query like this
Section[] mySections = GetSections(); //Third Party Source
IQueryable<RelatedItem> relatedItems = DbContext.RelatedItems
.Where(r=>
mySections.Any(s=> s.SchoolId == r.SchoolId &&
s.CourseId == r.CourseId &&
s.SectionId == r.SectionId)
);
Problem
At runtime, I receive the following error
Unable to create a constant value of type
'ProjectNamespace.Section'. Only primitive types or
enumeration types are supported in this context.
I found a work around, but it involves doing the following, but it doesn't take advantage of any of my table indexes.
var sectionIds = sections.Select(s=>string.Concat(s.SchoolId, "|",s.CourseId, "|",s.SectionId));
IQueryable<RelatedItem> relatedItems = DbContext.RelatedItems
.Where(r=>
sectionIds.Contains(string.Concat(r.SchoolId, "|",r.CourseId, "|",r.SectionId))
);
This block of code works, and currently is pretty fast (but this is dev, and my record count is small). Aside from converting my related items to a collection in memory, does anyone have any other suggestions?
Try using Contains instead:
Section[] mySections = GetSections(); //Third Party Source
IQueryable<RelatedItem> relatedItems = DbContext.RelatedItems.Where(r=>
mySections.Select(s => s.SchoolId).Contains(r.SchoolId) &&
mySections.Select(s => s.CourseId).Contains(r.CourseId) &&
mySections.Select(s => s.SectionId).Contains(r.SectionId)
);
Contains should translate to WHERE IN clauses in SQL.
This won't work if using .NET 3.5 and LINQ to Entities, as it wasn't implemented in that version.
Proper way to solve this is to implement IEquitable. Here is an example on how to do it Does LINQ to Entities support IEquatable in a 'where' clause predicate?
One tip when implementing Equals() and GetHashCode() do not call any .NET methods (like getType()) only compare primitives SchoolId, CourseId, SectionId, it should get converted to expression tree and work just fine.
This seems like it should be straightforward, but I can't figure out how to make it work. Imagine the following models:
public class ModelA
{
public string Code {get;set;}
}
public class ModelB
{
public string Code {get;set;}
}
I am reading a set of ModelA out of a database, and I also need to grab ModelB based on the Code. The kicker is I need to match only a part of the code.
For example, ModelA the code might be "1234.00" and ModelB the code might be "001234.20" -- I want to link the two models based on both codes containing the 1234 part. I would prefer it to be a join, but I don't see how that's possible, so iterating through the first data set and matching them is fine too, but I still can't make that work.
I created a string extension method called TruncateCode, which will give me the part that I want, but LINQ doesn't seem to support calling extensions within a comparison or join (throws an exception complaining the method is unrecognized and can't be used). This is basically what I was attempting:
var query = a in db.ModelASet
(where clause)
select a;
foreach(ModelA item in query)
{
var query2 = b in db.ModelBSet
where b.Code.TruncatedCode() == item.Code.TruncatedCode() // this doesn't work :(
select b;
}
Dear reader, how would you approach this?
(INB4 "Why is your data laid out like this?" - This is a legacy database from over 10 years ago man, I'm just trying to get this to work.)
You need to materialize the results using ToList() or something similar in order to use your extension method.
var list = (a in db.ModelASet
select a).ToList();
foreach(var item in list)
{
var query2 = b in list
where b.Code.TruncatedCode() == item.Code.TruncatedCode() // this doesn't work :(
select b;
}
I'm trying to implement encrypted columns in EF4 and using the CTP5 features to allow simple use of POCO's to query the database. Sorry that this is a lot of words, but I hope the below gives enough to explain the need and the problem!
So, bit of background, and my progress so far:
The intention is that if you query the tables without using our DAL then the data is rubbish, but I don't want the developers to worry about if/when/how the data is encrypted.
For simplicity, at this stage I'm working on the assumption any string column will be encrypted.
Now, I have successfully implemented this for returning the data using the Objectmaterialized event, and for data commits using the SavingChanges event.
So given the following class:
public class Thing
{
public int ID { get; set; }
[Required]
public string Name { get; set; }
public DateTime Date { get; set; }
public string OtherString { get; set; }
}
The below query returns all the required values and the POCO materialized has clear data in it.
var things = from t in myDbContext.Things
select t;
where myDbContext.Things is a DbSet<Thing>
Likewise, passing an instance of Thing to Things.Add()
(with clear string data in the Name and/or OtherString values)
and then calling myDbContext.SaveChanges() encrypts the strings before it gets to the data store.
Now, the problem I have is in this query:
var things = from t in myDbContext.Things
where t.Name == "Hairbrush"
select t;
This results in the unencrypted value being compared to the encrypted value in the DB. Obviously I don't want to get all the records from the database, materialize them, and then filter the results based on any supplied Where clause... so what I need to do is: intercept that query and rewrite it by encrypting any strings in the Where clause.
So I've looked at:
writing a query provider, but that doesn't seem like the right solution... (is it?)
writing my own IQueryable wrapper for the DbSet which will capture the expression, run over it using an expression tree visitor and then forward the new expression to the DbSet...
Attempts at both have left me somewhat lost! I prefer the second solution i think since it feels a bit neater, and is probably clearer to other developers in future. But I'm happy to go with either or another better option!!
The main thing I am struggling with is when/how the LINQ expression is applied to the object... I think i've got myself a bit confused as to where the expression executes in the IQueryable object thus I'm not sure which method I need to implement in my wrapper to then grab and manipulate the expression being passed in...
I'm sure I'm missing something fairly obvious here and I'm waiting for that light bulb moment... but its not coming!!
Any help will be very gratefully received!
Thought I'd let you know what my final solution was.
In the end I have gone a wrapper class which implements a Where method, but without going to the extent of implementing IQueryable entirely. LINQ will still execute against the class (at least to the extent that I want/need it to) and will call the Where method with the expression from the LINQ.
I then traverse this ExpressionTree and replace my strings with encrypted values before forwarding the new expressiontree to the internal DbSet. and then returning the result.
Its pretty crude, and has its limitation, but works for our particular circumstance without problem.
Thanks,
Ben
you should use the QueryInterceptor attribute, search here in SO or in google and you find examples on how to use it.
a snippet:
[QueryInterceptor("Orders")]
public Expression<Func<Order, bool>> FilterOrders()
{
return o => o.Customer.Name == /* Current principal name. */;
}
// Insures that the user accessing the customer(s) has the appropriate
// rights as defined in the QueryRules object to access the customer
// resource(s).
[QueryInterceptor ("Customers")]
public Expression<Func<Customer, bool>> FilterCustomers()
{
return c => c.Name == /* Current principal name. */ &&
this.CurrentDataSource.QueryRules.Contains(
rule => rule.Name == c.Name &&
rule.CustomerAllowedToQuery == true
);
}
You can use David Fowler's Query Interceptor:
https://github.com/davidfowl/QueryInterceptor
One example of its use:
IQueryable q = ...;
IQueryable modifed = q.InterceptWith(new MyInterceptor());
And on class MyInterceptor:
protected override Expression VisitBinary(BinaryExpression node) {
if (node.NodeType == ExpressionType.Equal) {
// Change == to !=
return Expression.NotEqual(node.Left, node.Right);
}
return base.VisitBinary(node);
}