Develop Reference

Filtering Entity Collections [duplicate]

This question already has answers here:
EF: Include with where clause [duplicate]
(5 answers)
Closed 6 years ago.
I want to be able to filter a child collection and only return items in that child collection which match certain conditions. Here's the code that I have now:
var q = from u in context.DbContext.Users select u;
q = q.Include(u => u.UserRoles.Select(ur => ur.Role))
.Where(u=> u.UserRoles.Any(ur=> ur.EnvironmentId == environmentId)
);
My issue with this code is that this is also returning UserRole objects in the UserRole collection that do not match.
For example, if my environmentId variable has a value of 1, in only want the UserRoles returned in the collection if they have a value of 1 for the EnvironmentId property.
As of right now, it is returning every UserRole regardless of the EnvironmentId value.
Edit
This is not a duplicate question as Gert Arnold has suggested. I do not want to create new or anonymous objects, and the solution i proposed below solves this problem, whereas the article linked to by Gert Arnold does not.

Your Where condition is not applied to the right collection. Here, you are applying the Where to the User collection so that it will only return users that have at least one role where EnvironmentId is 1. What you want to do instead is apply that to your Role collection to only join the ones you want. this doesn't workI believe something like this should work:
q = q.Include(
u => u.UserRoles.Where(ur => ur.EnvironmentId == environmentId)
.Select(ur => ur.Role))
What you can do instead would be to return a new object via a select (I'm getting into unsure territory right now :)
q = q.Select(u =>
new {
User = u,
Roles = u.UserRoles.Where(ur => ur.EnvironmentId == environmentId)
};
Now here comes the weird part... this will return you an anonymous object where the User propertie is your returned user, and Roles, your returned roles. If you wish you create a new class so that you can carry that value around outside of the scope of that block.
new class
public class UserWithRoles
{
Public User User {get; set;}
IEnumarable<Roles> Roles {get; set;}
}
query
q => q.Select(u =>
new UserWithRoles() {
User = u,
Roles = u.UserRoles.Where(ur => ur.EnvironmentId == environmentId)
};
That way you can declare a List<UserWithRoles> UserList and you could do UserList = q.ToList(); This might not be (probably is not) the best way to do it, but it is one way I believe it will work. If anyone is moire knowledgeable than me in LINQ's Include and knows how to make this work better, please post another answer or comment this one, I'd like to know too :)

you might consider just returning a list of userroles and you can select the users from this list if you need the user objects
var roles = from ur in context.DbContext.UsersRoles.Include("User")
where ur.EnvironmentId == environmentId
select ur;
var users = roles.SelectMany(a => a.Users).Distinct();

using the example provided here Filtering Related Entity Collections I came up with what seems to be a clear and elegant solution to the problem. this only loads the items in the collection if they match, and doesn't require creating any anonymous objects. (note: LazyLoading must be turned off for explicit loading to work)
User user;
var data = from u in context.DbContext.Users select u;
user = data.FirstOrDefault();
// load UserRoles and UserRoles.Role
context.Entry(user)
.Collection(u => u.UserRoles)
.Query()
.Include(ur => ur.Role)
.Where(ur => ur.EnvironmentId == environmentId)
.Load()
;

Get all users from WEB API

I want to create a service in a WEB API MVC5 project the returns users created.
This works
[Route("getUsers")]
// GET: User
public List<ApplicationUser> getUsers()
{
List<ApplicationUser> users = DbContext.Users.ToList();
return users;
}
but it returns all the data from the Users, where I'm only interested in returning FullName and Id.
Any suggestions on how to limit the result?

Three suggestions/enhancements:
Web API is meant to be used to create RESTful services. Check that you're calling the whole operation using HTTP/GET and the route itself says getXXX. Change the route to, at least, /users.
Don't return domain objects over the wire. You need to implement data-transfer objects. If you've few use cases to return users or one user, design a class which only have these properties. Also, this is important because objects returned by a DbContext are heavy: they also contain tracking information and other things since they're object proxies. You'll avoid some overhead and errors while serializing and deserializing.
Your code should look as follows: DbContext.Users.Select(u => new UserDto { Id = u.ID, Name = u.Name }).ToList(). In order to avoid manually mapping your objects, I would suggest that you take a look at AutoMapper library.
Design/implement a domain layer which can be injected/instantiated in your WebAPI in order to encapsulate this complexity and let the WebAPI call something like userService.ListUsers().

Change DbContext.Users.ToList() to DbContext.Users.Select(u => new ApplicationUser() {ID = u.ID, Name = u.Name}).ToList();

You don't even need to use List, Do this:
[Route("getUsers")]
// GET: User
public IQueryable<object> getUsers()
{
var users = DbContext.Users.Select( u => new {Id = u.Id, Fullname = u.Fullname});
return users;
}
If you don't have a direct Fullname property, you could get if from other properties like Firstname and Lastname like so:
var users = DbContext.Users.Select( u => new {Id = u.Id, Fullname = u.Firstname + " " + u.Lastname});

You can project your data using the .Select extension method for IEnumerable.
You could do something like this
var users = DbContext.Users(u=> new { Id = u.Id, Fullname = u.Fullname });
return users;

`from..where` or `FirstOrDefault` in LINQ

Traditionally, when I've tried to get data for a user from a database, and I've used the following method (to some degree):
DbUsers curUser = context.DbUsers.FirstOrDefault(x => x.u_LoginName == id);
string name = curUser.u_Name;
string email = curUser.u_Email;
You can see that all I want to do is get the Name and Email, but it seems to me that this LINQ query is getting everything stored in the database of that user, bringing it back, then allowing me to get what I want.
I have been doing some research and have found the following alternative:
var current = from s in context.DbUsers
where s.u_LoginName == id
select new {
name = s.u_Name,
email = s.u_Email
};
foreach (var user in current)
{
//Stuff Here
}
Which would be better, if any at all? Is there a lighter method to use when I only want to retrieve a few results / data?

If you want to get only two fields, then you should project your entity before query gets executed (and in this case query gets executed when you call FirstOrDefault). Use Select operator for projection to anonymous object with required fields:
var user = context.DbUsers
.Where(u => u.u_LoginName == id)
.Select(u => new { u.u_Name, u.u_Email })
.FirstOrDefault(); // query is executed here
string name = user.u_Name; // user is anonymous object
string email = user.u_Email;
That will generate SQL like:
SELECT TOP 1 u_Name, u_Email FROM DbUsers
WHERE u_LoginName = #id
In second case you are doing projection before query gets executed (i.e. enumeration started). That's why only required fields are loaded. But query will be slightly different (without TOP 1). Actually if you will convert second approach to lambda syntax, it will be almost same:
var query = context.DbUsers
.Where(u => u.u_LoginName == id)
.Select(u => new { u.u_Name, u.u_Email });
// query is defined but not executed yet
foreach (var user in query) // executed now
{
//Stuff Here
}
And just to show complete picture, without projection you get all fields of first found user:
DbUsers user = context.DbUsers
.Where(u => u.u_LoginName == id)
.FirstOrDefault(); // query is executed here
string name = user.u_Name; // user is DbUsers entity with all fields mapped
string email = user.u_Email;
In that case user entity is not projected before query is executed and you'll get all fields of user loaded from database and mapped to user entity:
SELECT TOP 1 u_LoginName, u_Name, u_Email /* etc */ FROM DbUsers
WHERE u_LoginName = #id

The second is better. You only get the needed data from database so the network traffic is lighter.
You can have the same result with extension methods:
var user = context.DbUsers
.Where(x => x.u_LoginName == id)
.Select(x => new {...})
.FirstOrDefault();

If you need not whole entity, but some values from it, then use new {name = s.u_Name, email = s.u_Email}. Because, this object is much "lighter" for cunstruction.
When you get entity with FirstOrDefault, it' saved in DBContext, but you don't do anything with it.
So, i advice you to get only data you need.

Entity Framework with LINQ aggregate to concatenate string?

This is easy for me to perform in TSQL, but I'm just sitting here banging my head against the desk trying to get it to work in EF4!
I have a table, lets call it TestData. It has fields, say: DataTypeID, Name, DataValue.
DataTypeID, Name, DataValue
1,"Data 1","Value1"
1,"Data 1","Value2"
2,"Data 1","Value3"
3,"Data 1","Value4"
I want to group on DataID/Name, and concatenate DataValue into a CSV string. The desired result should contain -
DataTypeID, Name, DataValues
1,"Data 1","Value1,Value2"
2,"Data 1","Value3"
3,"Data 1","Value4"
Now, here's how I'm trying to do it -
var query = (from t in context.TestData
group h by new { DataTypeID = h.DataTypeID, Name = h.Name } into g
select new
{
DataTypeID = g.Key.DataTypeID,
Name = g.Key.Name,
DataValues = (string)g.Aggregate("", (a, b) => (a != "" ? "," : "") + b.DataValue),
}).ToList()
The problem is that LINQ to Entities does not know how to convert this into SQL. This is part of a union of 3 LINQ queries, and I'd really like it to keep it that way. I imagine that I could retrieve the data and then perform the aggregate later. For performance reasons, that wouldn't work for my app. I also considered using a SQL server function. But that just doesn't seem "right" in the EF4 world.
Anyone care to take a crack at this?

If the ToList() is part of your original query and not just added for this example, then use LINQ to Objects on the resulting list to do the aggregation:
var query = (from t in context.TestData
group t by new { DataTypeID = t.DataTypeID, Name = t.Name } into g
select new { DataTypeID = g.Key.DataTypeID, Name = g.Key.Name, Data = g.AsEnumerable()})
.ToList()
.Select (q => new { DataTypeID = q.DataTypeID, Name = q.Name, DataValues = q.Data.Aggregate ("", (acc, t) => (acc == "" ? "" : acc + ",") + t.DataValue) });
Tested in LINQPad and it produces this result:

Some of the Answers suggest calling ToList() and then perform the calculation as LINQ to OBJECT. That's fine for a little amount of data, but what if I have a huge amount of data that I do not want to load into memory too early, then, ToList() may not be an option.
So, the better idea would be to process/format the data in the presentation layer and let the Data Access layer do only loading or saving raw data that SQL likes.
Moreover, in your presentation layer, most probably you are filtering the data by paging, or maybe you are showing one row in the details page, so, the data you will load into the memory is likely smaller than the data you load from the database. (Your situation/architecture may be different,.. but I am saying, most likely).
I had a similar requirement. My problem was to get the list of items from the Entity Framework object and create a formatted string (comma separated value)
I created a property in my View Model which will hold the raw data from the repository and when populating that property, the LINQ query won't be a problem because you are simply querying what SQL understands.
Then, I created a get-only property in my ViewModel which reads that Raw entity property and formats the data before displaying.
public class MyViewModel
{
public IEnumerable<Entity> RawChildItems { get; set; }
public string FormattedData
{
get
{
if (this.RawChildItems == null)
return string.Empty;
string[] theItems = this.RawChildItems.ToArray();
return theItems.Length > 0
? string.Format("{0} ( {1} )", this.AnotherRegularProperty, String.Join(", ", theItems.Select(z => z.Substring(0, 1))))
: string.Empty;
}
}
}
Ok, in that way, I loaded the Data from LINQ to Entity to this View Model easily without calling.ToList().
Example:
IQueryable<MyEntity> myEntities = _myRepository.GetData();
IQueryable<MyViewModel> viewModels = myEntities.Select(x => new MyViewModel() { RawChildItems = x.MyChildren })
Now, I can call the FormattedData property of MyViewModel anytime when I need and the Getter will be executed only when the property is called, which is another benefit of this pattern (lazy processing).
An architecture recommendation: I strongly recommend to keep the data access layer away from all formatting or view logic or anything that SQL does not understand.
Your Entity Framework classes should be simple POCO that can directly map to a database column without any special mapper. And your Data Access layer (say a Repository that fetches data from your DbContext using LINQ to SQL) should get only the data that is directly stored in your database. No extra logic.
Then, you should have a dedicated set of classes for your Presentation Layer (say ViewModels) which will contain all logic for formatting data that your user likes to see. In that way, you won't have to struggle with the limitation of Entity Framework LINQ. I will never pass my Entity Framework model directly to the View. Nor, I will let my Data Access layer creates the ViewModel for me. Creating ViewModel can be delegated to your domain service layer or application layer, which is an upper layer than your Data Access Layer.

Thanks to moi_meme for the answer. What I was hoping to do is NOT POSSIBLE with LINQ to Entities. As others have suggested, you have to use LINQ to Objects to get access to string manipulation methods.
See the link posted by moi_meme for more info.
Update 8/27/2018 - Updated Link (again) - https://web.archive.org/web/20141106094131/http://www.mythos-rini.com/blog/archives/4510
And since I'm taking flack for a link-only answer from 8 years ago, I'll clarify just in case the archived copy disappears some day. The basic gist of it is that you cannot access string.join in EF queries. You must create the LINQ query, then call ToList() in order to execute the query against the db. Then you have the data in memory (aka LINQ to Objects), so you can access string.join.
The suggested code from the referenced link above is as follows -
var result1 = (from a in users
b in roles
where (a.RoleCollection.Any(x => x.RoleId = b.RoleId))
select new
{
UserName = a.UserName,
RoleNames = b.RoleName)
});
var result2 = (from a in result1.ToList()
group a by a.UserName into userGroup
select new
{
UserName = userGroup.FirstOrDefault().UserName,
RoleNames = String.Join(", ", (userGroup.Select(x => x.RoleNames)).ToArray())
});
The author further suggests replacing string.join with aggregate for better performance, like so -
RoleNames = (userGroup.Select(x => x.RoleNames)).Aggregate((a,b) => (a + ", " + b))

You are so very close already. Try this:
var query = (from t in context.TestData
group h by new { DataTypeID = h.DataTypeID, Name = h.Name } into g
select new
{
DataTypeID = g.Key.DataTypeID,
Name = g.Key.Name,
DataValues = String.Join(",", g),
}).ToList()
Alternatively, you could do this, if EF doesn't allow the String.Join (which Linq-to-SQL does):
var qs = (from t in context.TestData
group h by new { DataTypeID = h.DataTypeID, Name = h.Name } into g
select new
{
DataTypeID = g.Key.DataTypeID,
Name = g.Key.Name,
DataValues = g
}).ToArray();
var query = (from q in qs
select new
{
q.DataTypeID,
q.Name,
DataValues = String.Join(",", q.DataValues),
}).ToList();

Maybe it's a good idea to create a view for this on the database (which concatenates the fields for you) and then make EF use this view instead of the original table?
I'm quite sure it's not possible in a LINQ statement or in the Mapping Details.

Dynamic where clause using Linq to SQL in a join query in a MVC application

I am looking for a way to query for products in a catalog using filters on properties which have been assigned to the product based on the category to which the product belongs. So I have the following entities involved:
Products
-Id
-CategoryId
Categories
[Id, Name, UrlName]
Properties
[Id, CategoryId, Name, UrlName]
PropertyValues
[Id, PropertyId, Text, UrlText]
ProductPropertyValues
[ProductId, PropertyValueId]
When I add a product to the catalog, multiple ProductPropertyValues will be added based on the category and I would like to be able to filter all products from a category by selecting values for one or more properties. The business logic and SQL indexes and constraints make sure that all UrlNames and texts are unique for values properties and categories.
The solution will be a MVC3 EF code first based application and the routing is setup as followed:
/products/{categoryUrlName}/{*filters}
The filter routing part has a variable length so multiple filters can be applied. Each filter contains the UrlName of the property and the UrlText of the value separated by an underscore.
An url could look like this /products/websites/framework_mvc3/language_csharp
I will gather all filters, which I will hold in a list, by reading the URL. Now it is time to actually get the products based on multiple properties and I have been trying to find the right strategy.
Maybe there is another way to implement the filters. All larger web shops use category depending filters and I am still looking for the best way to implement the persistence part for this type of functionality. The suggested solutions result in an "or" resultset if multiple filters are selected. I can imagine that adding a text property to the product table in which all property values are stores as a joined string can work as well. I have no idea what this would cost performance wise. At leased there will be no complex join and the properties and their values will be received as text anyway.
Maybe the filtering mechanism can be done client side ass well.

The tricky part about this is sending the whole list into the database as a filter. Your approach of building up more and more where clauses can work:
productsInCategory = ProductRepository
.Where(p => p.Category.Name == category);
foreach (PropertyFilter pf in filterList)
{
PropertyFilter localVariableCopy = pf;
productsInCategory = from product in productsInCategory
where product.ProductProperties
.Any(pp => pp.PropertyValueId == localVariableCopy.ValueId)
select product;
}
Another way to go is to send the whole list in using the List.Contains method
List<int> valueIds = filterList.Select(pf => pf.ValueId).ToList();
productsInCategory = ProductRepository
.Where(p => p.Category.Name == category)
.Where(p => p.ProductProperties
.Any(pp => valueIds.Contains(pp.PropertyValueId)
);

IEnumerable<int> filters = filterList.Select(pf => pf.ValueId);
var products = from pp in ProductPropertyRepository
where filters.Contains(pp.PropertyValueId)
&& pp.Product.Category.Name == category
select pp.Product;
Bear in mind that as Contains is used, the filters will be passed in as sproc parameters, this means that you have to be careful not to exceed the sproc parameter limit.

I came up with a solution that even I can understand... by using the 'Contains' method you can chain as many WHERE's as you like. If the WHERE is an empty string, it's ignored (or evaluated as a select all). Here is my example of joining 2 tables in LINQ, applying multiple where clauses and populating a model class to be returned to the view.
public ActionResult Index()
{
string AssetGroupCode = "";
string StatusCode = "";
string SearchString = "";
var mdl = from a in _db.Assets
join t in _db.Tags on a.ASSETID equals t.ASSETID
where a.ASSETGROUPCODE.Contains(AssetGroupCode)
&& a.STATUSCODE.Contains(StatusCode)
&& (
a.PO.Contains(SearchString)
|| a.MODEL.Contains(SearchString)
|| a.USERNAME.Contains(SearchString)
|| a.LOCATION.Contains(SearchString)
|| t.TAGNUMBER.Contains(SearchString)
|| t.SERIALNUMBER.Contains(SearchString)
)
select new AssetListView
{
AssetId = a.ASSETID,
TagId = t.TAGID,
PO = a.PO,
Model = a.MODEL,
UserName = a.USERNAME,
Location = a.LOCATION,
Tag = t.TAGNUMBER,
SerialNum = t.SERIALNUMBER
};
return View(mdl);
}

I know this an old answer but if someone see's this I've built this project:
https://github.com/PoweredSoft/DynamicLinq
Which should be downloadable on nuget as well:
https://www.nuget.org/packages/PoweredSoft.DynamicLinq
You could use this to loop through your filter coming from query string and do
something in the lines of
query = query.Query(q =>
{
q.Compare("AuthorId", ConditionOperators.Equal, 1);
q.And(sq =>
{
sq.Compare("Content", ConditionOperators.Equal, "World");
sq.Or("Title", ConditionOperators.Contains, 3);
});
});