Is there any mechanism for doing a JOIN between an in-memory collection and entity framework while preserving the order.
What I am trying is
var itemsToAdd =
myInMemoryList.Join(efRepo.All(), listitem => listitem.RECORD_NUMBER,
efRepoItem => efRepoItem.RECORD_NUMBER, (left, right) => right);
which gives me the rather curiously titled "This method supports the LINQ to Entities infrastructure and is not intended to be used directly from your code." error.
Now of course I can do this iteratively with something like
foreach (var item in myInMemoryList)
{
var ho = efRepo.Where(h => h.RECORD_NUMBER == item.RECORD_NUMBER).FirstOrDefault();
tmp.Add(ho);
}
but this is an N+1 query. Which is nasty as myInMemoryList might be quite large!
Resharper can refactor that for me to
tmp = (from TypeOfItemInTheList item in myInMemoryList
select efRepo.Where(h => h.RECORD_NUMBER == item.RECORD_NUMBER)
.FirstOrDefault());
which I suspect is still doing N+1 queries. So any ideas for a better approach to getting ef entities that match (on key field) with an in-memory collection. The resulting set must be in the same order as the in-memory collection was.
No you cannot join in-memory collection with database result set without loading whole result set to the memory and performing the join with linq-to-objects. Try using contains instead of join:
var myNumbers = myInMemoryList.Select(i => i.RECORD_NUMBER);
var itemsToAdd = efRepo.Where(e => myNumbers.Contains(e.RECORD_NUMBER));
This will generate query with IN operator
You can read how you can do this with the PredicateBuilder from the LINQKit or Stored Procedures in my blog post.
http://kalcik.net/2014/01/05/joining-data-in-memory-with-data-in-database-table/
try this:
var list = (from n in efRepo
where myInMemoryList.Select(m=>m.RECORD_NUMBER).Contains(n.RECORD_NUMBER)
select n).ToList();
Contains will be translated to IN operator in SQL (only if your RECORD_NUMBER member is a primitive type like int, string, Guid, etc)
What about loading the whole efRepo? I mean something like this (ToArray()):
var itemsToAdd = myInMemoryList.Join(
efRepo.ToArray(),
listitem => listitem.RECORD_NUMBER, efRepoItem => efRepoItem.RECORD_NUMBER, (left, right) => right);
Related
I have the following code containing LINQ statements:
public async Task<HashSet<long>> GetMembersRecursive(IEnumerable<long> groupIds)
{
var containsGroupId = InExpression<Group>("Id", groupIds);
var containsParentId = InExpression<RecursiveGroupModel>("ParentId", groupIds);
var groupIdsArray = groupIds as long[] ?? groupIds.ToArray();
return new HashSet<long>(await MyContext
.Groups
.Where(containsGroupId)
.Select(a => new
{
Members = MyContext
.ViewWithRecursiveGroups
.Where(containsParentId)
.SelectMany(c => c.Group.Members)
.Union(a.Members)
.Where(b => !b.User.IsActive)
})
.SelectMany(a => a.Members.Select(b => b.MemberId))
.Distinct()
.ToListAsync());
}
private static Expression<Func<T, bool>> InExpression<T>(string propertyName, IEnumerable<long> array)
{
var p = Expression.Parameter(typeof(T), "x");
var contains = typeof(Enumerable).GetMethods(BindingFlags.Static | BindingFlags.Public)
.Single(x => x.Name == "Contains" && x.GetParameters().Length == 2)
.MakeGenericMethod(typeof(long));
var property = Expression.PropertyOrField(p, propertyName);
var body = Expression.Call(
contains
, Expression.Constant(array)
, property
);
return Expression.Lambda<Func<T, bool>>(body, p);
}
The error I receive is:
Microsoft.EntityFrameworkCore: Processing of the LINQ expression 'DbSet<RecursiveGroupModel>
.Where(b => __groupIdsArray_1
.Contains(b.ParentId))
.SelectMany(c => c.Group.GroupMembers)
.Union((MaterializeCollectionNavigation(
navigation: Navigation: Group.GroupMembers,
subquery: (NavigationExpansionExpression
Source: DbSet<GroupMember>
.Where(l0 => EF.Property<Nullable<long>>(l, "Id") != null && EF.Property<Nullable<long>>(l, "Id") == EF.Property<Nullable<long>>(l0, "GroupId1"))
PendingSelector: l0 => (NavigationTreeExpression
Value: (EntityReference: GroupMember)
Expression: l0)
)
.Where(i => EF.Property<Nullable<long>>((NavigationTreeExpression
Value: (EntityReference: Group)
Expression: l), "Id") != null && EF.Property<Nullable<long>>((NavigationTreeExpression
Value: (EntityReference: Group)
Expression: l), "Id") == EF.Property<Nullable<long>>(i, "GroupId1"))))' by 'NavigationExpandingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core. See https://go.microsoft.com/fwlink/?linkid=2101433 for more detailed information.
The view:
CREATE VIEW [dbo].[View_WithRecursiveGroups] AS
WITH RecursiveGroups (GroupId, ParentId) AS
(
SELECT Id, ParentId
FROM Group
WHERE ParentId IS NOT NULL
UNION ALL
SELECT Group.Id, t.ParentId
FROM GroupTree t
JOIN Group ON t.GroupId = Group.ParentId
)
SELECT * FROM RecursiveGroups
Apologies in advance if some variable names don't match up- I had to sanitize before posting.
I understand that it cannot convert code to SQL and so it's asking me to enumerate early or rewrite so that it's translatable. I have tired rearranging the query and breaking it up into smaller queries but the SelectMany on the recursive view seems to not be possible to convert to SQL.
Is there a way to get this working in-database? Or am I going about this completely the wrong way?
As an alternative, you can use raw sql query. In Entity Framework Code, we need to define a POCO class and a DbSet for that class. In your case you will need to define some YourClass:
public DbQuery<YourClass> YourClasses { get; set; }
and code to execute:
var result = context.YourClasses.FromSql("YOURSQL_SCRIPT").ToList();
var asyncresult = await context.YourClasses.FromSql("YOURSQL_SCRIPT").ToListAsync();
Yeah, welcome to the wonderfull world of EfCore 3.1 where all you can do is "Hello world".
Your query has various "problems" because EfCore does not really do LINQ processing except for super easy cases.
.Union(a.Members)
Can not be translated to run server side and client side processing is not enabled. Your only choises are:
Force server execution for both parts (using AsEnumerable) then Union on the client. That only works if you do not use that as part of a larger statement (i.e. intersect) otherwise it is "pull all the data to the client" time and that is not good.
At the current point in time I can only advice you to throw out EfCore and use EntityFramework which - as per framework 3.1 - is again available. Or use Entity Framework Classic which is a port that runs on netstandard 2.0 and has global query filters (which are THE ONE feature of EfCore I like). At last this is what I am currently getting to because - well - "better but without any features and not working" is not cutting it for me.
Whether or not EfCore will be extended (they seem not to see it as a fix) to handle anything except the most basic LINQ statements (and sometimes even not those) is unknown at this point - a lot of the changes in 3.1 are quite discouraging.
You MAY be able to move it into views etc. - but you may find out quite fast that EfCore has even more limitations and maintaining all the views gets quite tendious, too. I run into serious problems with the fact that I can not put any condition in front of any projection even in the most simple cases. And even simple bugs get commented on "we do not feel comfortable changing the pipeline, please wait for version 5 in november". Example? https://github.com/dotnet/efcore/issues/15279.
give that if you want to convert this view to Linq...
CREATE VIEW [dbo].[View_WithRecursiveGroups] AS
WITH RecursiveGroups (GroupId, ParentId) AS
(
SELECT Id, ParentId
FROM Group
WHERE ParentId IS NOT NULL
UNION ALL
SELECT Group.Id, t.ParentId
FROM GroupTree t
JOIN Group ON t.GroupId = Group.ParentId
)
var data1 = db.Group.where(x=>x.ParentId != nul)
.Select(x=>new {x.Id, x.ParentId})
.Tolist()
var data2 = (from g in db.Groups
join gt in db.GroupTree on g.ParentId equals gt.GroupId
select new { d.Id, ParentId })
.ToList();
create a class reprocenting the data and have the query return as List of known type and
just union the two lists.
linqpad is a very useful tool for learn how to create the linq which give you the sql you want.
I have a simple scenario.I want to list out all the employees except the logged in user.
Similar SQL Condition is
select * from employee where id not in(_loggedUserId)
How can I acheive the above using LINQ.I have tried the following query but not getting the desired list
int _loggedUserId = Convert.ToInt32(Session["LoggedUserId"]);
List<int> _empIds = _cmn.GetEmployeeCenterWise(_loggedUserId)
.Select(e => e.Id)
.Except(_loggedUserId)
.ToList();
Except expects argument of type IEnumerable<T>, not T, so it should be something like
_empIds = _cmn.GetEmployeeCenterWise(_loggedUserId)
.Select(e => e.Id)
.Except(new[] {_loggedUserId})
.ToList();
Also note, this is really redundant in the case when exclusion list contains only one item and can be replaces with something like .Where(x => x != _loggedUserId)
Why not use a very simple Where condition?
_empIds = _cmn.GetEmployeeCenterWise(_loggedUserId).Where(e=>e.Id != _loggedUserId).ToList();
The title of your question is how to perform a not in query against a database using LINQ. However, as others have pointed out your specific problem is better solved by a using users.Where(user => user.Id != loggedInUserId).
But there is still an answer on how to perform a query against a database using LINQ that results in NOT IN SQL being generated:
var userIdsToFilter = new[] { ... };
var filteredUsers = users.Where(user => !userIdsToFilter.Contains(user.Id));
That should generate the desired SQL using either Entity Framework or LINQ to SQL.
Entity Framework also allows you to use Except but then you will have to project the sequence to ID's before filtering them and if you need to original rows you need to fetch them again from the filtered sequence of ID's. So my advice is use Where with a Contains in the predicate.
Use LINQ without filtering. This will make your query execute much faster:
List<int> _empIds = _cmn.GetEmployeeCenterWise(_loggedUserId)
.Select(e => e.Id).ToList();
Now use List.Remove() to remove the logged-in user.
_empIds.Remove(_loggedUserId);
Is nested GroupBy() then flatten allowed in SQL? or available via ling-to-sql or the entity framework? Currently, I need to perform a retrieve in the middle of the query to make it work:
var query = (
from s in Prices
group s by new { s.P1, s.P2 } into FirstGroups
select FirstGroups
)
.ToList() // without it, exception is thrown
.SelectMany(g1 =>
g1.GroupBy(i => i.P3).OrderBy(i => i.Key).Take(2)
.SelectMany((g2, index) => g.Select(j => new
{
P1 = g1.Key.P1,
P2 = g1.Key.P2,
Index = index,
P3 = g2.P3,
P4 = j.P4,
}));
});
Single SelectMany works. Nested expressed this way doesn't work in linq-to-sql. My question is does l2s support it at all? if yes, how to write the query. if not, does any other linq to db technology support it, for example, the new entity framework?
Grouping and then flattening the results is the same as simply ordering on that value.
Suppose I have one collection, call it ids it is of type IEnumerable<string>, I have a second collection call it objects it's of type MyObject[]. MyObject has a string property called id. I would like a LINQ statement that returns all off the objects in the objects collection who's id matches any value in the ids collection. ids will be a strict subset of objects.Select(x => x.id). Meaning, for every string in ids I know there will be exactly one corresponding MyObject in objects. Can someone post a pure LINQ solution? I've tried a couple things with no luck. I can come up with an iterative solution easily enough so unless it's impossible to do with only LINQ please don't post any.
"Just" LINQ:
var r = obj.Where(o => ids.Any(id => id == o.id));
But better, for larger n, with a set:
var hs = new HashSet(ids);
var r = obj.Where(o => hs.Contains(o.id));
I think this is pretty straightforward with query syntax.
It would look something like:
var a = from o in objects
join i in ids on o.id equals i
select o;
If you just want a list of MyObject that match, you can do :
var solution = objects.Where(x=> ids.Contains(x.id));
With this instead, you'll get a List<T> where T is an Anonymous type with 2 properties, Id that is the string that work as "key" in this specific case, and Obj, a list of MyObject which id correspond to the Id property.
var solution = ids.Select(x=>new{ Id = x, Obj=objects.Where(y=>y.id == x).ToList()})
.ToList();
If you just want to know if there is any object in the intersection (which was what I was looking for)
Based on this
var a = from o in objects
join i in ids on o.id equals i
select o;
You can do this as well
var isEmpty = objects.Any(x => ids.Any(y => y == x.ToString()));
The accepted answer is correct. However, if someone doesn't like using SQL style LINQ, here is the LINQ extension method approach to solving the same problem.
var filteredObjects = objects.Join(ids, obj => obj.Id, id => id, (obj, _) => obj);
We are joining two different types, so the 2nd & 3rd Join parameter signify that join will be made on id.
The fourth parameter is used to select an object out of the resultant (obj, id) pair after applying join.
I am using LINQ to create a list. But I want to use a function at the end to generate the object iself, something LINQ complains about
LINQ to Entities does not recognize the method 'WashroomStatusItem GetWashroomStatusForItem(WashroomStatus)' method, and this method cannot be translated into a store expression.
What am I doing wrong?
var query = (from c in context.WashroomStatus
where c.WashroomId == GroupItem.WashroomID
select GetWashroomStatusForItem(c));
private WashroomStatusItem GetWashroomStatusForItem(WashroomStatus item)
{
WashroomStatusItem temp = new WashroomMonitorWCF.WashroomStatusItem();
//do stuff with it
return temp;
}
The problem is that the SQL conversion can't convert your method into SQL. You should use AsEnumerable() to "switch" from the out-of-process provider to LINQ to Objects. For example:
var query = context.WashroomStatus
.Where(c => c.WashroomId == GroupItem.WashroomID)
.AsEnumerable()
.Select(c => GetWashroomStatusForItem(c));
Note that if GetWashroomStatusForItem only uses some properties, you may want to project to those separately first, to reduce the amount of information fetched from the server:
var query = context.WashroomStatus
.Where(c => c.WashroomId == GroupItem.WashroomID)
.Select(c => new { c.Location, c.Date };
.AsEnumerable()
.Select(p => GetWashroomStatusForItem(p.Location, p.Date));
Jon Skeet's answer is correct, but I'd add that depending on the nature of GetWashroomStatusForItem(), it should probably either be broken down into LINQ statements and added into the query itself, or it should be executed after the query has returned.
So, lets say GetWashroomStatusForItem() looks something like this: note that this is extremely oversimplified.
public static WashroomStatus GetWashroomStatusForItem(Item c)
{
return c.WashroomStatus;
}
it should just be added to the LINQ query like this:
var query = (from c in context.WashroomStatus
where c.WashroomId == GroupItem.WashroomID
select c.WashroomStatus);
But if it relies heavily on stuff not in the db, I'd just end the Linq statement before you get the WashroomStatus, and then call GetWashroomStatusForItem() on the results. It's not gonna a performance difference since Linq uses lazy evaluation, and you generally want to keep db operations separate from "programmatic" ones.