Develop Reference

Too much data in Contains (Linq): How to increase performace

I have a linq query like this:
from a in context.table_A
join b in
(from temp in context.table_B
where idlist.Contains(temp.id)
select temp)
on a.seq_id equals b.seq_id into c
where
idlist.Contains(a.id)
select new MyObject
{
...
}).ToList();
idlist is List
The problem I have is that the idlist has too many values (hundreds of thousands to several million records). It works fine with few records but when there are too many records, the contains function is error.
Error log is
The query plan cannot be created due to lack of internal resources of
the query processor. This is a rare event and only occurs for very
complex queries or queries that reference a very large number of
tables or partitions. Make the query easy.
I want to improve the performance of this section. Any ideas?

I would suggest to install extension linq2db.EntityFrameworkCore and use temporary tables with fast BulkCopy
public class IdItem
{
public int Id { get; set; }
}
...
var items = idList.Select(id => new IdItem { Id = id });
using var db = context.CreateLinqToDBConnection();
using var temp = db.CreateTempTable("#IdList", items);
var query =
from a in context.table_A
join id1 in temp on a.Id equals id1.Id
join b in context.table_B on a.seq_id equals b.seq_id
join id2 in temp on b.Id equals id2.Id
select new MyObject
{
...
};
// switch to alternative translator
var query = query.ToLinqToDB();
var result = query.ToList();

Refine LINQ join of two List<T> Objects, Group, and Count

I have two List objects. I want a count of how many items between the two list match. Now I could loop through with a counter as I come across matches....but that would be kinda lame.
However this one is stretching my LINQ knowledge. I believe what I want to do is Join, discern (where), group, and project the count. I came to this approach by reading similar SO questions and LINQ documentation.
However if this is flawed by all means it doesn't have to be this way. I just want the count of matching elements.
So my "master" object is:
public class Master
{
public StringModel OldText { get; private set; }
public StringModel NewText { get; private set; }
public Master()
{
OldText = new StringModel();
NewText = new StringModel();
}
}
StringModel is:
public class StringModel
{
public List<strPiece> Lines { get; private set; }
public StringModel()
{
Lines = new List<strPiece>();
}
}
My LINQ thus far is:
var unchangedJoin = from oldLine in Master.OldText.Lines
join newLine in Master.NewText.Lines
on oldLine.Type equals newLine.Type
where oldLine.Type == "ABC"
group oldLine by --as you can see I kinda break down here.
Any help finishing would be appreciated. Also if I need to post more code just let me know.
Thank You

Sounds like a good use for intersect
var infoQuery =
(from cust in db.Customers
select cust.Country)
.Intersect
(from emp in db.Employees
select emp.Country)
;

Just perform a GroupJoin rather than a Join.
var unchangedJoin = from oldLine in Master.OldText.Lines
join newLine in Master.NewText.Lines
on oldLine.Type equals newLine.Type
into newMatches
where oldLine.Type == "ABC"
select oldLine;
var matches = unchangedJoin.Count();

I too would recommend Intersect. If you go that way, using the fluent syntax makes more sense.
int matches = Master.OldText.Lines
.Where(line => line.Type == "ABC")
.Intersect(Master.NewText.Lines)
.Count();

Dynamic Join of multiple entities based on some filter in Entity Framework

I am pretty new to Entity Framework and LINQ and I have an entity with more than 10+ other associated entities (one-to-many relationships). Now, I'm planning to make a search page in my application in which users could select which fields (i.e. those 10+ tables) they want to be considered when searching.
Now, I'm trying to write a query to achieve the above goal. Any help how I could sort this out using LINQ method syntax? I mean, to write a multiple join query based on user's choice. (i.e. which of Class1, Class2, ... to join with main Entity to finally have all the related fields in one place). Below is a sample code (Just a hunch, in fact)
if(somefilter#1)
result = db.Companies.Join(db.Channels, p => p.Id, k => k.CId,
(p, k) => new {Company = p, Channels=k});
if(somefilter#2)
result = result.Join(db.BusinnessType, ........);
if(somefilter#3)
result = result.Join(db.Values, .......);

For complex queries it may be easier to use the other LINQ notation. You could join multiple entities like this:
from myEntity in dbContext.MyEntities
join myOtherEntity in dbContext.MyOtherEntities on myEntity.Id equals myOtherEntity.MyEntityId
join oneMoreEntity in dbContext.OneMoreEntities on myEntity.Id equals oneMoreEntity.MyEntityId
select new {
myEntity.Id,
myEntity.Name,
myOtherEntity.OtherProperty,
oneMoreEntity.OneMoreProperty
}
You can join in other entities by adding more join statements.
You can select properties of any entity from your query. The example I provided uses a dynamic class, but you can also define a class (like MyJoinedEntity) into which you can select instead. To do it you would use something like:
...
select new MyJoinedEntity {
Id = myEntity.Id,
Name = myEntity.Name,
OtherProperty = myOtherEntity.OtherProperty,
OneMoreProperty = oneMoreEntity.OneMoreProperty
}
EDIT:
In case when you want to have conditional joins you can define MyJoinedEntity with all the properties you will need if you were to join everything. Then break up the join into multiple methods. Like this:
public IEnumerable<MyJoinedEntity> GetEntities() {
var joinedEntities = from myEntity in dbContext.MyEntities
join myOtherEntity in dbContext.MyOtherEntities on myEntity.Id equals myOtherEntity.MyEntityId
join oneMoreEntity in dbContext.OneMoreEntities on myEntity.Id equals oneMoreEntity.MyEntityId
select new MyJoinedEntity {
Id = myEntity.Id,
Name = myEntity.Name,
OtherProperty = myOtherEntity.OtherProperty,
OneMoreProperty = oneMoreEntity.OneMoreProperty
};
if (condition1) {
joinedEntities = JoinWithRelated(joinedEntities);
}
}
public IEnumerable<MyJoinedEntity> JoinWithRelated(IEnumerable<MyJoinedEntity> joinedEntities) {
return from joinedEntity in joinedEntities
join relatedEntity in dbContext.RelatedEntities on joinedEntity.Id equals relatedEntity.MyEntityId
select new MyJoinedEntity(joinedEntity) {
Comments = relatedEntity.Comments
};
}

CRM LINQ Composite join "The method 'Join' is not supported" error

Im getting a "The method 'Join' is not supported" error... Funny thing is that i simply converted the 1st LINQ into the 2nd version and it doesnt work...
What i wanted to have was LINQ version #3, but it also doesnt work...
This works
var query_join9 = from s in orgSvcContext.CreateQuery(ServiceAppointment.EntityLogicalName)
join b in orgSvcContext.CreateQuery(bh_product.EntityLogicalName)
on s["bh_contract"] equals b["bh_contract"]
where ((EntityReference)s["bh_contract"]).Id == Guid.Parse("09BDD5A9-BBAF-E111-A06E-0050568B1372")
select new
{
Events = s,
Products = b
};
This doesn't
var query_join9 = from s in orgSvcContext.CreateQuery(ServiceAppointment.EntityLogicalName)
join b in orgSvcContext.CreateQuery(bh_product.EntityLogicalName)
on new { contractid = s["bh_contract"] }
equals new { contractid = b["bh_contract"] }
where ((EntityReference)s["bh_contract"]).Id == Guid.Parse("09BDD5A9-BBAF-E111-A06E-0050568B1372")
select new
{
Events = s,
Products = b
};
Also, this doesn't, which is a composite join and what i really aim for
var query_join9 = from s in orgSvcContext.CreateQuery(ServiceAppointment.EntityLogicalName)
join b in orgSvcContext.CreateQuery(bh_product.EntityLogicalName)
on new { contractid = s["bh_contract"], serviceid = s["serviceid"] }
equals new { contractid = b["bh_contract"], serviceid = s["serviceid"] }
where ((EntityReference)s["bh_contract"]).Id == Guid.Parse("09BDD5A9-BBAF-E111-A06E-0050568B1372")
select new
{
Events = s,
Products = b
};
I tried early binding and still doesnt work...
var query_join9 = from s in orgSvcContext.CreateQuery<ServiceAppointment>()
join b in orgSvcContext.CreateQuery<bh_product>()
on new { foo = s.bh_contract.Id }
equals new { foo = b.bh_Contract.Id }
where s.bh_contract.Id == Guid.Parse("09BDD5A9-BBAF-E111-A06E-0050568B1372")
select new
{
Events = s,
Products = b
};
stil not working
var query_join9 = from s in orgSvcContext.CreateQuery<ServiceAppointment>()
join b in orgSvcContext.CreateQuery<bh_product>()
on new { s.bh_contract.Id, s.ServiceId }
equals new { b.bh_Contract.Id, ServiceId = b.bh_Service }
where s.bh_contract.Id == Guid.Parse("09BDD5A9-BBAF-E111-A06E-0050568B1372")
select new
{
Events = s,
Products = b
};
But im simply trying to do the example(s) here How to do joins in LINQ on multiple fields in single join
What am i missing?
Thanks in advance

While I'm not entirely sure which CRM you're using, I think you're misunderstanding something.
In order for a LINQ query to work, there needs to be a LINQ provider for the underlying data source -- the bit of code responsible for translating chain of e.g. Join, Where, operator usage, etc, etc, into the query API of the data source. This might be SQL, some custom query language, or some chain of methods.
Two LINQ providers (such as, one for LINQ to DataSet and some custom provider you've written yourself) don't have to support the same methods and other code. The precise subset of LINQ methods (and/or other embedded statements) a LINQ provider supports is dependent on its implementation.
Looking at it like that, it's not that surprising that the LINQ provider you're using doesn't seem to comprehend the standard syntax for joins using multiple fields, or doesn't seem to comprehend the usage of anonymous types at all.
My advice is to search the documentation of the supplied LINQ provider to see which query operations it supports (perhaps there is a note about this specific mode of query not being supported). Failing that, you'll have to resort to some sort of other query -- one not involving an equijoin. Perhaps your best option is to perform the joins separately, and then intersect the two result groups. It really depends on the specifics of the case.

Have you looked at the MSDN samples. There are some multiple-column join examples there:
using (ServiceContext svcContext = new ServiceContext(_serviceProxy))
{
var list_join = (from a in svcContext.AccountSet
join c in svcContext.ContactSet
on a.PrimaryContactId.Id equals c.ContactId
where a.Name == "Contoso Ltd" && <<--- multiple join here
a.Address1_Name == "Contoso Pharmaceuticals"
select a).ToList();
foreach (var c in list_join)
{
System.Console.WriteLine("Account " + list_join[0].Name
+ " and it's primary contact "
+ list_join[0].PrimaryContactId.Id);
}
}
This other thread might be relevant

How can I query this hierarchical data using LINQ?

I have 3 kinds of objects: Agency, BusinessUnit and Client (each with their own respective table)
In terms of hierarchy, Agencies own BusinessUnits, and BusinessUnits own Clients.
I have 3 C# POCO Objects to represent them (I usually select new {} into them, rather than use the LINQ generated classes):
public class Agency
{
public IEnumerable<BusinessUnit> BusinessUnits { get; set; }
}
public class BusinessUnit
{
public IEnumerable<Client> Clients { get; set; }
}
public class Client
{
public int NumberOfAccounts { get; set; }
public Decimal AmountOfPlacement { get; set; }
public Decimal AvgBalance { get; set; }
public Double NeuPlacementScore { get; set; }
}
You can see that Agencies contain a list of BusinessUnits, and BusinessUnits contain a list of Clients.
I also have a mapping table called BAC_Map in the database which says which owns which, and it looks something like this:
How can I construct a query, so I can query for and return a list of Agencies? Meaning that, I want each Agency to have its list of BusinessUnit objects set, and I want the list of BusinessObjects to have its list of Clients set.
I can do basic LINQ queries, but this is a little over my head concerning the Map table and the multiple? queries.
How could I construct a method like GetAllAgencies() which would query, for not only all agencies, but populate its BusinessUnits that Agency owns, and the Clients those BusinessUnits own?
Edit: Any tips or info is appreciated. Do I need to do joins? Does this need to be multiple queries to return an Agency list, with its submembers populated?

If you drop all four tables (Agency, BusinessUnit, Client, Map) on the linq to sql designer, and draw relationships from Map to the other three, there will be some useful properties on Map.
//construct a query to fetch the row/column shaped results.
var query =
from m in db.map
//where m.... ?
let a = m.Agency
let b = m.BusinessUnit
let c = m.Client
// where something about a or b or c ?
select new {
AgencyID = a.AgencyID,
AgencyName = a.Name,
BusinessUnitID = b.BusinessUnitID,
ClientID = c.ClientID,
NumberOfAccounts = c.NumberOfAccounts,
Score = c.Score
};
//hit the database
var rawRecords = query.ToList();
//shape the results further into a hierarchy.
List<Agency> results = rawRecords
.GroupBy(x => x.AgencyID)
.Select(g => new Agency()
{
Name = g.First().AgencyName,
BusinessUnits = g
.GroupBy(y => y.BusinessUnitID)
.Select(g2 => new BusinessUnit()
{
Clients = g2
.Select(z => new Client()
{
NumberOfAccounts = z.NumberOfAccounts,
Score = z.Score
})
})
})
.ToList();
If approriate filters are supplied (see the commented out where clauses), then only the needed portions of the tables will be pulled into memory. This is standard SQL joining at work here.

I created your tables in a SQL Server database, and tried to recreate your scenario in LinqPad. I ended up with the following LINQ statements, which basically result in the same structure of your POCO classes:
var map = from bac in BAC_Maps
join a in Agencies on bac.Agency_ID equals a.Agency_ID
join b in BusinessUnits on bac.Business_Unit_ID equals b.Business_Unit_ID
join c in Clients on bac.Client_ID equals c.Client_ID
select new
{
AgencyID = a.Agency_ID,
BusinessUnitID = b.Business_Unit_ID,
Client = c
};
var results = from m in map.ToList()
group m by m.AgencyID into g
select new
{
BusinessUnits = from m2 in g
group m2 by m2.BusinessUnitID into g2
select new
{
Clients = from m3 in g2
select m3.Client
}
};
results.Dump();
Note that I called map.ToList() in the second query. This actually resulted in a single, efficient query. My initial attempt did not include .ToList(), and resulted in nine separate queries to produce the same results. The query generated by the .ToList() version is as follows:
SELECT [t1].[Agency_ID] AS [AgencyID], [t2].[Business_Unit_ID] AS [BusinessUnitID], [t3].[Client_ID], [t3].[NumberOfAccounts], [t3].[AmountOfPlacement], [t3].[AvgBalance], [t3].[NeuPlacementScore]
FROM [BAC_Map] AS [t0]
INNER JOIN [Agencies] AS [t1] ON [t0].[Agency_ID] = [t1].[Agency_ID]
INNER JOIN [BusinessUnits] AS [t2] ON [t0].[Business_Unit_ID] = [t2].[Business_Unit_ID]
INNER JOIN [Clients] AS [t3] ON [t0].[Client_ID] = [t3].[Client_ID]
Here is a screenshot of the results:
alt text http://img411.imageshack.us/img411/5003/agencybusinessunitclien.png

If you are doing this with direct LINQ to SQL, there is no way to do this without some kind of recursion, whether you do it yourself or you hide it behind an extension method. Recursive SQL is very bad (many round trips, many single queries).
There are two options here. One is to pull the entire table(s) with the hierarchy into memory and use LINQ to Objects on it. Leave the "details" tables in SQL. If you have less than several thousand entities, this is probably the most efficient way to go. You can keep a single copy of the table(s) in cache and refresh them when necessary. When you need to fetch more detailed data from the DB for a single record, you can reattach that entity from your cached hierarchy to a new DataContext and fetch it.
The other option is to use a more complex relationship model in your database. Storing parent only by nature demands recursion, but you can use the adjacency list model to construct a single query which can span many levels of inheritance. This will mean your LINQ to SQL queries become less intuitive (querying against Entity.Right and Entity.Left isn't quite as pretty as Parent or Children...) but you can do in one query what might take hundreds or thousands in the literal recursive approach.