Develop Reference

How to exclude from query's result in linq?

In the first picture I have the result of first query, the highlighted part indicates the rows that would be excluded by applying the filter on the second query, in the second I have the result of query select * from exlusion_table
I have to make a change to the first query to have it exclude the items retrieved from the second query
the first query:
var u = from a in cx.VW_LIST
where (a.PRJ == codPrj) && (a.DATE > date_ || a.DATE == null || date_ == null)
&& (x.Contains(a.CODE) || x.Count() == 0)
select a)
the second query:
var y = from esc in cx.EXCLUSION select esc
The first query should be modified to exclude all the rows that have the value fcode = the fcode of the second query (in the case in which the fscode of the second query = null) or that (fcode = fcode of the second query && fscode = fscode of the second query )

You can use Any(). ie:
var u = from a in cx.VW_LIST
where (a.PRJ == codPrj)
&& (a.DATE > date_ || a.DATE == null || date_ == null)
&& (x.Contains(a.CODE) || x.Count() == 0)
&& (!cx.EXCLUSION.Any( y => x.fscode == y.fscode && x.fcode == y.fcode ))
select a)

There are two approaches to this, one is to use ! and ANY() to filter out records found within the other list, this will compile into WHERE NOT EXISTS(_exclusion_) filter expression
var excluded = cx.EXCLUSION.AsQueryable();
var query = from vw in cx.VW_LIST
where vw.PRJ == codPrj
where vw.DATE == null || date_ == null || vw.DATE > date_
where !x.Any() || x.Contains(vw.CODE)
where !excluded.Any(esc => vw.fcode == esc.fcode
&& (esc.fscode == null || vw.fscode == esc.fscode))
select vw;
var results = query.ToList();
The tricky element is the null for fscode in the excluded table, that needs to act as wildcard match, or negate the fscode comparison.
It is not necessary to split the excluded query out into it's own query, we could have referenced to the cx.EXCLUSION table directly and it would have exactly the same effect, this shows you an encapsulation technique for building the LINQ query in a way that you could easily increase the complexity of the exclusion lookup without creating a mess of your overall query.
You may also find need to conditionally build the query, this is where fluent syntax provides a more modular approach:
bool filterExcludedRecords = true;
...
var excluded = cx.EXCLUSION.AsQueryable();
var query = cx.VW_LIST.Where(vw => vw.PRJ == codPrj)
.Where(vw => vw.DATE == null || date_ == null || vw.DATE > date_)
.Where(vw => !x.Any() || x.Contains(vw.CODE));
if(filterExcludedRecords)
query = query.Where(vw => !excluded.Any(esc => vw.fcode == esc.fcode
&& (esc.fscode == null || vw.fscode == esc.fscode)));
var results = query.ToList();
OUTER JOIN
Another method is to use a LFET OUTER JOIN where the exclusion match is not found:
var excluded = cx.EXCLUSION.AsQueryable();
var query = from vw in cx.VW_LIST
where vw.PRJ == codPrj
where vw.DATE == null || date_ == null || vw.DATE > date_
where !x.Any() || x.Contains(vw.CODE)
from esc in excluded.Where(e => vw.fcode == e.fcode
&& (e.fscode == null || vw.fscode == e.fscode))
.DefaultIfEmpty()
where esc.fscode == null
select vw;
var results = query.ToList();
The WHERE NOT EXISTS is often superior in terms of performance, OUTER JOIN may provide better response in an un-optimised table or when the number of rows in the exclusion list is significantly small and the number of rows in the main table is very large.
I include this query option for completeness, it is not well known that you can create simple outer joins by adding a new from clause to the query.

Why is my where condition in Linq query not working properly?

I have a linq query where i have the following where condition:
where obj.DATUMREAL == date && (real == 2 ? true : (obj.ZAPKONC == null ? 0 : ((bool)obj.ZAPKONC ? 1 : 0)) == real)
If I set parameter real to 2, it works normally, but if I set it to 1 or 0 the where statement just ignores the first condition obj.DATUMREAL == date and it only filters with the second part after the &&.
Here is the whole query:
var result = (from obj in entities.OBJKVPOPLANU
where obj.DATUMREAL == date
&& (real == 2 ?
true : (obj.ZAPKONC == null ? 0
: ((bool)obj.ZAPKONC ? 1 : 0)) == real)
group obj by obj.VODSISTEM into objg
join v in entities.SIVODSISTEM
on objg.FirstOrDefault().VODSISTEM equals v.SIFRA.ToString()
into vj
from subpet in vj.DefaultIfEmpty()
select new
{
subpet.SIFRA,
STEV = objg.Count(),
subpet.NAZIV
}).ToList();
return result;
If I set real value to 2 I get 30 items returned which are filtered by the first part of where, but if I set it to 1 or 0 I get more items that completely ignore the first part of where.
UPDATE
I solved it by swapping the two parts of the where statement around. I don't know what the issue was

Use Linq to search through list

Is it possible to use a Linq query to search through a List? In my web app I have to potentially process over 14k records based upon a spreadsheet uploaded by the user. With each record processed, I need to compare that record against what we currently have in our database in order to make sure we either aren't adding a duplicate or I know what record I need to be updating/editing.
Instead of hitting the database 14k times or more, I wanted to pull all the records contained on this table into a List, and then perform a search based on a set of conditions.
Here is the Linq query I currently have that hits the database. The business rules are pretty... complicated so I won't bother you with the details but these are the conditions that I need to satisfy for the search. I've tested this query and it returns the expected results.
var previousZips = (from z in db.ZipCodeTerritory
where (item.ZipCode.Equals(null) ?
z.StateCode.Equals(item.StateCode) &&
z.ChannelCode.Equals(item.ChannelCode) &&
SqlFunctions.DateDiff("DAY", z.EndDate, item.EndDate) == 0 :
z.StateCode.Equals(item.StateCode) &&
z.ChannelCode.Equals(item.ChannelCode) &&
SqlFunctions.DateDiff("DAY", z.EndDate, item.EndDate) == 0 &&
(z.ZipCode.Equals(null) || z.ZipCode.Equals(item.ZipCode)))
select z).ToList();
What I would like to do, however, is create a List of all the records on the table like this:
List<ZipCodeTerritory> allRecords = (from z in db.ZipCodeTerritory
select z).ToList()
and then use a query similar to this to pull the record I'm looking for from the list:
List<ZipCodeTerritory> previousZips = allRecords.Where(
z => (item.ZipCode.Equals(null)
? z.StateCode.Equals(item.StateCode) &&
z.ChannelCode.Equals(item.ChannelCode) &&
SqlFunctions.DateDiff("DAY", z.EndDate,
item.EndDate) == 0
: z.StateCode.Equals(item.StateCode) &&
z.ChannelCode.Equals(item.ChannelCode) &&
SqlFunctions.DateDiff("DAY", z.EndDate,item.EndDate) == 0 &&
(z.ZipCode.Equals(null) || z.ZipCode.Equals(item.ZipCode))
)
).ToList();
The query above (from the List), however, throws the following error:
This function can only be invoked from LINQ to Entities.

Answered my own question. The problem here was the SqlFunction. By removing that from the query and re-writing it like this it works
List<ZipCodeTerritory> previousZips = allRecords.Where(
z => (item.ZipCode.Equals(null)
? z.StateCode.Equals(item.StateCode) &&
z.ChannelCode.Equals(item.ChannelCode) &&
z.EndDate.Date == item.EndDate.Date
: z.StateCode.Equals(item.StateCode) &&
z.ChannelCode.Equals(item.ChannelCode) &&
z.EndDate.Date == item.EndDate.Date &&
(z.ZipCode.Equals(null) || z.ZipCode.Equals(item.ZipCode))
)
).ToList();

Instead of
SqlFunctions.DateDiff("DAY", z.EndDate,item.EndDate) == 0
use
z.EndDate.Subtract(item.EndDate).TotalDays == 0

How to improve Linq-To-Sql code

Using the StackExchange.Profiling.MiniProfiler class to profile an ASP.NET MVC application with Linq-To-Sql as ORM.
I'm trying to reduce one action to one SQL, so that I don't have any duplicates anymore.
So I changed my linq-to-sql code accordingly, but it didn't have any positive effect on the speed.
Then I checked the time that is needed for the SQL.
This shows the MiniProfiler:
When I fire up the exact same SQL in Management Studio it is super fast:
Here is the code:
from t in type
let tDoc = (from d in context.documents
where d.Key == t.No
&& d.RType == (int)RType.Art
&& d.AType == (int)AType.Doc
select d).FirstOrDefault(d => d.UseForThumb)
select new Time
{
Id = t.Id,
//... more simple mappings here
// then a complex one:
DocsCount = context.documents.Count(d =>
(d.Key == t.Id.ToString()
&& d.RType == (int)RType.Type
&& d.AType == (int)AType.Doc)
||
(d.Key == t.No
&& d.RType == (int)RType.Art
&& d.AType == (int)AType.Doc)),
// and another one
ThumbId = (tDoc != null && tDoc.FRKey.HasValue) ? tDoc.FRKey.Value : 0
};
What can be the reason for the huge difference? - Edit: There is no difference, I just misenterpreted SSMS :(
Anyway, my problem persits. What could I change to make it faster?
I read sometime that the mapping from Linq-To-Sql has a performance problem. Is there a way to workaround this?

I did some trial and error and changed the Linq-To-Sql code to this:
from t in types
let docs = context.documents.Where(d => (d.RKey == t.Id.ToString()
&& d.RType == (int)RType.Type
&& d.AType == (int)AType.Doc)
||
(d.RKey == t.No
&& d.RType == (int)RType.Art
&& d.AType == (int)AType.Doc))
let tDoc = docs.FirstOrDefault(d => d.RType == (int)RType.Art && d.UseForThumb)
let docsCount = docs.Count()
select new Time
{
Id = t.Id,
//... more simple mappings here
DocsCount = docsCount,
ThumbId = (tDoc != null && tDoc.FRKey.HasValue) ? tDoc.FRKey.Value : 0,
}
This made the query much, much faster.

Optimizing A LINQ To Objects Query

I'm trying to optimize the below LINQ query to improve it's speed performance. The number of objects it's searching against could be in the tens of thousands.
var lQuery = from o in oEvents
where (o.oSalesEvent != null && o.oSalesEvent.OccurDate < oCalcMgr.OccurDate && (
(oCalcMgr.InclTransTypes == Definitions.TransactionTypes.SalesAll) ?
(o.oSalesEvent.EventStateID == ApprovedID || o.oSalesEvent.EventStateID == PendingID) :
o.oSalesEvent.EventStateID == ApprovedID)) &&
((oCalcMgr.InclTransTypes == Definitions.TransactionTypes.SalesAll) ?
(o.oSalesMan.oEmployment.EventStateID == ApprovedID || o.oSalesMan.oEmployment.EventStateID == PendingID) :
o.oSalesMan.oEmployment.EventStateID == ApprovedID)
select new { SaleAmount = o.SaleAmount.GetValueOrDefault(), CompanyID = o.oSalesEvent.CompanyID };
The query basically says, give me the sales amounts and company ids from all sale events that occurred prior to a certain date. The sale event's status and the salesman's employment status should either always be "approved" or they can be also "pending" if specified.
As you can see there's a date comparison and a couple of integer comparisons. Which integer comparison used is based on whether or not a property matches a certain Enum value.
I have some ideas of my own on ways to go about the optimization, but I want to hear others thoughts, who might have more insight into how LINQ would translate this query behind the scenes.
Thanks

It seems to me that your biggest challenge is that you're doing multiple criteria checks in your Linq statement that will take alot of time.
What about creating a new property in oEvents - Say "IsEligable" and set it's value within the Set statements of your other variables (much faster than constant re-querying of each variable in the Linq).
Then, by the time you get to this part of your code, you could update your Linq to be something along the lines of:
var lQuery = from o in oEvents
where (o.oSalesEvent != null && o.oSalesEvent.OccurDate < oCalcMgr.OccurDate && o.IsEligable == True)
select new { SaleAmount = o.SaleAmount.GetValueOrDefault(), CompanyID = o.oSalesEvent.CompanyID };
... I'm guessing that would speed up the execution, but just a thought...

This is as much to improve readability as to possibly speed it up, but give this a try:
var lQueryTemp = from o in oEvents
where (o.oSalesEvent != null && o.oSalesEvent.OccurDate < oCalcMgr.OccurDate)
if (oCalcMgr.InclTransTypes == Definitions.TransactionTypes.SalesAll)
{
lQueryTemp = from o in lQueryTemp
where (o.oSalesEvent.EventStateID == ApprovedID || o.oSalesEvent.EventStateID == PendingID) &&
(o.oSalesMan.oEmployment.EventStateID == ApprovedID || o.oSalesMan.oEmployment.EventStateID == PendingID);
}
else
{
lQueryTemp = from o in lQueryTemp
where (o.oSalesEvent.EventStateID == ApprovedID && o.oSalesMan.oEmployment.EventStateID == ApprovedID);
}
var lQuery = from o in lQueryTemp
select new { SaleAmount = o.SaleAmount.GetValueOrDefault(), CompanyID = o.oSalesEvent.CompanyID };
This might speed it up by pulling out both checks of oCalcMgr.InclTransTypes, which is effectively a constant for purposes of this query.