Could someone help me with this exception? I don't understand what it means or how to fix it... It is an SqlException with the following message:
All queries combined using a UNION, INTERSECT or EXCEPT operator must have an equal number of expressions in their target lists.
I get it when running a query in pseudo code looking like this:
// Some filtering of data
var query = data.Subjects
.Where(has value)
.Where(has other value among some set of values);
// More filtering, where I need to have two different options
var a = query
.Where(some foreign key is null);
var b = query
.Where(some foreign key is not null)
.Where(and that foreign key has a property which is what I want);
query = a.Union(b);
// Final filter and then get result as a list
var list = query
.Where(last requirement)
.ToList();
If I remove the a.Union(b) parts, it runs without the exception. So I know the error is there. But why do I get it? And how can I fix it? Am I doing something too crazy here? Have I misunderstood how to use the Union thing?
Basically what I have is some entities which have a foreign key to some other entity. And I need to get all the entities which either have that foreign key set to null or where that foreign entity fulfills some requirements.
Judging from the SQL error you listed you may be experiencing the same issue I was. Basically when Linq to SQL queries that use the Concat or Union extension method on two different queries it appears that there is a bug in Linq to SQL which optimizes each projection separately without regard to the fact that the projection must stay the same in order to accomplish the SQL Union.
References:
LINQ to SQL produces incorrect TSQL when using UNION or CONCAT
Linq to SQL Union Same Fieldname generating Error
If this happens to be your problem as well I've found a solution that is working for me as shown below.
var queryA =
from a in context.TableA
select new
{
id,
name,
onlyInTableA,
}
var queryB =
from b in context.TableB
let onlyInTableA = default(string)
select new
{
id,
name,
onlyInTableA,
}
var results = queryA.Union(queryB).ToList();
Since this looks like a problem with the generated SQL, you should try to use either an SQL Profiler, or use this code for DebuggerWritter class to write the SQL to your Output Window in Visual Studio.
The SQL error is normally caused by the fields retrieved for UNION is not the same for the 2 queries. For example, if the first query might have 3 fields, but the second query has 4 fields, this error will occur. So, seeing the generated SQL will definitely help in this case.
Can you perhaps write it in a single query?
.Where(row => row.ForeignKey == null || row.ForeignKey.SomeCondition);
There are also ways of merging expressions (OrElse), but that isn't trivial.
Not sure where the error comes from, though!
edit: haven't tested it, but this should be logically equivalent to a UNION:
public static IQueryable<T> WhereAnyOf<T>(
this IQueryable<T> source,
params Expression<Func<T, bool>>[] predicates)
{
if (source == null) throw new ArgumentNullException("source");
if (predicates == null) throw new ArgumentNullException("predicates");
if (predicates.Length == 0) return source.Where(row => false);
if (predicates.Length == 1) return source.Where(predicates[0]);
var param = Expression.Parameter(typeof(T), "row");
Expression body = Expression.Invoke(predicates[0], param);
for (int i = 1; i < predicates.Length; i++)
{
body = Expression.OrElse(body,
Expression.Invoke(predicates[i], param));
}
return source.Where(Expression.Lambda<Func<T, bool>>(body, param));
}
query = a.Union(b);
Not a good idea to mutate captured variables... Likely the cause of the error.
UPDATE: ok not
Here is another idea. The hint is in the error message.
var a = query
.Where(some foreign key is null)
.Select(x => x);
Or play by adding another 'fake' Where till they do become equal :)
I would call data.GetCommand(query) and analyze the resulting DbCommand (especially the generated SQL string). That should give you a clue to what goes wrong.
There is no projection going on anywhere so I would expect both target lists to be the same.
You could try to reduce your query to a smaller one that still doesn't work. Start with query.Union(query) (this should at least work). Than add your Where calls one by one to see when it stops working.
It must be one of your Where calls that adds extra columns to your select list.
Are you by any chance passing in a value to the 'select' side in a variable, or are you returning the same field more than once? SP1 introduced a bug where it tries to 'optimize' out such things and that can cause union queries to break (due to the query parts 'optimizing' out different passed-in params).
If you post your actual query rather than pseudo code it makes it easier to identify if this is the case.
(And a workaround if this is the case is to materialize the individual parts first and then do a client-side (L2O) union).
jpierson has the problem summarised correctly.
I also had the problem, this time caused by some literals in the select statement:
Dim results = (From t in TestDataContext.Table1 _
Where t.ID = WantedID _
Select t.name, SpecialField = 0, AnotherSpecialField = 0, t.Address).Union _
From t in TestDataContext.Table1 _
Where t.SecondID = WantedSecondID _
Select t.name, SpecialField = 1, AnotherSpecialField = 0, t.Address)
The first sub-query of "SpecialField = 0" and the "AnotherSpecialField = 0" were optimised, resulting in one field instead of two being used in the union, which will obviously fail.
I had to change the first query so that the SpecialField & AnotherSpecialField had different values, much like in the second sub-query.
Well I had an issue with this. Using Sql 08 i had two table functions that returned an int and a string in both cases. I created a complex object and used linq to attempt a UNION. Had an IEqualityComparer to do the comparision. All compiled fine, but crashed with a unsupported overload. Ok, i realised the problem discussed seemed to smack of defered execution. So i get the collections, and place ToList(), then do the UNION and it is all good. Not sure if this is helpful, but it works for me
Related
I have the following query:
var vendors = (from pp in this.ProductPricings
join pic in this.ProductItemCompanies
on pp.CompanyId equals pic.CompanyId into left
from pic in left.DefaultIfEmpty()
orderby pp.EffectiveDate descending
group pp by new { pp.Company, SortOrder = (pic != null) ? pic.SortOrder : short.MinValue } into v
select v).OrderBy(z => z.Key.SortOrder);
Does anyone know how the last OrderBy() is applied? Does that become part of the SQL query, or are all the results loaded in to memory and then passed to OrderBy()?
And if it's the second case, is there any way to make it all one query? I only need the first item and it would be very inefficent to return all the results.
Well it will try to apply the OrderBy to the original query since you are still using an IQueryable - meaning it hasn't been converted to an IEnumerable or hydrated to a collection using ToList or an equivalent.
Whether it can or not depends on the complexity of the resulting query. You'd have to try it to find out. My guess is it will turn the main query into a subquery and layer on a "SELECT * FROM (...) ORDER BY SortOrder" outer query.
Given your specific example the order by in this situation most, likely be appliead as part of the expression tree when it getting build, there for it will be applied to sql generated by the LINQ query, if you would convert it to Enumarable like ToList as mentioned in another answer then Order by would be applied as an extension to Enumerable.
Might use readable code, because as you write it is not understandable.
You will have a problem in the future with the linq statement. The problem is that if your statement does not return any value the value will be null and whenever you make cause a exception.
You must be careful.
I recommend you to do everything separately to understand the code friend.
Ok, I must be doing something dumb, but shouldn't this work? I have following three lists:
var commonViews = (from v in context.TPM_VIEWS where v.VIEWID < 0 select v); // IQueryable<TPM_VIEWS>
var ownedViews = (from v in context.TPM_VIEWS where v.OWNERID == userId && v.VIEWID > 0 select v); // IQueryable<TPM_VIEWS>
var sharedViews = (from v in context.TPM_USER.Include("TPM_VIEWS2") where v.USERID == userId select v).First().TPM_VIEWS2; // EntityCollection<TPM_VIEWS>
Each list has the proper values and count. I can return any one of these lists:
return commonViews.ToList();
And I can return a any two of these lists:
return commonViews.Concat(ownedViews).ToList();
However, when I try to return all three:
return commonViews.Concat(ownedViews).Concat(sharedViews).ToList();
I get the exception:
Unable to create a constant value of type 'Entity.TPM_VIEWS'. Only
primitive types or enumeration types are supported in this context.
What am I doing wrong? All three values are indeed enumerable. Mostly, I'm asking this question because it's the best possible way to guarantee I'll notice the problem 30 seconds after posting.
UPDATE:
I'm 93% sure the problem is here:
var sharedViews = (from v in context.TPM_USER.Include("TPM_VIEWS2") where v.USERID == userId select v).First().TPM_VIEWS2;
This looks like an enumerable list of TPM_VIEWS object, and I can call ToList() on it and get the correct data, but it doesn't play well with the other lists.
UPDATE 2:
This actually works. Points to the person who can tell me why!
commonViews.ToList().Concat(ownedViews.ToList()).Concat(sharedViews.ToList()).ToList();
The problem is that Concat() on an EF IQueryable<T> will turn the entire concatenation into a single query.
When you call .Concat(sharedViews), you're passing a scalar (pre-loaded) collection of your nested entity class.
EF doesn't know how to convert that into a query, so it complains.
You can make it faster by calling AsEnumerable() instead of ToList().
This actually works. Points to the person who can tell me why!
commonViews.ToList().Concat(ownedViews.ToList()).Concat(sharedViews.ToList()).ToList();
That's because each of the original queries is executed separately; you're only concatenating the results in memory. There seems to be a bug in the Entity Framework query translator when you combine the 3 queries, but when you call ToList on each of them, they're no longer EF queries, they're just lists, so they're concatenated using Linq to Objects.
I have the following LINQ code used in a unified search function.
var searchObjects =
from objectA in this.context.DB.objectAs
join objectB in this.context.DB.objectBs on objectA equals objectB.objectA into objectAB
from AB in objectAB.Where(o => o.Type == "BasicGroup").DefaultIfEmpty()
select new { objectA, objectB = AB};
foreach (var searchWord in searchWords)
{
var searchObjects =
searchObjects.Where(p => p.objectA.Name.Contains(searchWord) ||
(p.objectB != null &&
(p.objectB.Name.Contains(searchWord) ||
p.objectB.ID.contains(searchWord))));
}
The goal is to look for the search words in objectA's Name field, or in the associated objectB's Name or ID fields. The problem is that when I try to enumerate searchObjects, it just returns a NullReferenceException, with no further details.
The first part of the query returns a proper list of the combination objects (without any filters), so I don't think the problem is with the left join?
I can't figure out what is causing the exception, so any help would be appreciated.
I am also using Telerik's OpenAccess ORM, but I don't think that should be causing any problems here?
EDIT:
Turns out this was an issue with th Telerik OpenAccess ORM, which under certain conditions would just give up on producing sensible SQL, draw everything into memory and treat it as L2Objects (which should fail on nulls, as pointed out by #Dead.Rabit). The condition that seemed at least part of the problem was the .Where(o => o.Type == "BasicGroup") in front of the .DefaultIfEmpty(). Updating to the latest version of OpenAccess (2013 Q1 SPI I think it was) allows me to rewrite that condition as part of the equals statement
on new { objectA.ID, Type = "BasicGroup" } equals new { ID = objectB.AID, Type = object.Type }
This wasn't possible before the SP1. With this new query, I am able to compose the searchword Where clauses into the query and still have it produce SQL rather than drawing it into memory.
AB can be null since your creating it with DefaultIfEmpty() function, but also if ObjectA.Name can be null; Contains() in your second statement will throw an error. Similarly if ObjectB.ID or ObjectB.Name can be null when ObjectB is not null then your ObjectB != null guard clause won't have the desired effect.
Try a simple enumaration of your objects to ensure the issue is with your first query
foreach( var item in searchObjects )
Console.WriteLine( item.Type );
Your redefining the searchObjects variable every itteration which I assume is something lost in the code-dump to SO, but worth a mention JIC.
Finally I'm not sure about the OpenAccess ORM since I've never used it, but this query would fail on the Dynamics CRM implementation of LINQ which doesn't support certain types of conditions in where clauses (I won't go into details because it's irrelevant, but you've broken all of them in this query :p). It's definitely worth browsing the documentation for gotcha's in a 3rd party LINQ implementation.
I'm not 100% sure but i think you need to use AB instead of objectB in your anonymous type.
select new { objectA, AB }
because thats the result of the left join.
Problem statement
Say I have a query which searches the names of people:
var result = (from person in container.people select person)
.Where(p => p.Name.Contains(some_criterion)
This will be translated to a SQL query containing the following like clause:
WHERE NAME LIKE '%some_criterion%'
This has some performance implications, as the database is unable to effectively use the index on the name column (index scan v.s. index seek if I'm not mistaken).
To remedy this, I can decide to just StartsWith() instead, generating a query using a like clause like:
WHERE NAME LIKE 'some_criterion%'
Which enables SQL server to use the index seek and delivering performance at the cost of some functionality.
I'd like to be able to provide the user with a choice: defaulting the behavior to use StartsWith, but if the user want the 'added flexibility' of searching using Contains(), than that should used.
What have I tried
I thought this to be trivial and went on and implemented an extension method on string. But of course, LINQ does not accept this and an exception is thrown.
Now, of course I can go about and use an if or switch statement and create a query for each of the cases, but I'd much rather solve this 'on a higher level' or more generically.
In short: using an if statement to differentiate between use cases isn't feasible due to the complexity of the real-life application. This would lead to alot of repetition and clutter. I'd really like to be able to encapsulate the varying behavior (Contains, StartsWith, EndsWith) somehow.
Question
Where should I look or what should I look for? Is this a case for composability with IQueryables? I'm quite puzzled!
Rather than overcomplicate things, how about just using an if statement?
var query = from person in container.people
select person;
if (userWantsStartsWith)
{
query = from p in query
where p.Name.Contains(some_criterion)
select p;
}
else
{
query = from p in query
where p.Name.StartsWith(some_criterion)
select p;
}
Update
If you really need something more complex try looking at LinqKit. It allows you to do the following.
var stringFunction = Lambda.Expression((string s1, string s2) => s1.Contains(s2));
if (userWantsStartsWith)
{
stringFunction = Lambda.Expression((string s1, string s2) => s1.StartsWith(s2));
}
var query = from p in container.people.AsExpandable()
where stringFunction.Invoke(p.Name, some_criterion)
select p;
I believe this fulfils your requirement of
I'd really like to be able to encapsulate the varying behavior
(Contains, StartsWith, EndsWith) somehow.
You can dynamically alter the query before enumerating it.
var query = container.people.AsQueryable();
if (contains)
{
query = query.Where(p => p.Name.Contains(filter));
}
else
{
query = query.Where(p => p.Name.StartsWith(filter));
}
Try dis:
var result = (from person in container.people select person)
.Where(p => some_bool_variable ? p.Name.Contains(some_criterium) : p.Name.StartsWith(some_criterium));
The real life queries are quite huge and unioned with several others. This, like my problem states, isn't te solution I'm looking for
Sinse your queries are huge: can't you just define stored procedure that handles everything and call it with specific to query parameters (probably several stored procedures that are called by main, e.g. one of em searches by Name, another - by Age, have different sort order and so on to keep code clear)?
I have an IQueryable that has a list of pages.
I want to do: Pages.OrderByDescending(o => CalculateSort(o.page));
the method calculate sort is similar to that here is a plain english version:
public int calculatesort(page p)
{
int rating = (from r in db.rating select r). sum();
int comments = //query database for comments;
float timedecayfactor = math.exp(-page.totalhoursago);
return sortscore = (rating +comments)* timedecayfactor;
}
when I run a code similar to the one above an error is thrown that the mothode calculatesort cannot be converted to sql.
How can I do a conver the function above to be understood by sql so that I can use it to sort the pages?
Is this not a good approach for large data? Is there another method used to sort sets of results other than dynamically at the database?
I havent slept for days trying to fix this one :(
your code is nowhere near compiling so I'm guessing a lot here but I hope this gives an idea none the less.
As several have posted you need to give Linq-2-Sql an expression tree. Using query syntax that's what happens (by compiler magic)
from p in pages
let rating = (from r in db.rating
where r.PageId == p.PageId
select r.Value).Sum()
let comments = (from c in db.Comments
where c.PageId == p.PageId
select 1).Count()
let timedecayfactor = Math.Exp(-(p.totalhoursago))
orderby (rating + comments)*timedecayfactor descending
select p;
I haven't actually tried this against a database, there's simply too many unknown based on your code, so there might still be stuff that can't be translated.
The error occurs because LINQ cannot convert custom code/methods into SQL. It can convert only Expression<Func<>> objects into SQL.
In your case, you have a complex logic to do while sorting, so it might make sense to do it using a Stored Procedure, if you want to do it in the DB Layer.
Or load all the objects into main memory, and run the calculate sort method on the objects in memory
EDIT :
I don't have the code, so Describing in english is the best I can do :
Have table with structure capable of temporarily storing all the current users data.
Have a calculated field in the Pages table that holds the value calculated from all the non-user specific fields
Write a stored procedure that uses values from these two sources (temp table and calc field) to actually do the sort.
Delete the temp table as the last part in the stored proc
You can read about stored procs here and here
var comments = db.comments.Where(...);
Pages.OrderByDescending(p=>(db.rating.Sum(r=>r.rate) + comments.Count()) * Math.Exp(-p.totalhoursago))
Linq is expecting Calculatesort to return a "queryable" expression in order to generate its own SQL.
In can embed your 'calculatesort' method in this lambda expression. (I replaced your variables with constants in order to compile in my environment)
public static void ComplexSort(IQueryable<string> Pages)
{
Pages.OrderByDescending(p =>
{
int rating = 99;//(from r in db.rating select r). sum();
int comments = 33;//query database for comments;
double timedecayfactor = Math.Exp(88);
return (rating + comments) * timedecayfactor;
});
}
Also, you can even try to run that in parallel (since .net 4.0) replacing the first line with
Pages.AsParallel().OrderByDescending(p =>
Yes, counting previous answers: the LINQ to SQL doesn't know how to translate CalculateSort method. You should convert LINQ to SQL to ordinary LINQ to Object before using custom method.
Try to use this in the way you call the CalculateSort by adding AsEnumerable:
Pages.AsEnumerable().OrderByDescending(o => CalculateSort(o.page));
Then you're fine to use the OrderByDescending extension method.
UPDATE:
LINQ to SQL will always translate the query in the code into Expression tree. It's quite almost the same concept as AST of any programming language. These expression trees are further translated into SQL expression specific to SQL Server's SQL, because currently LINQ to SQL only supports SQL Server 2005 and 2008.