SQL: Nested query does not have appropriate key

SQL: Nested query does not have appropriate key - c#

In LINQ, I am trying to inner join custom function written for full-text search and an Iqueryable result.
However, I get the following error when I try to to_ret.select(--something--).ToList()
Nested query does not have appropriate key
LINQ Code:
var sql_query = db.search(st);
var to_ret = from ts in sql_query
from t in table
where t.Id == ts.Value select t;
to_ret = to_ret.Include(x => x.table1)
.Include(x=> x.table2.Select(y=> y.table2Col));
to_ret.select(-something-).toList();
SQL Code:
create function [dbo].[search]
(#keywords nvarchar(4000))
returns table
as
return (
select [key] from containstable(tb,(Name,Description),#keywords)
)
Code that works in place of above LINQ Code :
var ids = (from t in table join ts in db.search(st) on t.Id equals ts.Value select t.Id).ToList();
to_ret = to_ret.Where(x => ids.Contains(x.Id));
However, the code that works isn't efficient enough as it eagerly loads all the ids for comparison

Do not join the table using LINQ, it is not effective.
You need to include all the joined functions into [dbo].[search] table valued function (like a view). Then do just call the [dbo].[search] from EF and filter it.
I have mentioned joined Fulltext table valued function here.
Note that fulltext and filtering together in one query could take time, because it is not easy job for query optimizer. Query optimizer selects to perform first fulltext on entire table(s) and then filtering or the opposite way.

Related

Using 'OR' inside a LinQ join query (adapting SQL into LinQ)

I have a SQL query that basically joins the MyWords table to the MyTranslations table on two keys. Here are my tables for better understanding.
MyWords table:
Id
WordString
LangType
MyTranslations table:
Id
WordColumnAId
WordColumnBId
Please note that WordColumnAIdand WordColumnBId columns are FKs that represents the MyWords.Id.
My SQL query:
SELECT MyTranslations.Id
,WordColumnAId
,WordColumnBId
,MyWords.WordString
,MyWords.LangType
FROM MyTranslations
join MyWords on (MyTranslations.WordColumnAId = MyWords.Id or MyTranslations.WordColumnBId = MyWords.Id)
where MyWords.LangType !=#currentLangType
I know it doesn't make sense to use join on with multiple keys at first glance but I'm trying to a cross-query that would join two tables whether the key is on WordColumnAId or WordColumnBId.
Problem
I'm trying to adapt the above T-SQL query into a LinQ query. The problem is that I can't find my way around LinQ to use two keys in a single join query.
Here's what I've got so far:
from translation in queryableTranslation
join word in _myWordRepository on translation.WordColumnAId equals word.Id // This is where I want to add `or translation.WordColumnBId equals word.Id` but get errors.
where word.LangType != currentLangType
select new QueryResultDto {
MyTranslationId = translation.Id,
WordString = word.WordString,
LanguageType = word.LangType,
WordColumnAId = translation.WordColumnAId,
WordColumnbId=translation.WordColumnbId,
};
I'm very new to the LinQ and trying to learn.
So my question: is there a way to achieve this in LinQ or am I trying the impossible? I'm also open to better approaches.

EF and other LINQ providers should translate query to INNER JOIN when using this syntax:
var query =
from translation in queryableTranslation
from word in _myWordRepository.Where(word => translation.WordColumnAId == word.Id
|| translation.WordColumnBId = word.Id)
where word.LangType != currentLangType
select new QueryResultDto
{
MyTranslationId = translation.Id,
WordString = word.WordString,
LanguageType = word.LangType,
WordColumnAId = translation.WordColumnAId,
WordColumnbId=translation.WordColumnbId,
};

Is there any way to make this query faster and build where clause outside of loop?

I have the following code, (and I am completely aware about parameterized queries and SQL Injection):
foreach(var item in items)
{
string query = "select sum(convert(decimal(18,3),tbl.Price)) p, sum(convert(decimal(18,2),tbl.Sale)) s from table1 tbl " +
$"where tbl.ID = {item .ID}";
Execute(query);
//Do stuff with query result
}
The problem is I have a lot of items and I have to execute the query for each of the items because the where clause will be complete in each step. I think if I will be able to make my query out side of my loop, my query will be faster. But I don't know how. Is there any way to do this?

Instead of executing the query for every item. You can add group by to your query and execute only once.
string query = "select tbl.ID, sum(convert(decimal(18,3),tbl.Price)) p, sum(convert(decimal(18,2),tbl.Sale)) s from table1 tbl group by tbl.ID ";
var result = Execute(query);
foreach(var item in items)
{
var row = result.Select(r => r.ID == item.ID).FirstOrDefault();
//Do stuff with query result
}

Do not execute the query for each ID separately. Instead, execute a single query for all Ids using group by to get the p and s values for each id and a parameterized in clause (or better yet, a stored procedure with a table valued parameter).
Here is the IN version of the query:
select Id,
sum(convert(decimal(18,3),tbl.Price)) p,
sum(convert(decimal(18,2),tbl.Sale)) s
from table1 tbl
Where Id IN(<1,2,3,4....>)
group by Id
Replace <1,2,3,4....> with parameters like described in this answer.
Here is the table valued parameter version of the query:
select tbl.Id,
sum(convert(decimal(18,3),tbl.Price)) p,
sum(convert(decimal(18,2),tbl.Sale)) s
from table1 tbl
inner join #items i on tbl.Id = i.Id
group by tbl.Id
For a detailed explanation about using table valued parameters, read this answer.

Entity Framework select most recent record for each type of record

I have a table called values that looks like this:
+-------+------------+-----------+----------+
|Id |DateTime |SensorId |Value |
+-------+------------+-----------+----------+
SensorId is a foreign key to a table of the sensor details. There will be 10m+ records in this values table.
I can run this sql command to return the most recent record for each SensorId and it runs in about 0.3 seconds.
SELECT a.*
FROM Values as a
INNER JOIN (
SELECT SensorId, MAX(ID) maxId
FROM Values
GROUP BY SensorId
) b ON a.SensorId = b.SensorId
AND a.Id = b.maxId
ORDER BY a.SensorId ASC
How can I achieve the same output with entity framework in a c# application while maintaining (or improving) the performance?

With LINQ to Entities and lambdas you can do it like this:
dataContext.Values.GroupBy(p => p.SensorId)
.Select(p => p.FirstOrDefault(w => w.Id == p.Max(m => m.Id)))
.OrderBy(p => p.SensorId).ToList()
where dataContext is your instance of ObjectContext class. ToList() compiles the query.

I guess it would not be possible reach better performance than pure SQl query because by using EF you are adding abstraction layer to the process.
EF is usually very slow with GroupBy command. I suggest try sql query in EF directly
(this code is for EF Core)
context.Values.FromSqlRaw<Values>("SELECT a.*
FROM Values as a
INNER JOIN (
SELECT SensorId, MAX(ID) maxId
FROM Values
GROUP BY SensorId
) b ON a.SensorId = b.SensorId
AND a.Id = b.maxId
ORDER BY a.SensorId ASC").ToList<Values>();
FromSqlRaw is faster than a normal Linq Query. for EF you can try context.ExecuteQuery<Values> by the same way.

Using IQueryable with Linq

What is the use of IQueryable in the context of LINQ?
Is it used for developing extension methods or any other purpose?

Marc Gravell's answer is very complete, but I thought I'd add something about this from the user's point of view, as well...
The main difference, from a user's perspective, is that, when you use IQueryable<T> (with a provider that supports things correctly), you can save a lot of resources.
For example, if you're working against a remote database, with many ORM systems, you have the option of fetching data from a table in two ways, one which returns IEnumerable<T>, and one which returns an IQueryable<T>. Say, for example, you have a Products table, and you want to get all of the products whose cost is >$25.
If you do:
IEnumerable<Product> products = myORM.GetProducts();
var productsOver25 = products.Where(p => p.Cost >= 25.00);
What happens here, is the database loads all of the products, and passes them across the wire to your program. Your program then filters the data. In essence, the database does a SELECT * FROM Products, and returns EVERY product to you.
With the right IQueryable<T> provider, on the other hand, you can do:
IQueryable<Product> products = myORM.GetQueryableProducts();
var productsOver25 = products.Where(p => p.Cost >= 25.00);
The code looks the same, but the difference here is that the SQL executed will be SELECT * FROM Products WHERE Cost >= 25.
From your POV as a developer, this looks the same. However, from a performance standpoint, you may only return 2 records across the network instead of 20,000....

In essence its job is very similar to IEnumerable<T> - to represent a queryable data source - the difference being that the various LINQ methods (on Queryable) can be more specific, to build the query using Expression trees rather than delegates (which is what Enumerable uses).
The expression trees can be inspected by your chosen LINQ provider and turned into an actual query - although that is a black art in itself.
This is really down to the ElementType, Expression and Provider - but in reality you rarely need to care about this as a user. Only a LINQ implementer needs to know the gory details.
Re comments; I'm not quite sure what you want by way of example, but consider LINQ-to-SQL; the central object here is a DataContext, which represents our database-wrapper. This typically has a property per table (for example, Customers), and a table implements IQueryable<Customer>. But we don't use that much directly; consider:
using(var ctx = new MyDataContext()) {
var qry = from cust in ctx.Customers
where cust.Region == "North"
select new { cust.Id, cust.Name };
foreach(var row in qry) {
Console.WriteLine("{0}: {1}", row.Id, row.Name);
}
}
this becomes (by the C# compiler):
var qry = ctx.Customers.Where(cust => cust.Region == "North")
.Select(cust => new { cust.Id, cust.Name });
which is again interpreted (by the C# compiler) as:
var qry = Queryable.Select(
Queryable.Where(
ctx.Customers,
cust => cust.Region == "North"),
cust => new { cust.Id, cust.Name });
Importantly, the static methods on Queryable take expression trees, which - rather than regular IL, get compiled to an object model. For example - just looking at the "Where", this gives us something comparable to:
var cust = Expression.Parameter(typeof(Customer), "cust");
var lambda = Expression.Lambda<Func<Customer,bool>>(
Expression.Equal(
Expression.Property(cust, "Region"),
Expression.Constant("North")
), cust);
... Queryable.Where(ctx.Customers, lambda) ...
Didn't the compiler do a lot for us? This object model can be torn apart, inspected for what it means, and put back together again by the TSQL generator - giving something like:
SELECT c.Id, c.Name
FROM [dbo].[Customer] c
WHERE c.Region = 'North'
(the string might end up as a parameter; I can't remember)
None of this would be possible if we had just used a delegate. And this is the point of Queryable / IQueryable<T>: it provides the entry-point for using expression trees.
All this is very complex, so it is a good job that the compiler makes it nice and easy for us.
For more information, look at "C# in Depth" or "LINQ in Action", both of which provide coverage of these topics.

Although Reed Copsey and Marc Gravell already described about IQueryable (and also IEnumerable) enough,mI want to add little more here by providing a small example on IQueryable and IEnumerable as many users asked for it
Example: I have created two table in database
CREATE TABLE [dbo].[Employee]([PersonId] [int] NOT NULL PRIMARY KEY,[Gender] [nchar](1) NOT NULL)
CREATE TABLE [dbo].[Person]([PersonId] [int] NOT NULL PRIMARY KEY,[FirstName] [nvarchar](50) NOT NULL,[LastName] [nvarchar](50) NOT NULL)
The Primary key(PersonId) of table Employee is also a forgein key(personid) of table Person
Next i added ado.net entity model in my application and create below service class on that
public class SomeServiceClass
{
public IQueryable<Employee> GetEmployeeAndPersonDetailIQueryable(IEnumerable<int> employeesToCollect)
{
DemoIQueryableEntities db = new DemoIQueryableEntities();
var allDetails = from Employee e in db.Employees
join Person p in db.People on e.PersonId equals p.PersonId
where employeesToCollect.Contains(e.PersonId)
select e;
return allDetails;
}
public IEnumerable<Employee> GetEmployeeAndPersonDetailIEnumerable(IEnumerable<int> employeesToCollect)
{
DemoIQueryableEntities db = new DemoIQueryableEntities();
var allDetails = from Employee e in db.Employees
join Person p in db.People on e.PersonId equals p.PersonId
where employeesToCollect.Contains(e.PersonId)
select e;
return allDetails;
}
}
they contains same linq. It called in program.cs as defined below
class Program
{
static void Main(string[] args)
{
SomeServiceClass s= new SomeServiceClass();
var employeesToCollect= new []{0,1,2,3};
//IQueryable execution part
var IQueryableList = s.GetEmployeeAndPersonDetailIQueryable(employeesToCollect).Where(i => i.Gender=="M");
foreach (var emp in IQueryableList)
{
System.Console.WriteLine("ID:{0}, EName:{1},Gender:{2}", emp.PersonId, emp.Person.FirstName, emp.Gender);
}
System.Console.WriteLine("IQueryable contain {0} row in result set", IQueryableList.Count());
//IEnumerable execution part
var IEnumerableList = s.GetEmployeeAndPersonDetailIEnumerable(employeesToCollect).Where(i => i.Gender == "M");
foreach (var emp in IEnumerableList)
{
System.Console.WriteLine("ID:{0}, EName:{1},Gender:{2}", emp.PersonId, emp.Person.FirstName, emp.Gender);
}
System.Console.WriteLine("IEnumerable contain {0} row in result set", IEnumerableList.Count());
Console.ReadKey();
}
}
The output is same for both obviously
ID:1, EName:Ken,Gender:M
ID:3, EName:Roberto,Gender:M
IQueryable contain 2 row in result set
ID:1, EName:Ken,Gender:M
ID:3, EName:Roberto,Gender:M
IEnumerable contain 2 row in result set
So the question is what/where is the difference? It does not seem to
have any difference right? Really!!
Let's have a look on sql queries generated and executed by entity
framwork 5 during these period
IQueryable execution part
--IQueryableQuery1
SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[Gender] AS [Gender]
FROM [dbo].[Employee] AS [Extent1]
WHERE ([Extent1].[PersonId] IN (0,1,2,3)) AND (N'M' = [Extent1].[Gender])
--IQueryableQuery2
SELECT
[GroupBy1].[A1] AS [C1]
FROM ( SELECT
COUNT(1) AS [A1]
FROM [dbo].[Employee] AS [Extent1]
WHERE ([Extent1].[PersonId] IN (0,1,2,3)) AND (N'M' = [Extent1].[Gender])
) AS [GroupBy1]
IEnumerable execution part
--IEnumerableQuery1
SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[Gender] AS [Gender]
FROM [dbo].[Employee] AS [Extent1]
WHERE [Extent1].[PersonId] IN (0,1,2,3)
--IEnumerableQuery2
SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[Gender] AS [Gender]
FROM [dbo].[Employee] AS [Extent1]
WHERE [Extent1].[PersonId] IN (0,1,2,3)
Common script for both execution part
/* these two query will execute for both IQueryable or IEnumerable to get details from Person table
Ignore these two queries here because it has nothing to do with IQueryable vs IEnumerable
--ICommonQuery1
exec sp_executesql N'SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[LastName] AS [LastName]
FROM [dbo].[Person] AS [Extent1]
WHERE [Extent1].[PersonId] = #EntityKeyValue1',N'#EntityKeyValue1 int',#EntityKeyValue1=1
--ICommonQuery2
exec sp_executesql N'SELECT
[Extent1].[PersonId] AS [PersonId],
[Extent1].[FirstName] AS [FirstName],
[Extent1].[LastName] AS [LastName]
FROM [dbo].[Person] AS [Extent1]
WHERE [Extent1].[PersonId] = #EntityKeyValue1',N'#EntityKeyValue1 int',#EntityKeyValue1=3
*/
So you have few questions now, let me guess those and try to answer them
Why are different scripts generated for same result?
Lets find out some points here,
all queries has one common part
WHERE [Extent1].[PersonId] IN (0,1,2,3)
why? Because both function IQueryable<Employee> GetEmployeeAndPersonDetailIQueryable and
IEnumerable<Employee> GetEmployeeAndPersonDetailIEnumerable of SomeServiceClass contains one common line in linq queries
where employeesToCollect.Contains(e.PersonId)
Than why is the
AND (N'M' = [Extent1].[Gender]) part is missing in IEnumerable execution part, while in both function calling we used Where(i => i.Gender == "M") inprogram.cs`
Now we are in the point where difference came between IQueryable and
IEnumerable
What entity framwork does when an IQueryable method called, it tooks linq statement written inside the method and try to find out if more linq expressions are defined on the resultset, it then gathers all linq queries defined until the result need to fetch and constructs more appropriate sql query to execute.
It provide a lots of benefits like,
only those rows populated by sql server which could be valid by the
whole linq query execution
helps sql server performance by not selecting unnecessary rows
network cost get reduce
like here in example sql server returned to application only two rows after IQueryable execution` but returned THREE rows for IEnumerable query why?
In case of IEnumerable method, entity framework took linq statement written inside the method and constructs sql query when result need to fetch. it does not include rest linq part to constructs the sql query. Like here no filtering is done in sql server on column gender.
But the outputs are same? Because 'IEnumerable filters the result further in application level after retrieving result from sql server
SO, what should someone choose?
I personally prefer to define function result as IQueryable<T> because there are lots of benefit it has over IEnumerable like, you could join two or more IQueryable functions, which generate more specific script to sql server.
Here in example you can see an IQueryable Query(IQueryableQuery2) generates a more specific script than IEnumerable query(IEnumerableQuery2) which is much more acceptable in my point of view.

It allows for further querying further down the line. If this was beyond a service boundary say, then the user of this IQueryable object would be allowed to do more with it.
For instance if you were using lazy loading with nhibernate this might result in graph being loaded when/if needed.

NHibernate: hql to criteria query - help needed

I have this hql query, which works perfect:
select m
from Media m
join m.Productlines p
join m.Categories c
join m.Spaces sp
join m.Solutions so
where m.Uid != 0
and p.Uid in (:productlines)
and c.Uid in (13)
and sp.Uid in (52)
and so.Uid in (15,18)
group by m.Uid
But now it needs to be parameterized/made dynamic, not only the parameters, but also the joins (it is possible to select only from Media, without any joins, and so no *.Uid in will be required in this case).
I dont want to mess around with a StringBuilder instance and build the hql query that way, I would rather like to use the Criteria API, but I cant get a
SELECT m.*
....
GROUP BY m.Uid
query to work with Criteria.
If I add a
Projections.GroupProperty("Uid")
to my query, nhibernate selects
SELECT m.Uid
....
GROUP BY m.Uid
which is of course wrong.
After that, I also need to count the unique rows the query returned, as the result is paged.
So, my other query is quite similiar, but I cant find a Criteria equivalent for
SELECT COUNT(DISTINCT m.Uid)
Here is the HQL:
select count(distinct m.Uid)
from Media m
join m.Productlines p
join m.Categories c
join m.Spaces sp
join m.Solutions so
where m.Uid != 0
and p.Uid in (:productlines)
and c.Uid in (13)
and sp.Uid in (52)
and so.Uid in (15,18)
How can this be done with Criteria API?
Please, (N)Hibernate experts - help me with this, I cant find a working solution. Any help is greatly appreciated!

Group columns are implicitly returned as result, but you can add more columns. AFAIK, you can return full entities:
var query = session.CreateCriteria(typeof(Media), "m")
.Add(Projections.GroupProperty("m"))
.Add(Restrictions.NotEq("m.Uid", 0));
// dynamically add filters
if (filterProductLines)
{
query
.CreateCriteria("m.Productlines", "p")
.Add(Restrictions.Eq("p.Uid", productLines));
}
// more dynamic filters of this kind follow here...
IList<Media> results = query.List<Media>();
To count the full number of results you can just build up the same query with different projection:
var query = session.CreateCriteria(typeof(Media), "m")
.SetProjection(Projections.CountDistinct("m.Uid"));
// rest of the query the same way as above
long totalNumberOfResults = query.UniqueResult<long>();
I'm getting unsure about the Projections.GroupProperty("m"), you need to try this. If it doesn't work, you could make it an DetachedQuery that only returns ids:
var subquery = DetachedCriteria.For(typeof(Media), "m")
.Add(Projections.GroupProperty("m.Uid"))
.Add(Restrictions.NotEq("m.Uid", 0));
// add filtering
var query = session.CreateCriteria(typeof(Media), "outer")
.Add(Subqueries.PropertyIn("outer.Uid", subquery));
IList<Media> results = query.List<Media>();
This creates a sql query like this:
select outer.* // all properties of Media to create an instance
from Media outer
where outer.Uid in (
select Uid
from media m
where // filter
)

var count = session.CreateCriteria(typeof(Media))
// Add other criterias...
.SetProjection(Projections.CountDistinct("Id")) // or whatever the id property of Media class is called
.UniqueResult<long>();
As to your GROUP BY question, the query:
SELECT m.*
....
GROUP BY m.Uid
makes no sense because you need to select only columns that appear in the group by clause or aggregate functions. Could you elaborate a little more as to what exactly are you trying to achieve?

Develop Reference

C# (C-Sharp) is a programming language developed by Microsoft that runs on the .NET Framework.