Let's say I have a the following query:
int x = 5;
var result = from p in db.products
where p.CategoryId == x
select p;
int count = result.Count();
List<product> products = result.ToList();
That's what I have now. But aditionally I need to have a DataReader from result:
// that's what I need:
var reader = ConvertSubSonicLinqQueryToDataReader(result);
How can I convert the linq statement to something I can work with?
A DataReader or a DbCommand or even plain sql with a list of paramters.
I know SubSonic can do that (since it translates the query to plain sql anyway) but I haven't found anything in the public accessible methods yet.
Any suggestions?
Converting the LINQ query is the wrong approach. LINQ returns results at a level of abstraction higher than a DataReader works at.
There's also the issue of deferred execution so your LINQ query may not be executed as a single SQL statement anyway.
Rater than use a LINQ statement why not just use an SqlQuery instead?
var qry = new Select().From(Product.Schema).Where(Product.CategoryIdColumn).IsEqualTo(x);
return qry.ExecuteReader();
Edit:
Just seen you're using SubSonic3 (not 2 as the above code would be for) but the potential misuse of LINQ and duplication of work still stands.
The code that creates object from the DataReader can be found in DbDataProvider.ToEnumerable. It's called from DbQueryProvider's Execute method (line 227). The best way to "understand" the LINQ magic is to place some breakpoints on DbQueryProvider methods.
Related
I know this was partly asked before, but none of the questions completely answer this.
What happens when one uses LINQ to SQL to retrieve data from the database?
I've read the following questions:
Optimizing a LINQ to SQL query
Linq-To-Sql optimization for queries
What is unclear to me is: at which point is the database accessed? When are the queries run?
If I run the following query, how will it translate to a SQL query?
DatabaseDataContext db = new DatabaseDataContext();
var users = from x in db.Users
where x.Rank > 10
orderby x.RegistrationDate descending
select x)
.Skip(pageIndex * recordCount)
.Take(recordCount);
And then, later, if I try to access some property of some user, how will the query be constructed (this is partly answered here)?
var temp = users.ToList()[0].SomeProperty;
Basically, what I really want to know is how the LINQ to SQL works under the hood, how it goes from the C# language statement to the SQL and how it is optimized.
The LINQ to SQL framework will take your Linq query, which is in fact an expression tree, and then will convert this expression tree into a pure SQL query. See How to: Use Expression Trees to Build Dynamic Queries
In fact, every expression tree can be translated into whatever langage or database you need. You will have different providers implementing IQueryable for different databases (Oracle, SQLite, etc.). Note that LINQ to SQL is the abbreviation of LINQ to SQL Server. On the other hand, Entity framework/LINQ to Entities can be extent more easily for other databases.
The main point here is the IQueryable interface, which contains an expression tree, and also the implementation of the provider. For an example on how to implement a provider, ie how to translate from an expression tree to a query, see LINQ: Building an IQueryable Provider
Here is a snippet that will give you a flavor of what happens under the hood:
if (select.OrderBy != null && select.OrderBy.Count > 0)
{
this.AppendNewLine(Indentation.Same);
sb.Append("ORDER BY ");
for (int i = 0, n = select.OrderBy.Count; i < n; i++)
{
OrderExpression exp = select.OrderBy[i];
if (i > 0)
{
sb.Append(", ");
}
this.Visit(exp.Expression);
if (exp.OrderType != OrderType.Ascending)
{
sb.Append(" DESC");
}
}
}
The queries are run as soon as you demand the result.
var qry = (from x in db.Users where x.Rank > 10 orderby x.RegistrationDate descending
select x)
at this point the query has not run, becuase you haven't used the result.
Put it in a foreach or transfor it to a List and the query is forced to Materiliaze.
The rule of thumb is:
Whenever GetEnumerator is called on an IQueryable - the query is forced to materiliaze (wich meens "to to the database and get the actual recourds")
All you want to know is answered in the article on MSDN about LINQ to SQL: http://msdn.microsoft.com/en-us/library/bb425822.aspx
By the way, if you're only going to use a part of your result, as in your code above, it's better to modify your query, like so:
var prop = (from x in db.Users
where x.Rank > 10
orderby x.RegistrationDate descending
select x.SomeProperty)
.Skip(pageIndex)
.First()
.Select(x => x);
Optimization you do in your query is often more important than how the system performs peephole optimization under the hood...
I recently found out that i cannot call any methods from within a linq query. I am trying to write a query that, on the where clause compares two byte arrays. The value on the database is a GUID of type Raw(32) and it is returned as a byte array. This is the record ID for this table. I need to compare it to another byte array. the second byte array could be converted to a string but since i cannot call methods from within linq i was unable to compare.
I tied a custom "Compare" method, i also wrote an extension method. All received an error indicating "LINQ to Entities does not recognize the method"
Here is the code for what i am trying to do. The where clause causes this error:
LINQ to Entities does not recognize the method 'Boolean SequenceEqual[Byte] (System.Collections.Generic.IEnumerable1[System.Byte], System.Collections.Generic.IEnumerable1[System.Byte])' method, and this method cannot be translated into a store expression."
EPSGEntities dbContex = new EPSGEntities();
byte[] byteArray = ParseHex(ViewState["itemID"].ToString());
var q = (from d in dbContex.EPSG_VSOREJECTS
where d.SDSRECID.SequenceEqual(byteArray)
select d).First();
What version of EntityFramework are you using? On EF6 I am able to simply do the following against a SQL 2012 table with a varbinary column:
var q = dbContext.EPSG_VSOREJECTS.FirstOrDefault(e => e.SDSRECID == byteArray);
Is the SDSRECID property on EPSGEntities of type byte[]?
The alternative here would be to go to straight Sql to get your object. Something like:
dbContext.Database.SqlQuery<EPSG_VSOREJECT>("SELECT TOP 1 *" +
"FROM dbo.EPSGEntities" +
"WHERE SDSRECID = #byteString",
new SqlParameter
{
ParameterName = "byteString",
Value = ViewState["itemID"].ToString(),
}).FirstOrDefault();
Linq to Entities in EF is awesome for most queries, but I sometimes drop into sql when I need to do something unsupported, complex, or just fast. Hope this helps!
I'm not entirely sure this works, but I've found calling .AsEnumerable() on the IQueryable object set lets me apply pretty much any code I wish:
var q = dbContex.EPSG_VSOREJECTS.
.AsEnumerable()
.Where(d => d.SDSRECID.SequenceEqual(byteArray));
Doing so seems to prevent EF from trying to translate the Where() clause into SQL syntax, but I have no real idea what the performance hit would/will be.
This is also using method syntax, since I'm not real familiar with query syntax. HTH.
EDIT:
As some others have noted, you have to be careful with how you add any of the iterative methods (AsEnumerable(), ToList(), etc.) since past that point you are no longer building SQL against your data store. Once you start iterating, EF will execute whatever query has been built up to that point, and from then on you are filtering the result set from the LINQ query.
In this case, I don't know that this can be avoided, unless someone can build the same query as a sproc (which EF can execute on your behalf).
I have an IQueryable that has a list of pages.
I want to do: Pages.OrderByDescending(o => CalculateSort(o.page));
the method calculate sort is similar to that here is a plain english version:
public int calculatesort(page p)
{
int rating = (from r in db.rating select r). sum();
int comments = //query database for comments;
float timedecayfactor = math.exp(-page.totalhoursago);
return sortscore = (rating +comments)* timedecayfactor;
}
when I run a code similar to the one above an error is thrown that the mothode calculatesort cannot be converted to sql.
How can I do a conver the function above to be understood by sql so that I can use it to sort the pages?
Is this not a good approach for large data? Is there another method used to sort sets of results other than dynamically at the database?
I havent slept for days trying to fix this one :(
your code is nowhere near compiling so I'm guessing a lot here but I hope this gives an idea none the less.
As several have posted you need to give Linq-2-Sql an expression tree. Using query syntax that's what happens (by compiler magic)
from p in pages
let rating = (from r in db.rating
where r.PageId == p.PageId
select r.Value).Sum()
let comments = (from c in db.Comments
where c.PageId == p.PageId
select 1).Count()
let timedecayfactor = Math.Exp(-(p.totalhoursago))
orderby (rating + comments)*timedecayfactor descending
select p;
I haven't actually tried this against a database, there's simply too many unknown based on your code, so there might still be stuff that can't be translated.
The error occurs because LINQ cannot convert custom code/methods into SQL. It can convert only Expression<Func<>> objects into SQL.
In your case, you have a complex logic to do while sorting, so it might make sense to do it using a Stored Procedure, if you want to do it in the DB Layer.
Or load all the objects into main memory, and run the calculate sort method on the objects in memory
EDIT :
I don't have the code, so Describing in english is the best I can do :
Have table with structure capable of temporarily storing all the current users data.
Have a calculated field in the Pages table that holds the value calculated from all the non-user specific fields
Write a stored procedure that uses values from these two sources (temp table and calc field) to actually do the sort.
Delete the temp table as the last part in the stored proc
You can read about stored procs here and here
var comments = db.comments.Where(...);
Pages.OrderByDescending(p=>(db.rating.Sum(r=>r.rate) + comments.Count()) * Math.Exp(-p.totalhoursago))
Linq is expecting Calculatesort to return a "queryable" expression in order to generate its own SQL.
In can embed your 'calculatesort' method in this lambda expression. (I replaced your variables with constants in order to compile in my environment)
public static void ComplexSort(IQueryable<string> Pages)
{
Pages.OrderByDescending(p =>
{
int rating = 99;//(from r in db.rating select r). sum();
int comments = 33;//query database for comments;
double timedecayfactor = Math.Exp(88);
return (rating + comments) * timedecayfactor;
});
}
Also, you can even try to run that in parallel (since .net 4.0) replacing the first line with
Pages.AsParallel().OrderByDescending(p =>
Yes, counting previous answers: the LINQ to SQL doesn't know how to translate CalculateSort method. You should convert LINQ to SQL to ordinary LINQ to Object before using custom method.
Try to use this in the way you call the CalculateSort by adding AsEnumerable:
Pages.AsEnumerable().OrderByDescending(o => CalculateSort(o.page));
Then you're fine to use the OrderByDescending extension method.
UPDATE:
LINQ to SQL will always translate the query in the code into Expression tree. It's quite almost the same concept as AST of any programming language. These expression trees are further translated into SQL expression specific to SQL Server's SQL, because currently LINQ to SQL only supports SQL Server 2005 and 2008.
I have a situation where my application constructs a dynamic LINQ query using PredicateBuilder based on user-specified filter criteria (aside: check out this link for the best EF PredicateBuilder implementation). The problem is that this query usually takes a long time to run and I need the results of this query to perform other queries (i.e., joining the results with other tables). If I were writing T-SQL, I'd put the results of the first query into a temporary table or a table variable and then write my other queries around that. I thought of getting a list of IDs (e.g., List<Int32> query1IDs) from the first query and then doing something like this:
var query2 = DbContext.TableName.Where(x => query1IDs.Contains(x.ID))
This will work in theory; however, the number of IDs in query1IDs can be in the hundreds or thousands (and the LINQ expression x => query1IDs.Contains(x.ID) gets translated into a T-SQL "IN" statement, which is bad for obvious reasons) and the number of rows in TableName is in the millions. Does anyone have any suggestions as to the best way to deal with this kind of situation?
Edit 1: Additional clarification as to what I'm doing.
Okay, I'm constructing my first query (query1) which just contains the IDs that I'm interested in. Basically, I'm going to use query1 to "filter" other tables. Note: I am not using a ToList() at the end of the LINQ statement---the query is not executed at this time and no results are sent to the client:
var query1 = DbContext.TableName1.Where(ComplexFilterLogic).Select(x => x.ID)
Then I take query1 and use it to filter another table (TableName2). I now put ToList() at the end of this statement because I want to execute it and bring the results to the client:
var query2 = (from a in DbContext.TableName2 join b in query1 on a.ID equals b.ID select new { b.Column1, b.column2, b.column3,...,b.columnM }).ToList();
Then I take query1 and re-use it to filter yet another table (TableName3), execute it and bring the results to the client:
var query3 = (from a in DbContext.TableName3 join b in query1 on a.ID equals b.ID select new { b.Column1, b.column2, b.column3,...,b.columnM }).ToList();
I can keep doing this for as many queries as I like:
var queryN = (from a in DbContext.TableNameN join b in query1 on a.ID equals b.ID select new { b.Column1, b.column2, b.column3,...,b.columnM }).ToList();
The Problem: query1 is takes a long time to execute. When I execute query2, query3...queryN, query1 is being executed (N-1) times...this is not a very efficient way of doing things (especially since query1 isn't changing). As I said before, if I were writing T-SQL, I would put the result of query1 into a temporary table and then use that table in the subsequent queries.
Edit 2:
I'm going to give the credit for answering this question to Albin Sunnanbo for his comment:
When I had similar problems with a heavy query that I wanted to reuse in several other queries I always went back to the solution of creating a join in each query and put more effort in optimizing the query execution (mostly by tweaking my indexes).
I think that's really the best that one can do with Entity Framework. In the end, if the performance gets really bad, I'll probably go with John Wooley's suggestion:
This may be a situation where dropping to native ADO against a stored proc returning multiple results and using an internal temp table might be your best option for this operation. Use EF for the other 90% of your app.
Thanks to everyone who commented on this post...I appreciate everyone's input!
If the size of TableName is not too big to load the whole table you use
var tableNameById = DbContext.TableName.ToDictionary(x => x.ID);
to fetch the whole table and automatically put it in a local Dictionary with ID as key.
Another way is to just "force" the LINQ evaluation with .ToList(), in the case fetch the whole table and do the Where part locally with Linq2Objects.
var query1Lookup = new Hashset<int>(query1IDs);
var query2 = DbContext.TableName.ToList().Where(x => query1IDs.Contains(x.ID));
Edit:
Storing a list of ID:s from one query in a list and use that list as filter in another query can usually be rewritten as a join.
When I had similar problems with a heavy query that I wanted to reuse in several other queries I always went back to the solution of creating a join in each query and put more effort in optimizing the query execution (mostly by tweaking my indexes).
Since you are running a subsequent query off the results, take your first query and use it as a View on your SQL Server, add the view to your context, and build your LINQ queries against the view.
Have you considered composing your query as per this article (using the decorator design pattern):
Composed LINQ Queries using the Decorator Pattern
The premise is that, instead of enumerating your first (very constly) query, you basically use the decorator pattern to produce a chain of IQueryable that is a result of query 1 and query N. This way you always execute the filtered form of the query.
Hope this might help
Is it possible to use custom method In query for example:
var result = from u in context.MyTable where MyMethod(u) == 10 select u;
As Pranay explains, you cannot have a custom (C#) method as part of the LINQ to SQL query, because LINQ to SQL wouldn't be able to look at the expression tree of the method and so it cannot translate it to SQL.
One option that you have is to write your function in SQL and store it as a SQL function on the SQL Server (possibly, you could also use SQL CLR, but I have not tried that). Then you can add the function to your DataContext type and LINQ to SQL will translate it to calls to the function on SQL server. Something like:
var result = from u in context.MyTable
where context.MyMethod(u) == 10 select u;
The problem, of course, is that you'll need to write the function in SQL (I think SQL CLR could also work - not sure about the performance and other possible complications though)
I also wrote an article (some time ago) that shows how to do this when you write the "method" as an expression tree way (as a value of type Expression<Func<...>>), which is possible, because in this case, the code is compiled as an expression tree. However, there is some postprocessing that has to be done and you can still write just a single expression that can be easily inlined in the LINQ query.
Check this full article : What is and what isn't possible with linq
Following is not possible
// function used in filter
static bool MyFunc(Nwind.Product p)
{
return p.ProductName.StartsWith("B");
}
// query that uses MyFunc
var q =
from p in db.Products
where MyPriceFunc(p.UnitPrice) > 30m
select p
It compiles with no errors, but when you execute it LINQ to SQL throws an exception saying: "Static method System.Boolean MyTest(LINQTest.Nwind.Product) has no supported translation to SQL."
The exception is actually thrown when you try to fetch results from q (for example using the foreach statement), because LINQ to SQL attempts to convert the expression trees to T-SQL only when the results are needed and the query must be executed.
To fix the example you can simply copy the code that checks whether product name starts with "B" to the where clause of the query and it would work fine.
Yes, but if you are using Linq-to-Sql - your method has to have special code to handle to SQL conversion.