I was reading this post here on micro ORM used on SO.
The author showed this stack-trace:
System.Reflection.Emit.DynamicMethod.CreateDelegate
System.Data.Linq.SqlClient.ObjectReaderCompiler.Compile
System.Data.Linq.SqlClient.SqlProvider.GetReaderFactory
System.Data.Linq.SqlClient.SqlProvider.System.Data.Linq.Provider.IProvider.Compile
System.Data.Linq.CommonDataServices+DeferredSourceFactory`1.ExecuteKeyQuery
System.Data.Linq.CommonDataServices+DeferredSourceFactory`1.Execute
System.Linq.Enumerable.SingleOrDefault
System.Data.Linq.EntityRef`1.get_Entity
Then said:
In the trace above you can see that 'EntityRef' is baking a method, which is not a problem, unless it is happening 100s of times a second.
Could someone explain the stack-trace in relation to what he meant by "baking a method" and why it would be a performance problem?
When you do something like:
int[] items = whatever;
IEnumerable<int> query = from item in items where item % 2 == 0 select item;
then the compiler turns that into something like:
static bool Predicate(int item) { return item % 2 == 0; }
...
IEnumerable<int> query = Enumerable.Where<int>(items, new Func<int, bool> (Predicate));
That is, the compiler generates the IL for the method. When you build the query object at runtime, the object returned by Where holds on to a delegate to the predicate, and executes the predicate when necessary.
But it is possible to build the IL for the delegate at runtime if you want to. What you do is persist the body of the predicate as an expression tree. At runtime the expression tree can dynamically compile itself into brand-new IL; basically we start up a very simplified compiler that knows how to generate IL for expression trees. That way you can change the details of the predicate at runtime and recompile the predicate without having to recompile the whole program.
The author of that comment is using "baking" as a slang for "dynamic lightweight code generation".
It refers to creating a method dynamically at run time, for example by using an expression tree. Compiling the expression tree to a method and returning a delegate for the compiled method could be called "baking" the method.
It's a fancy way of saying that they're emitting dynamic code using Refrection.Emit. A notoriously slow process.
Related
Why can't I use lambda expressions while debugging in “Quick watch” window?
UPD: see also
Link
Link
No you cannot use lambda expressions in the watch / locals / immediate window. As Marc has pointed out this is incredibly complex. I wanted to dive a bit further into the topic though.
What most people don't consider with executing an anonymous function in the debugger is that it does not occur in a vaccuum. The very act of defining and running an anonymous function changes the underlying structure of the code base. Changing the code, in general, and in particular from the immediate window, is a very difficult task.
Consider the following code.
void Example() {
var v1 = 42;
var v2 = 56;
Func<int> func1 = () => v1;
System.Diagnostics.Debugger.Break();
var v3 = v1 + v2;
}
This particular code creates a single closure to capture the value v1. Closure capture is required whenever an anonymous function uses a variable declared outside it's scope. For all intents and purposes v1 no longer exists in this function. The last line actually looks more like the following
var v3 = closure1.v1 + v2;
If the function Example is run in the debugger it will stop at the Break line. Now imagine if the user typed the following into the watch window
(Func<int>)(() => v2);
In order to properly execute this the debugger (or more appropriate the EE) would need to create a closure for variable v2. This is difficult but not impossible to do.
What really makes this a tough job for the EE though is that last line. How should that line now be executed? For all intents and purposes the anonymous function deleted the v2 variable and replaced it with closure2.v2. So the last line of code really now needs to read
var v3 = closure1.v1 + closure2.v2;
Yet to actually get this effect in code requires the EE to change the last line of code which is actually an ENC action. While this specific example is possible, a good portion of the scenarios are not.
What's even worse is executing that lambda expression shouldn't be creating a new closure. It should actually be appending data to the original closure. At this point you run straight on into the limitations ENC.
My small example unfortunately only scratches the surface of the problems we run into. I keep saying I'll write a full blog post on this subject and hopefully I'll have time this weekend.
Lambda expressions, like anonymous methods, are actually very complex beasts. Even if we rule out Expression (.NET 3.5), that still leaves a lot of complexity, not least being captured variables, which fundamentally re-structure the code that uses them (what you think of as variables become fields on compiler-generated classes), with a bit of smoke and mirrors.
As such, I'm not in the least surprised that you can't use them idly - there is a lot of compiler work (and type generation behind the scenes) that supports this magic.
You can't use lambda expressions in the Immediate or Watch windows.
You can however use System.Linq.Dynamic expressions, which take the form .Where("Id = #0", 2) - it doesn't have the full range of methods available in standard Linq, and doesn't have the full power of lambda expressions, but still, it's better than nothing!
The future has come!
Support for debugging lambda expressions has been added to Visual Studio 2015 (Preview at the time of writing).
Expression Evaluator had to be rewritten, so many features are missing: remote debugging ASP.NET, declaring variables in Immediate window, inspecting dynamic variables etc. Also lambda expressions that require calls to native functions aren't currently supported.
this might help:
Extended Immediate Window for Visual Studio (use Linq, Lambda Expr in Debugging)
http://extendedimmediatewin.codeplex.com/
http://dvuyka.spaces.live.com/blog/cns!305B02907E9BE19A!381.entry
All the best,
Patrick
Lambda expressions are not supported by the debugger's expression evaluator... which is hardly surprising since at compile time they are used to create methods (or Expression Trees) rather than expressions (take a look in Reflector with the display switched to .NET 2 to see them).
Plus of course they could form a closure, another whole layer of structure.
In VS 2015 you can do so now,this is one of the new feature they added.
If you still need to use Visual Studio 2013, you can actually write a loop, or lambda expression in the immediate window using also the package manager console window. In my case, I added a list at the top of the function:
private void RemoveRoleHierarchy()
{
#if DEBUG
var departments = _unitOfWork.DepartmentRepository.GetAll().ToList();
var roleHierarchies = _unitOfWork.RoleHierarchyRepository.GetAll().ToList();
#endif
try
{
//RoleHierarchy
foreach (SchoolBo.RoleHierarchy item in _listSoRoleHierarchy.Where(r => r.BusinessKeyMatched == false))
_unitOfWork.RoleHierarchyRepository.Remove(item.Id);
_unitOfWork.Save();
}
catch (Exception e)
{
Debug.WriteLine(e.ToString());
throw;
}
}
Where my GetAll() function is:
private DbSet<T> _dbSet;
public virtual IList<T> GetAll()
{
List<T> list;
IQueryable<T> dbQuery = _dbSet;
list = dbQuery
.ToList<T>();
return list;
}
Here I kept getting the following error, so I wanted to print out all the items in the various repositories:
InnerException {"The DELETE statement conflicted with the REFERENCE constraint \"FK_dbo.Department_dbo.RoleHierarchy_OranizationalRoleId\". The conflict occurred in database \"CC_Portal_SchoolObjectModel\", table \"dbo.Department\", column 'OranizationalRoleId'.\r\nThe statement has been terminated."} System.Exception {System.Data.SqlClient.SqlException}
Then, I find out how many records are in the department repository by executing this in the immediate window:
_unitOfWork.DepartmentRepository.GetAll().ToList().Count
Which returned 243.
So, if you execute the following in the package manager console, it prints out all the items:
PM> for($i = 0; $i -lt 243; $i++) { $a = $dte.Debugger.GetExpression("departments[$i].OrgagnizationalRoleId"); Write-Host $a.Value $i }
The author for the idea can be found here
To answer your question, here's the Visual Studio Program Manager's official explanation of why you can't do this. In short, because "it's really, really hard" to implement in VS. But the feature is currently in progress (as updated on Aug 2014).
Allow the evaluation of lambda expressions while debugging
Add your vote while you're there!
I just came across the concept of expression trees which I have heard multiple times. I just want to understand what is meant by an expression tree and its purpose.
I would love it if someone could also direct me to simple explanations and samples of use.
An Expression Tree is a data structure that contains Expressions, which is basically code. So it is a tree structure that represents a calculation you may make in code. These pieces of code can then be executed by "running" the expression tree over a set of data.
A great thing about expression trees is that you can build them up in code; that is, you build executable code (or a sequence of steps) in code. You can also modify the code before you execute it by replacing expressions by other expressions.
An Expression is then a function delegate, such as (int x => return x * x).
See also http://blogs.msdn.com/b/charlie/archive/2008/01/31/expression-tree-basics.aspx
Consider the factorial function defined within a method body as a lambda expression and assigned to a variable:
Func<int, int> factfail = n =>
{
if (n == 0)
return 1;
else
return n * factfail(n-1);
};
This fails, since factfail isn't bound by the the local variable yet.
Is there a way to add a sort of fixpoint - by abstracting over the function itself?!
Func<Func<int, int>, int, int> fact_ = (fact, n) =>
{
if (n == 0)
return 1;
else
return n * fact(n-1);
};
fact_(??);
long story:
I need to write a recursive function that has the side effect of changing some outer state.
Therefore i am trying to write that method as a lambda expression that captures that outer state.
I am still experimenting with different styles how to write that and - besides of that one dictionary that needs to be the same for all recursive calls - i want to be as purely functional and lazy as possible.
So i was playing with LINQ, since it helps me reducing mutual data.
It also helps with understanding which parts of the code can be expressed in a functional style.
To be brief in the LINQ statement it is helpful to be able to define some helper functions in front of and i did that by binding lambda expressions to variables.
And with lamda expression i also can capture my dictionary without the need to pass its reference to the method explicitely, which is quite nice.
not sure if i am on the right track though...
You can find more information about recursive lambda expressions in this blog post by Mads Torgersen. He shows how to define the usual fixed point combinator. He uses factorial function as an example, so you can find your exact sample there :-).
However, in practice, you can just define a local Func<..> variable and then mutate it. If you want to give a name to the delegate, then it works just fine (it is a bit dirty, but simple):
Func<int, int> fact = null;
fact = (n) => (n == 0) ? 1 : n * fact(n-1);
This works, because the closure captures reference to the fact variable, so when you actually call it (during the recursive call), the value is not null anymore, but references the delegate.
Is there any particular reason? Is it not possible at all or is it just not implemented yet? Maybe there are any third-party addins that allow lambda evaluations?
UPDATE:
I've found this project on codeplex Extended Immediate Window. Seems that it has been abandoned for some time, but this can be a proof of a concept. Does anybody know any other immediate window extension addins? The ones that can run for/foreach statements in C# for instance?
JaredPar of Microsoft wrote a couple of blog posts answering your question: part 1 and part 2. You'll find the answers there.
When writing a lambda, the act of capturing variables significantly alters the construction of the underlying code (moving variables into fields of compiler-generated classes, that could very easily themselves be chained closure-contexts).
Not even considering the general complexity of doing this, it would then have two choices:
capture all the variable values as constants; feasible and pretty simple, but could easily mean that the result of executing in the immediate window is very different to the result of executing in the main body (very undesirable)
rewrite the entire code (for the reasons outlined above) on the fly (at a guess, impossible)
Given a choice between "undesirable" and "impossible", I guess they simply chose not to implement a feature that would be inherently brittle, and very complex to write.
Well, I think it's because the immediate window can only evaluate expressions, or rather it can only do invocations and assignments. To evaluate a Lambda expression a closure would have to be created for that lambda, typchecked and then executed.
I think it comes down to that the Immediate window is just an evaluator and not an interpreter.
http://msdn.microsoft.com/en-us/library/f177hahy(VS.80).aspx
"The Immediate window is used at design time to debug and evaluate expressions, execute statements, print variable values, and so forth. It allows you to enter expressions to be evaluated or executed by the development language during debugging."
So in effect, your question boils down to why you can't define functions in the immediate window (since lambdas are just annonymous functions), and the answer I think is that it simply wasn't designed for that.
If you still need to use Visual Studio 2013, you can actually write a loop, or lambda expression in the immediate window using also the package manager console window. In my case, I added a list at the top of the function:
private void RemoveRoleHierarchy()
{
#if DEBUG
var departments = _unitOfWork.DepartmentRepository.GetAll().ToList();
var roleHierarchies = _unitOfWork.RoleHierarchyRepository.GetAll().ToList();
#endif
try
{
//RoleHierarchy
foreach (SchoolBo.RoleHierarchy item in _listSoRoleHierarchy.Where(r => r.BusinessKeyMatched == false))
_unitOfWork.RoleHierarchyRepository.Remove(item.Id);
_unitOfWork.Save();
}
catch (Exception e)
{
Debug.WriteLine(e.ToString());
throw;
}
}
Where my GetAll() function is:
private DbSet<T> _dbSet;
public virtual IList<T> GetAll()
{
List<T> list;
IQueryable<T> dbQuery = _dbSet;
list = dbQuery
.ToList<T>();
return list;
}
Here I kept getting the following error, so I wanted to print out all the items in the various repositories:
InnerException {"The DELETE statement conflicted with the REFERENCE constraint \"FK_dbo.Department_dbo.RoleHierarchy_OranizationalRoleId\". The conflict occurred in database \"CC_Portal_SchoolObjectModel\", table \"dbo.Department\", column 'OranizationalRoleId'.\r\nThe statement has been terminated."} System.Exception {System.Data.SqlClient.SqlException}
Then, I find out how many records are in the department repository by executing this in the immediate window:
_unitOfWork.DepartmentRepository.GetAll().ToList().Count
Which returned 243.
So, if you execute the following in the package manager console, it prints out all the items:
PM> for($i = 0; $i -lt 243; $i++) { $a = $dte.Debugger.GetExpression("departments[$i].OrgagnizationalRoleId"); Write-Host $a.Value $i }
The author for the idea can be found here: http://ogresoft.blogspot.ca/2013/06/how-to-write-loop-or-lambda-expression.html
I assume, that because it is lazy evaluation, the immediate window cannot know beforehand what values, the captured variables (closure), should have.
It is fairly common to take an Expression tree, and convert it to some other form, such as a string representation (for example this question and this question, and I suspect Linq2Sql does something similar).
In many cases, perhaps even most cases, the Expression tree conversion will always be the same, i.e. if I have a function
public string GenerateSomeSql(Expression<Func<TResult, TProperty>> expression)
then any call with the same argument will always return the same result for example:
GenerateSomeSql(x => x.Age) //suppose this will always return "select Age from Person"
GenerateSomeSql(x => x.Ssn) //suppose this will always return "select Ssn from Person"
So, in essence, the function call with a particular argument is really just a constant, except time is wasted at runtime re-computing it continuously.
Assuming, for the sake of argument, that the conversion was sufficiently complex to cause a noticeable performance hit, is there any way to pre-compile the function call into an actual constant?
Edit
It appears that there is no way to do this exactly within C# itself. The closest you can probably come within c# is the accepted answer (though of course you would want to make sure that the caching itself wasn't slower than regenerating). To actually convert to true constants, I suspect that with some work, you could use something like mono-cecil to modify the bytecodes after compilation.
The excellent LINQ IQueryable Toolkit project has a query cache that does something similar to what you've described. It contains an ExpressionComparer class that walks the hierarchy of two expressions and determines if they are equivalent. This technique is also used to collect references to common properties for parameterization and in the removal of redundant joins.
All you would need to do is come up with an expression hashing strategy so you can store the results of your processed expressions in a dictionary, ready for future reuse.
Your method would then look something like this:
private readonly IDictionary<Expression, string> _cache
= new Dictionary<Expression, string>(new ExpressionEqualityComparer());
public string GenerateSomeSql(Expression<Func<TResult, TProperty>> expression)
{
string sql;
if (!_cache.TryGetValue(expression, out sql))
{
//process expression
_cache.Add(expression, sql);
}
return sql;
}
class ExpressionEqualityComparer : IEqualityComparer<Expression>
{
public bool Equals(Expression x, Expression y)
{
return ExpressionComparer.AreEqual(x, y);
}
public int GetHashCode(Expression obj)
{
return ExpressionHasher.GetHash(obj);
}
}
First of all, I suspect that your assumption about compiling an expression causing a performance hit will not actually pan out in reality. My experience shows that there are many more factors (database access, network latency, very poor algorithms) that cause performance bottlenecks before regular "good" code causes issues. Premature optimization is the root of all evil, so build your application and run stress tests to find the actual performance bottlenecks, as they are often not where you would expect.
With that said, I think that pre-compilation depends on what the Expression is being tranlated into. I know that with LINQ to SQL you can call DataContext.GetCommand(Expression) and retrieve a DBCommand, which you could then cache and reuse.