I currently use linq expressions to create some dynamic objects using the new features of .net 4.0 such as the DLR.
So from my understanding the compiler can generate these sort of expressions. I also saw that the DynamicMetaObjects used by the framework actually has an underlying linq expression.
But from my point of view these expressions really seem static and I hardly understand how are they evaluated. Are they compiled or just include instructions in them so simulate normal code?
Also how do they fit with the rest of the DLR stuff?
Related
Eric Lippert has said on the record here at SO,
Reflection.Emit is too weak to use to build a real compiler. It's great for little toy compilation tasks like emitting dynamic call sites and expression trees in LINQ queries, but for the sorts of problems you'll face in a compiler you will quickly exceed its capabilities. Use CCI, not Reflection.Emit.
I've got a real compiler that was unfortunately built (not by me) on Reflection.Emit. It's butting up against those limits painfully, and I'd like to convert the emit code over to CCI. I'm finding a few things that there doesn't seem to be any equivalent for, though.
For example, the lines:
_asmBuilder.DefineVersionInfoResource();
_moduleBuilder.CreateGlobalFunctions(); //setup global .data
I don't see any way to do the same things, especially as I can't find any equivalent to ModuleBuilder in the first place.
Is there any good reference or documentation available for how to convert a Reflection.Emit project over to CCI?
I am trying to implement a DSL like feature in C#. It may look something similar to LINQ queries. I am wondering if it is possible to implement new unary or binary operators using Roslyn.
I have been googling last few days without much success. It would be great if someone could point me to some samples or Roslyn documentations.
There are two ways how you could use Roslyn to implement a new C#-based language.
Use the Roslyn API to parse the source code into a syntax tree, then transform the syntax tree into actual C# and compile that.
This is ideal if your language is actually syntactically valid C# code, but the semantics are different. For example, you could implement await this way, if you forced await to look like a function call (e.g. await(x) would be valid, but not await x).
If you want to introduce new syntax (like a new operator), it might work, since Roslyn does support parsing “broken” code. But it most likely won't work that well, because then the syntax tree might not look the way you want. Worse, the results might not be consistent (sometimes, your new syntax will be parsed one way, sometimes another).
Since Roslyn is now open source, you can actually modify the source code of the compiler in any way you want, including adding a new operator.
But doing that is most likely not going to be simple. And I think the workflow is also going to be more complicated: you need to compile your own version of the compiler, not just use a library from NuGet like in option 1.
I'm looking to see if there is a method or tool to view how things like closures or query expressions are created by the c# compiler "under the hood". I've noticed that many blog posts dealing with these issues will have the original code with syntactic sugar and the underlying c# code that the compiler converts that to. So for example with linq and query expressions they would show:
var query = from x in myList select x.ToString();
then the resulting code would be
var query = myList.Select(x=>x.ToString());
Is it possible with a tool or do you just have to know how it works from the spec and go from there?
Resharper can do this conversion (LINQ expression syntax to lambda syntax) for you very easily.
LINQPad has a tab that can show you the lambda expression syntax for a query you enter into it, and it has another tab that disassembles it all the way down to the IL code level. (There's another tab that shows you the SQL that gets generated if you're using LINQ to SQL or LINQ to Entities).
SharpLab.io is the tool you are looking for.
From the site, SharpLab shows intermediate steps and results of code compilation. It allows you to see the code as compiler sees it, and get a better understanding of .NET languages.
I'm not a user of Resharper, but I'm sure that will work.
You already seem to know the alternate form of your Linq syntax query. I think that is the answer; Know what you are doing. Read up on it. If you don't know, you are dangerous, and you will go nowhere near my code base. :)
Seriously, this takes a minimum of reading up on, and provides you with immense power in C#. If you are at all interested in your work, you know this.
How far under the hood do you want to go?
If you're really interested in seeing what your code looks like after you compile it, you should check out ildasm.exe. This tool will show you the cold, hard, IL, generated when you compile your application. This tool will allow you to open up any of your compiled assemblies are view the real nuts and bolts that are under the hood.
I'm making a Genetic Program, but I'm hitting a limitation with C# where I want to present new functions to the algorithm but I can't do it without recompiling the program. In essence I want the user of the program to provide the allowed functions and the GP will automatically use them. It would be great if the user is required to know as little about programming as possible.
I want to plug in the new functions without compiling them into the program. In Python this is easy, since it's all interpreted, but I have no clue how to do it with C#. Does anybody know how to achieve this in C#? Are there any libraries, techniques, etc?
It depends on how you want the user of the program to "provide the allowed functions."
If the user is choosing functions that you've already implemented, you can pass these around as delegates or expression trees.
If the user is going to write their own methods in C# or another .NET language, and compile them into an assembly, you can load them using Reflection.
If you want the user to be able to type C# source code into your program, you can compile that using CodeDom, then call the resulting assembly using Reflection.
If you want to provide a custom expression language for the user, e.g. a simple mathematical language, then (assuming you can parse the language) you can use Reflection.Emit to generate a dynamic assembly and call that using -- you guessed it -- Reflection. Or you can construct an expression tree from the user code and compile that using LINQ -- depends on how much flexibility you need. (And if you can afford to wait, expression trees in .NET 4.0 remove many of the limitations that were in 3.5, so you may be able to avoid Reflection.Emit altogether.)
If you are happy for the user to enter expressions using Python, Ruby or another DLR language, you can host the Dynamic Language Runtime, which will interpret the user's code for you.
Hosting the DLR (and IronPython or IronRuby) could be a good choice here because you get a well tested environment and all the optimisations the DLR provides. Here's a how-to using IronPython.
Added in response to your performance question: The DLR is reasonably smart about optimisation. It doesn't blindly re-interpret the source code every time: once it has transformed the source code (or, specifically, a given function or class) to MSIL, it will keep reusing that compiled representation until the source code changes (e.g. the function is redefined). So if the user keeps using the same function but over different data sets, then as long as you can keep the same ScriptScope around, you should get decent perf; ditto if your concern is just that you're going to run the same function zillions of times during the genetic algorithm. Hosting the DLR is pretty easy to do, so it shouldn't be hard to do a proof of concept and measure to see if it's up to your needs.
You can try to create and manipulate Expression Trees. Use Linq to evaluate expression trees.
You can also use CodeDom to compile and run a function.
For sure you can google to see some examples that might fit your needs.
It seems that this article "How to dynamically compile C# code" and this article "Dynamically executing code in .Net" could help you.
You have access to the Compiler from within code, you can then create instances of the compiled code and use them without restarting the application. There are examples of it around
Here
and
Here
The second one is a javascript evaluator but could be adapted easily enough.
You can take a look at System.Reflection.Emit to generate code at the IL level.
Or generate C#, compile into a library and load that dynamically. Not nearly as flexible.
It is in fact very easy to generate IL. See this tutorial: http://www.meta-alternative.net/calc.pdf
What exactly is Expression<> used for in C#? Are there any scenarios where you would instantiate Expression<>'s yourself as an object? If so, please give an example!
Thank you!
Expression<T> is almost entirely used for LINQ, but it doesn't have to be. Within LINQ, it's usually used to "capture" the logic expressed in code, but keep it in data. That data can then be examined by the LINQ provider and handled appropriately - e.g. by converting it into SQL. Usually the expression trees in LINQ are created by the compiler from lambda expressions or query expressions - but in other cases it can be handy to use the API directly yourself.
A few examples of other places I've used it and seen it used:
In MiscUtil, Marc Gravell used it to implement "generic arithmetic" - if a type has the relevant operator, it can be used generically.
In UnconstrainedMelody I used it in a similar way to perform operations on flags enums, regardless of their underlying type (which is trickier than you might expect, due to long and ulong having different ranges)
In Visual LINQ I used query expressions to "animate" LINQ, so you can see what's going on. While obviously this is a LINQ usage, it's not the traditional form of translating logic into another form.
In terms of LINQ, there are things you can do to create more versatile LINQ queries at runtime than you can purely in lambdas.
I've used Expression many times as a micro-compiler, as an alternative to DynamicMethod and IL. This approach gets stronger in .NET 4.0 (as discussed on InfoQ), but even in 3.5 there are lots of things you can do (generally based on runtime data; configuration etc):
generic operators
object cloning
complex initialization
object comparison
I also used it as part of a maths engine for some work I did with Microsoft - i.e. parse a math expression ("(x + 12) * y = z" etc) into an Expression tree, compile it and run it.
Another intersting use (illustrated by Jason Bock, here) is in genetic programming; build your candidates as Expression trees, and you have the necessary code to execute them quickly (after Compile()), but importantly (for genetic programming), also to swap fragments around.
Take a look at my before & after code in my answer to another SO question.
Summary: Expression<> greatly simplified the code, made it easier to understand, and even fixed a phantom bug.