Develop Reference

Evaluating mathematical expressions with custom script functions

I am looking for an algorithm or approach to evaluate mathematical expressions that are stated as string. The expression contains mathematical components but also custom functions. I look to implement said algorithm in C#/.Net.
I am aware that Roslyn allows me to evaluate an expression of the kind
"var value = 3+5*11-Math.Sqrt(9);"
I am also familiar how to use "node re-writing" in order to accomplish avoidance of variable declarations or fully qualified function names or the omission of the trailing semicolon in order to evaluate
"value = 3+5*11-Sqrt(9)"
However, what I want to implement on top of this is to offer custom script functions such as
"value = Ratio(A,B)", where Ratio is a custom function that divides each element in vector A by each element in vector B and returns a same length vector.
or
"value = Sma(A, 10)", where Sma is a custom function that calculates the simple moving average of vector/timeseries A with a lookback window of 10.
Ideally I want to get to the ability to provide more complexity such as
"value = Ratio(A,B) * Pi + 0.5 * Spread(C,D) + Sma(E, lookback)", whereby the parsing engine would respect operator precedence and build a parsing tree in order to fetch values, required to evaluate the expression.
I can't wrap my head around how I could solve such kind of problem with Roslyn.
What other approaches are out there to get me started or am I missing features that Roslyn offers that may assist in solving this problem?

Assuming that all your expressions are valid C# expressions you can make use of Roslyn in multiple ways.
You could use Roslyn only for parsing. SyntaxFactory.ParseExpression would give you the syntax tree of an expression. Note that your first (var v = expr;) example is not an expression, but a variable declaration. However v = expr is an expression, namely an AssignmentExpressionSyntax. Then you could traverse this AST, and do with each node what you want to do, basically you'd write an interpreter. The benefit of this approach is that you don't have to write your own parser, walking an AST is very simple, and this approach would be flexible, as defining what you do with "unknown" methods would be perfectly up to you.
Use Roslyn for evaluation too. This can be done in multiple flavors: either putting together a valid C# file, and compiling that into an assembly, or you could go through the Scripting API. This approach would basically require a class library that contains the implementation of all your extra methods, like Sma, Spread, ... But these would also be needed in some form in the first approach, so it's not really an extra effort.
If the only goal is to evaluate the expression, then I would go with the 2nd approach. If there are extra requirements (which you haven't mentioned) like being able to let's say produce a simplified form of an expression, then I'd consider the first solution.
If you find a library that does exactly what you need (and the perf is good, and you don't mind the dependency on 3rd party tools, ...), I'd go with that. MathParser.org-mXparser suggested in the comment seems pretty much what you're looking for.

How does C# async/await relates to more general constructs, e.g. F# workflows or monads?

The C# language design have always (historically) been geared towards solving specific problems rather then finding to address the underlying general problems: see for example http://blogs.msdn.com/b/ericlippert/archive/2009/07/09/iterator-blocks-part-one.aspx for "IEnumerable vs. coroutines":
We could have made it much more general. Our iterator blocks can be seen as a weak kind of coroutine. We could have chosen to implement full coroutines and just made iterator blocks a special case of coroutines. And of course, coroutines are in turn less general than first-class continuations; we could have implemented continuations, implemented coroutines in terms of continuations, and iterators in terms of coroutines.
or http://blogs.msdn.com/b/wesdyer/archive/2008/01/11/the-marvels-of-monads.aspx for SelectMany as a surrogate for (some kind of) Monads:
The C# type system is not powerful enough to create a generalized abstraction for monads which was the primary motivator for creating extension methods and the "query pattern"
I do not want to ask why has been so (many good answers have been already given, especially in Eric's blog, which may apply to all these design decisions: from performance to increased complexity, both for the compiler and the programmer).
What I am trying to understand is to which "general construct" the async/await keywords relate to (my best guess is the continuation monad - after all, F# async is implemented using workflows, which to my understanding is a continuation monad), and how they relate to it (how they differ?, what is missing?, why there is a gap, if any?)
I'm looking for an answer similar to the Eric Lippert article I linked, but related to async/await instead of IEnumerable/yield.
Edit: besides the great answers, some useful links to related questions and blog posts where suggested, I'm editing my question to list them:
A starting point for bind using await
Implementation details of the state machine behind await
Other details on how await gets compiled/rewritten
Alternative, hypothetical implementation using continuations (call/cc)

The asynchronous programming model in C# is very similar to asynchronous workflows in F#, which are an instance of the general monad pattern. In fact, the C# iterator syntax is also an instance of this pattern, although it needs some additional structure, so it is not just simple monad.
Explaining this is well beyond the scope of a single SO answer, but let me explain the key ideas.
Monadic operations.
The C# async essentially consists of two primitive operations. You can await an asynchronous computation and you can return the result from an asynchronous computation (in the first case, this is done using a new keyword, while in the second case, we're re-using a keyword that is already in the language).
If you were following the general pattern (monad) then you would translate the asynchronous code into calls to the following two operations:
Task<R> Bind<T, R>(Task<T> computation, Func<T, Task<R>> continuation);
Task<T> Return<T>(T value);
They can both be quite easily implemented using the standard task API - the first one is essentially a combination of ContinueWith and Unwrap and the second one simply creates a task that returns the value immediately. I'm going to use the above two operations, because they better capture the idea.
Translation. The key thing is to translate asynchronous code to normal code that uses the above operations.
Let's look at a case when we await an expression e and then assign the result to a variable x and evaluate expression (or statement block) body (in C#, you can await inside expression, but you could always translate that to code that first assigns the result to a variable):
[| var x = await e; body |]
= Bind(e, x => [| body |])
I'm using a notation that is quite common in programming languages. The meaning of [| e |] = (...) is that we translate the expression e (in "semantic brackets") to some other expression (...).
In the above case, when you have an expression with await e, it is translated to the Bind operation and the body (the rest of the code following await) is pushed into a lambda function that is passed as a second parameter to Bind.
This is where the interesting thing happens! Instead of evaluating the rest of the code immediately (or blocking a thread while waiting), the Bind operation can run the asynchronous operation (represented by e which is of type Task<T>) and, when the operation completes, it can finally invoke the lambda function (continuation) to run the rest of the body.
The idea of the translation is that it turns ordinary code that returns some type R to a task that returns the value asynchronously - that is Task<R>. In the above equation, the return type of Bind is, indeed, a task. This is also why we need to translate return:
[| return e |]
= Return(e)
This is quite simple - when you have a resulting value and you want to return it, you simply wrap it in a task that immediately completes. This might sound useless, but remember that we need to return a Task because the Bind operation (and our entire translation) requires that.
Larger example. If you look at a larger example that contains multiple awaits:
var x = await AsyncOperation();
return await x.AnotherAsyncOperation();
The code would be translated to something like this:
Bind(AsyncOperation(), x =>
Bind(x.AnotherAsyncOperation(), temp =>
Return(temp));
The key trick is that every Bind turns the rest of the code into a continuation (meaning that it can be evaluated when an asynchronous operation is completed).
Continuation monad. In C#, the async mechanism is not actually implemented using the above translation. The reason is that if you focus just on async, you can do a more efficient compilation (which is what C# does) and produce a state machine directly. However, the above is pretty much how asynchronous workflows work in F#. This is also the source of additional flexibility in F# - you can define your own Bind and Return to mean other things - such as operations for working with sequences, tracking logging, creating resumable computations or even combining asynchronous computations with sequences (async sequence can yield multiple results, but can also await).
The F# implementation is based on the continuation monad which means that Task<T> (actually, Async<T>) in F# is defined roughly like this:
Async<T> = Action<Action<T>>
That is, an asynchronous computation is some action. When you give it Action<T> (a continuation) as an argument, it will start doing some work and then, when it eventually finishes, it invokes this action that you specified. If you search for continuation monads, then I'm sure you can find better explanation of this in both C# and F#, so I'll stop here...

Tomas's answer is very good. To add a few more things:
The C# language design have always (historically) been geared towards solving specific problems rather then finding to address the underlying general problems
Though there is some truth to that, I don't think it is an entirely fair or accurate characterization, so I'm going to start my answer by denying the premise of your question.
It is certainly true that there is a spectrum with "very specific" on one end and "very general" on the other, and that solutions to specific problems fall on that spectrum. C# is designed as a whole to be a highly general solution to a great many specific problems; that's what a general-purpose programming language is. You can use C# to write everything from web services to XBOX 360 games.
Since C# is designed to be a general-purpose programming language, when the design team identifies a specific user problem they always consider the more general case. LINQ is an excellent case in point. In the very early days of the design of LINQ, it was little more than a way to put SQL statements in a C# program, because that's the problem space that was identified. But quite soon in the design process the team realized that the concepts of sorting, filtering, grouping and joining data applied not just to tabular data in a relational database, but also hierarchical data in XML, and to ad-hoc objects in memory. And so they decided to go for the far more general solution that we have today.
The trick of design is figuring out where on the spectrum it makes sense to stop. The design team could have said, well, the query comprehension problem is actually just a specific case of the more general problem of binding monads. And the binding monads problem is actually just a specific case of the more general problem of defining operations on higher kinds of types. And surely there is some abstraction over type systems... and enough is enough. By the time we get to solving the bind-an-arbitrary-monad problem, the solution is now so general that the line-of-business SQL programmers who were the motivation for the feature in the first place are completely lost, and we haven't actually solved their problem.
The really major features added since C# 1.0 -- generic types, anonymous functions, iterator blocks, LINQ, dynamic, async -- all have the property that they are highly general features useful in many different domains. They can all be treated as specific examples of a more general problem, but that is true of any solution to any problem; you can always make it more general. The idea of the design of each of these features is to find the point where they can't be made more general without confusing their users.
Now that I've denied the premise of your question, let's look at the actual question:
What I am trying to understand is to which "general construct" the async/await keywords relate to
It depends on how you look at it.
The async-await feature is built around the Task<T> type, which is as you note, a monad. And of course if you talked about this with Erik Meijer he would immediately point out that Task<T> is actually a comonad; you can get the T value back out the other end.
Another way to look at the feature is to take the paragraph you quoted about iterator blocks and substitute "async" for "iterator". Asynchronous methods are, like iterator methods, a kind of coroutine. You can think of Task<T> as just an implementation detail of the coroutine mechanism if you like.
A third way to look at the feature is to say that it is a kind of call-with-current-continuation (commonly abbreviated call/cc). It is not a complete implementation of call/cc because it does not take the state of the call stack at the time that the continuation is signed up into account. See this question for details:
How could the new async feature in c# 5.0 be implemented with call/cc?
I will wait and see if someone (Eric? Jon? maybe you?) can fill in more details on how actually C# generates code to implement await,
The rewrite is essentially just a variation on how iterator blocks are rewritten. Mads goes through all the details in his MSDN Magazine article:
http://msdn.microsoft.com/en-us/magazine/hh456403.aspx

Using an expression parser in .NET

i am not sure if this is a typical stackoverflow question, but i am working on an application where i should constantly examine some conditions (for example if a certain variable's value is over a threshold). Conditions can be changed at any time and preferably from outside the code.
People suggested i should be using expression parsers, but i still don't understand what advantage do they provide me over basic mathematical operations provided by .NET.
Do you recommend a good .NET expression parser?

I think you need Dynamic LINQ. You can pass the conditions as strings.
Here is a blog post about that by ScottGu: http://weblogs.asp.net/scottgu/archive/2008/01/07/dynamic-linq-part-1-using-the-linq-dynamic-query-library.aspx
I found this by a similar question: Dynamic WHERE clause in LINQ

An expression parser would offer more flexibility. Your expressions could be written as formulars in strings and they could be application data instead of harcoded classes/methods/whatever.
You could do things like:
// Assign an action to an expression given as a string
ExpressionObserver.Add("(a+b+c)/2 > x-y", () => { DoSomething(); });
Or:
// Replace the old expression by something written by the user in the UI
someExpressionActionAssignment.Expression = MyLineEdit1.Text;
But I do not know if the added complexity of all this really pays off in your case. If you only have a few simple expressions then it's probably overkill.

Are lambda expressions/delegates in C# "pure", or can they be?

I recently asked about functional programs having no side effects, and learned what this means for making parallelized tasks trivial. Specifically, that "pure" functions make this trivial as they have no side effects.
I've also recently been looking into LINQ and lambda expressions as I've run across examples many times here on StackOverflow involving enumeration. That got me to wondering if parallelizing an enumeration or loop can be "easier" in C# now.
Are lambda expressions "pure" enough to pull off trivial parallelizing? Maybe it depends on what you're doing with the expression, but can they be pure enough? Would something like this be theoretically possible/trivial in C#?:
Break the loop into chunks
Run a thread to loop through each chunk
Run a function that does something with the value from the
current loop position of each thread
For instance, say I had a bunch of objects in a game loop (as I am developing a game and was thinking about the possibility of multiple threads) and had to do something with each of them every frame, would the above be trivial to pull off? Looking at IEnumerable it seems it only keeps track of the current position, so I'm not sure I could use the normal generic collections to break the enumeration into "chunks".
Sorry about this question. I used bullets above instead of pseudo-code because I don't even know enough to write pseudo-code off the top of my head. My .NET knowledge has been purely simple business stuff and I'm new to delegates and threads, etc. I mainly want to know if the above approach is good for pursuing, and if delegates/lambdas don't have to be worried about when it comes to their parallelization.

First off, note that in order to be "pure" a method must not only have no side effects. It must also always return the same result when given the same arguments. So, for example, the "Math.Sin" method is pure. You feed in 12, it gives you back sin(12) and it is the same every time. A method GetCurrentTime() is not pure even if it has no side effects; it returns a different value every time you call it, no matter what arguments you pass in.
Also note that a pure method really ought not to ever throw an exception; exceptions count as observable side effects for our purposes.
Second, yes, if you can reason about the purity of a method then you can do interesting things to automatically parallelize it. The trouble is, almost no methods are actually pure. Furthermore, suppose you do have a pure method; since a pure method is a perfect candidate for memoization, and since memoization introduces a side effect (it mutates a cache!) it is very attractive to take what ought to be pure methods and then make them impure.
What we really need is some way to "tame side effects" as Joe Duffy says. Some way to draw a box around a method and say "this method isn't side-effect-free, but its side effects are not visible outside of this box", and then use that fact to drive safe automatic parallelization.
I'd love to figure out some way to add these concepts to languages like C#, but this is all totally blue-sky open-research-problem stuff here; no promises intended or implied.

Lambda's should be pure. And then the FrameWork offers automatic paralellization with a simple .AsParallel addition to a LINQ query (PLINQ).
But it is not automatic or guaranteed, the programmer is responsible to make/keep them pure.

Whether or not a lambda is pure is tied to what it is doing. As a concept it is neither pure or impure.
For example: The following lambda expression is impure as it is reading and writing a single variable in the body. Running it in parallel creates a race condition.
var i = 0;
Func<bool> del = () => {
if ( i == 42 ) { return true; }
else ( i++ ) { return false; }
};
Contrarily the following delegate is pure and has no race conditions.
Func<bool> del = () => true;

As for the loop part, you could also use the Parallel.For and Parallel.ForEach for the example about the objects in a game. This is also part of .net 4 , but you can get it as a download.

There is a 13 parts reading that discuss about the new Parallelism support in .NET 4.0 here. It includes discussion on LINQ and PLINQ as well in Part 7. It is a great read, so check it out

Ab-using languages

Some time ago I had to address a certain C# design problem when I was implementing a JavaScript code-generation framework. One of the solutions I came with was using the “using” keyword in a totally different (hackish, if you please) way. I used it as a syntax sugar (well, originally it is one anyway) for building hierarchical code structure. Something that looked like this:
CodeBuilder cb = new CodeBuilder();
using(cb.Function("foo"))
{
// Generate some function code
cb.Add(someStatement);
cb.Add(someOtherStatement);
using(cb.While(someCondition))
{
cb.Add(someLoopStatement);
// Generate some more code
}
}
It is working because the Function and the While methods return IDisposable object, that, upon dispose, tells the builder to close the current scope. Such thing can be helpful for any tree-like structure that need to be hard-codded.
Do you think such “hacks” are justified? Because you can say that in C++, for example, many of the features such as templates and operator overloading get over-abused and this behavior is encouraged by many (look at boost for example). On the other side, you can say that many modern languages discourage such abuse and give you specific, much more restricted features.
My example is, of course, somewhat esoteric, but real. So what do you think about the specific hack and of the whole issue? Have you encountered similar dilemmas? How much abuse can you tolerate?

I think this is something that has blown over from languages like Ruby that have much more extensive mechanisms to let you create languages within your language (google for "dsl" or "domain specific languages" if you want to know more). C# is less flexible in this respect.
I think creating DSL's in this way is a good thing. It makes for more readable code. Using blocks can be a useful part of a DSL in C#. In this case I think there are better alternatives. The use of using is this case strays a bit too far from its original purpose. This can confuse the reader. I like Anton Gogolev's solution better for example.

Offtopic, but just take a look at how pretty this becomes with lambdas:
var codeBuilder = new CodeBuilder();
codeBuilder.DefineFunction("Foo", x =>
{
codeBuilder.While(condition, y =>
{
}
}

It would be better if the disposable object returned from cb.Function(name) was the object on which the statements should be added. That internally this function builder passed through the calls to private/internal functions on the CodeBuilder is fine, just that to public consumers the sequence is clear.
So long as the Dispose implementation would make the following code cause a runtime error.
CodeBuilder cb = new CodeBuilder();
var f = cb.Function("foo")
using(function)
{
// Generate some function code
f.Add(someStatement);
}
function.Add(something); // this should throw
Then the behaviour is intuitive and relatively reasonable and correct usage (below) encourages and prevents this happening
CodeBuilder cb = new CodeBuilder();
using(var function = cb.Function("foo"))
{
// Generate some function code
function.Add(someStatement);
}
I have to ask why you are using your own classes rather than the provided CodeDomProvider implementations though. (There are good reasons for this, notably that the current implementation lacks many of the c# 3.0 features) but since you don't mention it yourself...
Edit: I would second Anoton's suggest to use lamdas. The readability is much improved (and you have the option of allowing Expression Trees

If you go by the strictest definitions of IDisposable then this is an abuse. It's meant to be used as a method for releasing native resources in a deterministic fashion by a managed object.
The use of IDisposable has evolved to essentially be used by "any object which should have a deterministic lifetime". I'm not saying this is write or wrong but that's how many API's and users are choosing to use IDisposable. Given that definition it's not an abuse.

I wouldn't consider it terribly bad abuse, but I also wouldn't consider it good form because of the cognitive wall you're building for your maintenance developers. The using statement implies a certain class of lifetime management. This is fine in its usual uses and in slightly customized ones (like #heeen's reference to an RAII analogue), but those situations still keep the spirit of the using statement intact.
In your particular case, I might argue that a more functional approach like #Anton Gogolev's would be more in the spirit of the language as well as maintainable.
As to your primary question, I think each such hack must ultimately stand on its own merits as the "best" solution for a particular language in a particular situation. The definition of best is subjective, of course, but there are definitely times (especially when the external constraints of budgets and schedules are thrown into the mix) where a slightly more hackish approach is the only reasonable answer.

I often "abuse" using blocks. I think they provide a great way of defining scope. I have a whole series of objects that I use for capture and restoring state (e.g. of Combo boxes or the mouse pointer) during operations that may change the state. I also use them for creating and dropping database connections.
E.g.:
using(_cursorStack.ChangeCursor(System.Windows.Forms.Cursors.WaitCursor))
{
...
}

I wouldn't call it abuse. Looks more like a fancied up RAII technique to me. People have been using these for things like monitors.