Develop Reference

Running CPython functions from C#

I'm working on a project where I need to be able to run a python function that depends on SciPy/NumPy. Due to this being an add-on to a project already in progress, using IronPython would not be an option.
Additional info:
Python.NET seemed to be a good fit, but I was unable to get the return value from RunString() (it would only return NULL).
Passing arguments and catching the return value (a tuple) is necessary.
The function is in a statistical package that was created by a support group for the team, so modification of that would also not be possible.
I'm at quite a loss for what to do. Any hints in the right direction are appreciated. Thanks for any help you can give!
I understand that this may be quite vague, but I cannot give explicit details to the project. If any clarification is needed please let me know and I'll do my best!

I guess you could write a DLL that uses the CPython API to expose the function, then call it in C#?
It's possible to embed the Python interpreter; although I've never done this personally, I guess it would be useful: http://docs.python.org/extending/embedding.html

Does it need to be portable beyond Windows? If not, perhaps you can embed the CPython interpreter with C++/CLI, wrap that in a nice .Net-ish interface and use the resulting code from C#. Never tried that, so I don't know if it's going to work.
Regardless if you go through this route or the 'write a native DLL' route, it will probably be easier to to embed python using Boost.Python, though I'm not sure if your wrapper code enough is going to be large enough to make all of this (compiling the Boost behemoth, learning Boost.Python, making sure it works with C++/CLR, increasing your target file size) worth it.

IronPython using DLR might be the way to go. Mind you it won't be the fastest way, but it seems like something worth pursuing if you're going to do this a lot. Another useful link

The ironclad project was started to allow using CPython extensions from IronPython, especially SciPy/NumPy it seems. I don't know how usable it is (and how actively it is still being developed)

c# compile source code from database

I would like to build an application framework that is mainly interpreted.
Say that the source code would be stored in the database that could be edited by the users and always the latest version would be executed.
Can anyone give me some ideas how does one implement sth like this !
cheers,
gabor

In .Net, you can use reflection and CodeDOM to compile code on the fly. But neither approach is really very simple or practical. Mono has some ability to interpret c# on the fly as well, but I haven't looked closely at it yet.
Another alternative is to go with an interpreted .Net language like Boo or IronPython as the language for your database code.
Either way, make sure you think long and hard about the security of your platform. Allowing users to execute arbitrary code is always an exercise fraught with peril. It's often too tempting to look for a simple eval() method, and even if one exists, that is not good enough for this kind of scenario.

Try Mono ( http://www.monoproject.org ). It supports many scripting languages including JavaScript.

If you don't want to use any scripting you can use CodeDOM or Reflection (see Reflection.Emit).
Here are really useful links on the topic :
Dynamically executing code in .Net (Here you can find a tool which can be very helpul)
Late Binding and On-the-Fly Code
Generation Using Reflection in C#
Dynamic Source Code Generation and
Compilation

Usually the Program uses a scripting language for the scriptable parts, i.e. Lua or Javascript.

To answer your technical question: You don't want to write your own language and interpreter. That's too much work for you to do. So pick some other language, say Python or Lua, and look for the documentation that lets your C program hand it blocks of code to execute. Of course, the script needs to be able to do something, so you'll need to find how to expose your program's objects to the script. Also, what will happen if a client is running the program when you update its source code in the database? Should the client restart? Are you going to store the entire program as a single row in this database, or did you want to store individual functions? That affects how you structure your updates.
To address other issues with your question: Why do you want to do this? Making "interpreted language" part of your design spec for a system is not often a good sign. Is the real requirement something like this: "I update the program often and I want users to always have the latest copy?" If so, there are other, better ways to go about this (just give us your actual scenario and requirements).

how to transition from C# to python?

i feel like i'm going back to the stone age.
how do i relearn to develop without intellisense (pydev intellisense doesn't count).. in general, how does one successfully leave the comfort of visual studio ?

I recently learned python with a strong C# background.
My advise: Just do it. Sorry, couldn't resist but I'm also serious. Install python and read: Python.org documentation (v2.6). A book might help too -- I like the Python PhraseBook. From there, I started using python to implement solutions for various things. Most notably, ProjectEuler.net questions.
It forced me to consider the languages and built in data structures.
Python is truly easy to use and intuitive. To learn the basics, took me about an hour. To get pretty good with it, took around 5 hours. Of course, there is always more to learn.
Also, I want to note that I would discourage using IronPython or Jython because I feel learning core, regular python is the first step.

Python has rich "introspection" features. In particular, you can find out a lot about built-in features using a command called help() from the Python command line.
Suppose you want to use regular expressions, and want to find out how to use them.
>>> import re
>>> help(re)
You get a nice display of information, automatically shown to you a page at a time (hit the space bar to see the next page).
If you already know you want to use the sub() function from the re module, you can get help on just that:
>>> help(re.sub)
And this help() feature will even work on your own code, as long as you define Python docstrings for your modules, classes, and functions.
You can enable features in the vim editor (or gvim, or vim for Windows) that enable an "IntelliSense"-like autocompletion feature, and you can use Exuberant Ctags to generate hyperlink "tags" to let you navigate quickly through your code. These turn vim into something that is roughly as powerful as an IDE, with the full power of vim for editing. (There isn't an explicit refactoring tool built in to vim, but there are options available.
And as others have noted, you can get IDEs for Python too. I've used the Wingware IDE, and I recommend it. I try to do most of my work with free, open-source software, but that is one piece of proprietary software I would be willing to buy. I have also used Eclipse with the Pydev plugin (I used its refactoring tool and it worked fine).
P.S. Python has a richer feature set than C#, albeit at the cost that your code won't run as fast. Once you get used to Python, you won't feel like you are in the stone age anymore.

One step at a time?
Start off with simple programs (things you can write with your eyes closed in C#), and keep going... You will end up knowing the API by heart.

<rant>
This is sort of the reason that I think being a good visual studio user makes you a bad developer. Instead of learning an API, you look in the object browser until you find something that sounds more or less like what you are looking for, instantiate it, then hit . and start looking for what sounds like the right thing to use. While this is very quick, it also means it takes forever to learn an API in any depth, which often means you end up either re-inventing the wheel (because the wheel is buried under a mountain worth of household appliances and you had no idea it was there), or just doing things in a sub-optimal way. Just because you can find A solution quickly, doesn't mean it is THE BEST solution.
</rant>
In the case of .NET, which ships with about a billion APIs for everything under the sun, this is actually preferred. You need to sift through a lot of noise to find what you are aiming for.
Pythonic code favors one, obvious way to do any given thing. This tends to make the APIs very straight forward and readable. The trick is to learn the language and learn the APIs. It is not terribly hard to do, especially in the case of python, and while not being able to rely on intellisense may be jarring at first, it shouldn't take more then a week or two before you get used to it not being there.
Learn the language and the basic standard libraries with a good book. When it comes to a new library API, I find the REPL and some exploratory coding can get me to a pretty good understanding of what is going on fairly quickly.

You could always start with IronPython and continue to develop in Visual Studio.

The python ide from wingware is pretty nice. Thats what I ended up using going from visual studio C++/.net to doing python.

Don't worry about intellisense. Python is really simple and there really isn't that much to know, so after a few projects you'll be conceiving of python code while driving, eating, etc., without really even trying. Plus, python's built in text editor (IDLE) has a wimpy version of intellisense that has always been more than sufficient for my purposes. If you ever go blank and want to know what methods/properties and object has, just type this in an interactive session:
dir(myobject)

Same way you do anything else that doesn't have IntelliStuff.
I'm betting you wrote that question without the aid of an IntelliEnglish computer program that showed you a list of verbs you could use and automatically added punctuation at the end of your sentences, for example. :-)

I use Eclipse and PyDev, most of the time, and the limited auto-completion help that it provides is pretty useful.
It's not ever going to come up to the level of VS's IntelliSense, and it can't, because of the dynamic nature of Python. But there are compensations, big ones.
The biggest is the breaking of the code-compile-test cycle. It's so easy to write and test prototype code in IDLE that very often it's where I go first: I'll sketch out and test a couple of methods that have to interoperate, figure out that there's something I don't know, learn it, fix my test, and then port the whole thing over to PyDev and watch it work the first time.
Another is that it's a lot simpler. It's really important to know what the standard modules are, and what they do, but for the most part that can be picked up with a little reading. I only use a small handful of modules in my everyday programming - itertools, os, csv (yeah, well), datetime, StringIO - and everything else is there if I need it, but usually I don't.
The stuff that it's really important to know is stuff that IntelliSense couldn't help you with anyway. Auto-completion isn't going to make
self.__dict__.update(kwargs)
make a damn bit of sense; you have to learn what an amazing line of code that is, and why you'd write it, by yourself.
Then you'll think, "how would I even begin to implement something like that in C#?" and realize that the tools these stone-age people are using are a little more sophisticated than you think.

A mistake people coming from C# or Java make a lot is recreate a large class hierarchy like is common in those languages. With Python this is almost never useful. Instead, learn about duck typing. And don't worry too much about the lack of static typing: in practice, it's almost never an issue.

I'm not too familiar with Python so I'll give some general advice. Make sure to print out some reference pages of common functions. Keep them by you(or pinned to a wall or desk) at all times when your programming.. I'm traditionally a "stone age" developer and after developing in VS for a few months I'm finding my hobby programming is difficult without intellisense.. but keep at it to memorize the most common functions and eventually it'll just be natural
You'll also have to get use to a bit more "difficult" of debugging(not sure how that works with python)
While your learning though, you should enjoy the goodness(at least compared to C#) of Python even if you don't have intellisense.. hopefully this will motivate you!

I've only ever used IDLE for python development and it is not fun in the least. I would recommend getting your favorite text editor, mine is notepad++, and praying for a decent plugin for it. I only ever had to go from Eclipse to Visual Studio so my opinions are generally invalid in these contexts.

Pyscripter does a reasonable job at auto-completion/intellisense. I've been using it recently as I started to learn Python/Django, where I've been mainly a C# developer for the last few years.

I suggest going cold turkey - languages like Python shine with great text editors. Choose one you want to become amazing at (vim, emacs, etc.) and never look back.

I use Komodo Edit and it does reasonably good guessing at the autocompletion.

Others have suggested several editors that have intellisense-like capabilities. Try them out.
Also install ipython and use that to learn the language interactively. It is like a souped up version of the regular python interactive shell with lots and lots of added capabilities, and one of the most useful it its extensive context sensitive tab-completion and help.
For example if you type
import r<tab>
it will show all the modules you can import starting with r
import re
re.<tab>
will show all the objects in the re module
re.compile?
will show the docstring and other information about the re.compile function, automatically piping it through a pager if it is longer than a screenful.
re.compile??
will show the source code as well, if available.
Using this I find it is much faster to switch to ipython and query objects directly than it is to look up anything in the docs. You have access to the usual python help() system as well.
ipython has lots and lots of other features - far too many to cover in a short post.

Dynamic code generation

I am currently developing an application where you can create "programs" with it without writing source code, just click&play if you like.
Now the question is how do I generate an executable program from my data model. There are many possibilities but I am not sure which one is the best for me. I need to generate assemblies with classes and namespace and everything which can be part of the application.
CodeDOM class: I heard of lots of limitations and bugs of this class. I need to create attributes on method parameters and return values. Is this supported?
Create C# source code programmatically and then call CompileAssemblyFromFile on it: This would work since I can generate any code I want and C# supports most CLR features. But wouldn't this be slow?
Use the reflection ILGenerator class: I think with this I can generate every possible .NET code. But I think this is much more complicated and error prone than the other approaches?
Are there other possible solutions?
EDIT:
The tool is general for developing applications, it is not restricted to a specific domain. I don't know if it can be considered a visual programming language. The user can create classes, methods, method calls, all kinds of expressions. It won't be very limitating because you should be able to do most things which are allowed in real programming languages.
At the moment lots of things must still be written by the user as text, but the goal at the end is, that nearly everything can be clicked together.

You my find it is rewarding to look at the Dynamic Language Runtime which is more or less designed for creating high-level languages based on .NET.
It's perhaps also worth looking at some of the previous Stack Overflow threads on Domain Specific Languages which contain some useful links to tools for working with DSLs, which sounds a little like what you are planning although I'm still not absolutely clear from the question what exactly your aim is.

Most things "click and play" should be simple enough just to stick some pre-defined building-block objects together (probably using interfaces on the boundaries). Meaning: you might not need to do dynamic code generation - just "fake it". For example, using property-bag objects (like DataTable etc, although that isn't my first choice) for values, etc.
Another option for dynamic evaluation is the Expression class; especially in .NET 4.0, this is hugely versatile, and allows compilation to a delegate.

Do the C# source generation and don't care about speed until it matters. The C# compiler is quite quick.

When I wrote a dynamic code generator, I relied heavily on System.Reflection.Emit.
Basically, you programatically create dynamic assemblies and add new types to them. These types are constructed using the Emit constructs (properties, events, fields, etc..). When it comes to implementing methods, you'll have to use an ILGenerator to pump out MSIL op-codes into your method. That sounds super scary, but you can use a couple of tools to help:
A pre-built sample implementation
ILDasm to inspect the op-codes of the sample implementation.

It depends on your requirements, CodeDOM would certainly be the best fit for a "program" stored it in a "data model".
However its unlikely that using option 2 will be in any way measurably slower in comparision with any other approach.

I would echo others in that 1) the compiler is quick, and 2) "Click and Play" things should be simple enough so that no single widget added to a pile of widgets can make it an illegal pile.
Good luck. I'm skeptical that you can achieve point (2) for anything but really toy-level programs.

Best/fastest way to write a parser in c#

What is the best way to build a parser in c# to parse my own language?
Ideally I'd like to provide a grammar, and get Abstract Syntax Trees as an output.
Many thanks,
Nestor

I've had good experience with ANTLR v3. By far the biggest benefit is that it lets you write LL(*) parsers with infinite lookahead - these can be quite suboptimal, but the grammar can be written in the most straightforward and natural way with no need to refactor to work around parser limitations, and parser performance is often not a big deal (I hope you aren't writing a C++ compiler), especially in learning projects.
It also provides pretty good means of constructing meaningful ASTs without need to write any code - for every grammar production, you indicate the "crucial" token or sub-production, and that becomes a tree node. Or you can write a tree production.
Have a look at the following ANTLR grammars (listed here in order of increasing complexity) to get a gist of how it looks and feels
JSON grammar - with tree productions
Lua grammar
C grammar

I've played wtih Irony. It looks simple and useful.

You could study the source code for the Mono C# compiler.

While it is still in early beta the Oslo Modeling language and MGrammar tools from Microsoft are showing some promise.

I would also take a look at SableCC. Its very easy to create the EBNF grammer. Here is a simple C# calculator example.

There's a short paper here on constructing an LL(1) parser here, of course you could use a generator too.

Lex and yacc are still my favorites. Obscure if you're just starting out, but extremely simple, fast, and easy once you've got the lingo down.
You can make it do whatever you want; generate C# code, build other grammars, emulate instructions, whatever.
It's not pretty, it's a text based format and LL1, so your syntax has to accomodate that.
On the plus side, it's everywhere. There are great O'reilly books about it, lots of sample code, lots of premade grammars, and lots of native language libraries.