Develop Reference

Parsing C# code for contextually aware semantic highlighting

I'm working on a semantic highlighting plugin for VS. Here you can see a web Example.
The goal:
Acquiring all variables and creating different Classifications for every one of them.
The problem:
Getting the variables from the code without writing a C# lexer.
My current approach uses an ITagger. I use an ITagAggregator to get the tags of all the spans that get passed to the ITagger. Then I filter those and get only spans with the "identifier" classification which includes varibles, methods names, class names, usings and properties.
public class Classifier : ITagger<ClassificationTag> {
public IEnumerable<ITagSpan<ClassificationTag>> GetTags(NormalizedSnapshotSpanCollection spans) {
ITextSnapshot snapshot = spans[0].Snapshot;
var tags = _aggregator.GetTags(spans).Where((span) => span.Tag.ClassificationType.Classification.Equals("identifier")).ToArray();
foreach(var classifiedSpan in tags) {
foreach(SnapshotSpan span in classifiedSpan.Span.GetSpans(snapshot)) {
//generate classification based on variable name
yield return new TagSpan<ClassificationTag>(span, new ClassificationTag(_classification));
}
}
}
}
It would be a lot easier to use the builtin C# Lexer to get a list of all variables bundled to a bunch of meta data. Is this data available for plugin development? Is there an alternative way I could acquire it, if not?

The problem: Getting the variables from the code without writing a C# lexer.
Roslyn can do this: https://roslyn.codeplex.com/
There's even a Syntax Visualizer sample that might interest you. I also found an example using Roslyn to create a Syntax Highlighter.

Visual Studio exposes that information as a code model.
Here is an example how you can access class, and then find attribute on the class, and parse attribute arguments:
Accessing attribute info from DTE
Here is more information about code models:
http://msdn.microsoft.com/en-us/library/ms228763.aspx
Here's also automation object model chart what I've been using quite few times: http://msdn.microsoft.com/en-us/library/za2b25t3.aspx
Also, as said, Roslyn is indeed also a possible option. Here is an example for VS2015 using roslyn: https://github.com/tomasr/roslyn-colorizer/blob/master/RoslynColorizer/RoslynColorizer.cs

For building language tools if may be better to use a parser generator for C#. The GOLD parsing system is one such toolkit which can handle LALR grammars. It has a .NET component based engine that you can use in your project and it can be used to integrate with any IDE. You can also find the grammars for various programming languages including C#.

NRefactory code completion with FastColoredTextBox in c#

I am building a simple IDE for C#.
..and I am using FastColoredTextBox control for source code editing, with built-in autocomplete menu (http://www.codeproject.com/Articles/161871/Fast-Colored-TextBox-for-syntax-highlighting)
The problem is, that I want to use NRefactor to do automatic code completion, just like in Visual Studio.
I did some research, I found this: https://github.com/lukebuehler/NRefactory-Completion-Sample
...but I didn't understand the code, so I can't reuse it.
To be exact, what I need?
I need an function, which will take as input list of assemblies (array of strings, which are paths to DLLs), current offset, and editor's text.
And as output, I need an array of objects, which will contain name of the object, and it's type (variable, function, type, ...)
So, how do I do it?
Is it necessary to run this autocompletion code in another thread?
Thank you for answers,
VitekST
PS: Don't hate me, I'm not familiar with NRefactor...

I starts to explore NRefactory in same day you asks this question) So, I can't call myself as NRefactory expert. But I can tell you the way, how I use it.
Theoretical material: http://www.codeproject.com/Articles/408663/Using-NRefactory-for-analyzing-Csharp-code
First of all you can use a Visual Studio solution model from this sample project to make similar classes for your IDE solution format: https://github.com/icsharpcode/NRefactory/tree/master/ICSharpCode.NRefactory.ConsistencyCheck
There are three classes you need: CSharpFile, CSharpProject and Solution. Please look at its source code and you will know, how NRefactory performs syntactic analysis of code files. Note "Compilation" field of CSharpProject and "CreateResolver" method of CSharpFile.
Second you need to perform semantic analysis of code. For this purpose you need to implement
ICSharpCode.NRefactory.CSharp.Resolver.IResolveVisitorNavigator
interface. But in your case you better use existing NRefactory implementation:
ICSharpCode.NRefactory.CSharp.Resolver.FindReferencedEntities
Constructor with two parameters waits two handlers. First being executed then resolved syntax element is reference of type, contained in your solution or in the referenced assembly (like variable declaration). Second being executed when resolved syntax element is reference of some type's member (for example "(new object()).ToString()" as a reference of member "ToString" of "object" instance). First parameter of both handlers is AstNode: unresolved syntax element of code. Second parameter is corresponding resolved semantic element: IType for first handler and IMember for another. So you need to create instance of navigator with handlers, that, for example, saving syntax and corresponding semantic elements in dictionaries.
Your code can looks like this:
var solution = new Solution(slnPath);
IDictionary<AstNode, IType> typesMap = new Dictionary<AstNode, IType>();
IDictionary<AstNode, IMember> membersMap = new Dictionary<AstNode, IMember>();
var navigator = new FindReferencedEntities(typesMap.Add, membersMap.Add);
foreach (var codeFile in solution.AllFiles)
{
codeFile.CreateResolver().ApplyNavigator(navigator);
}
After execution of this code dictionaries typesMap and membersMap will containing syntax elements of code, that NRefactory able to resolve, in the keys of dictionaries, and corresponding resolved semantic in values. If you writing autocompletion, in moment of usage processed code can not be compiled with a high probability (because user write it at this moment). So, your app must use not only resolved code, but unresolved too. To get the unresolved syntax elements of code file, than user edit at this moment, you must use:
var syntaxTree = solution.Projects
.Where(p => p.Title.Equals(editableProjName))
.Files
.Where(f => f.FileName.Equals(editableFileNamePath))
.SyntaxTree;
SyntaxTree is a "ICSharpCode.NRefactory.CSharp.SyntaxTree", that inherits AstNode class. It is a root syntax node of the code file. Variable syntaxTree will contain all unresolved syntax elements of current file being edited. To move through the syntax tree you can use AstNode enumerable members like Descendants, Children, GetParent() etc. Or you can use "Visitor" pattern and AstNode method "AcceptVisitor".
While generating list for autocompletion you can use typesMap and membersMap as fast storage of declared types and its members, and syntaxTree as slow storage, if user input stay not found.
NRefactory is a very broad topic, I can't reveal it fully. And it is very sad that this powerful framework haven't got detailed documentations. But I hope my answer can help you.

Getting a specific method source code from .cs file (at runtime)

1 - I have this file content on the disc (cs file, not compiled):
namespace Test
{
using System;
public class TestClass
{
public string SomeTestMethod(){
return "test here";
}
}
}
How do I get in run time into a variable the method:
public string SomeTestMethod(){
return "test here";
}
for example: SourceCodeParser.GetMothod("path to file","SomeTestMethod");
2 - Is it possible the content of accessor member?
public string SomeMember {
get {
return "test here";
}
}

Roslyn is what you need. You can easily install it using nuget. Here is a working code for getting a method body:
string GetMethod(string filename, string methodName)
{
var syntaxTree = SyntaxTree.ParseFile(filename);
var root = syntaxTree.GetRoot();
var method = root.DescendantNodes()
.OfType<MethodDeclarationSyntax>()
.Where(md => md.Identifier.ValueText.Equals(methodName))
.FirstOrDefault();
return method.ToString();
}
and code for getting body of property getter:
string GetPropertyGetter(string filename, string propertyName)
{
var syntaxTree = SyntaxTree.ParseFile(filename);
var root = syntaxTree.GetRoot();
var property = root.DescendantNodes()
.OfType<PropertyDeclarationSyntax>()
.Where(md => md.Identifier.ValueText.Equals(propertyName))
.FirstOrDefault();
var getter = property.AccessorList.Accessors.First(a => a.Kind == SyntaxKind.GetAccessorDeclaration);
return getter.ToString();
}

You need a tool that can parse the source code, tracks code locations, and knows how to look up methods (or variables, or whatever named thing you care about) in the source code. With that, finding the lines of interest is pretty easy. Such a tool isn't easy to build because parsers for full languages aren't easy to build. Nor are the lookup functions easy; parameters, namespaces, templates, inheritance all combine to make name lookup for modern languages remarkably complex.
Program transformation (PT) tools (which often have such full parsers already available) often do this by building an AST in memory which represents the code. Then, given rules for name lookup, finding the code in the AST by name is relatively straightforward, and one can use the prettyprinter function of such a tool to pretty print the named-entity into a buffer/string/diskfile wherever you want it parked.
You are not likely to find a PT as a subroutine you can call from C# directly. You may be able to invoke such a tool from your program, and have it return the string as a text result/in a file/via a pipe whatever you thinks is best for interprocess communication.
If you want to show the method text embedded in an HTML page, you can often configure the PT to generate the entire page containing the prettyprinted entity text. (See the JavaSource browser via my bio for an example like this).

The .NET Reflection class does not support decompiling to C#. The best you can do is use MMethodInfo.GetMethodBody() and call MethodBody.GetILAsByteArray() on the response. That will give you the MSIL which is the best that .NET reflection can give you.
To decompile to C# you will need a decompiler - of which there are at least a dozen legitimate options. You'll need to investigate for one that meets your requirements and budget.
One option is Red Gate's .NET Reflector. Nick Harrison has a long and thorough article on using .NET Reflector in an application to render source code as HTML.

How to find all methods in assembly that instantiate object of class X?

Interesting question here: In my current project we're using a custom performance monitoring suite that is very config-heavy (it uses Perfmon so we have to manually register each performance counter. There is one performance counter for each method that we monitor, and there are a lot of those).
I was wondering if there are any tools that would, say, allow me to analyse the project assembly, find all methods that instantiate an instance of class XClass, then write them out to a file? This would allow me to cut down the amount of manual config I need to do by a large margin.
Thanks, Ed
EDIT
Sorry, the 'write them to a file' thing was a little contrived: really I need to reformat them with some extra data and write them in a config-specific XML format. This would be best if I can code it up so it can be set as a build task (so I don't have to run it manually) and any future changes can be made easily and documented etc.

Open the assembly in Reflector (the free version is fine); find the type (F3), then bring up the anaylyzer (Ctrl+R) and expand the "Instantiated By" node.
Then right-click on the "Instantiated By" node itself and click copy; for example:
System.Data.SqlClient.SqlConnectionStringBuilder
Depends On
Used By
Exposed By
Instantiated By
SqlDependencyProcessDispatcher.GetHashHelper(String, SqlConnectionStringBuilder&, DbConnectionPoolIdentity&, String&, String) : SqlConnectionContainerHashHelper
System.Data.SqlClient.SqlClientFactory.CreateConnectionStringBuilder() : DbConnectionStringBuilder
System.Web.Management.SqlWebEventProvider.Initialize(String, NameValueCollection) : Void
System.Web.SessionState.SqlSessionStateStore.CreatePartitionInfo(String) : IPartitionInfo
System.Web.SessionState.SqlSessionStateStore+SqlPartitionInfo.get_TracingPartitionString() : String
System.Web.SessionState.SqlSessionStateStore+SqlStateConnection..ctor(SqlPartitionInfo, TimeSpan)

If you need to write code to automate your task, it's easy to do it with Mono.Cecil. As an example, this code searches through all the methods of the top level types of an assembly for the instantiation of a class Foo.Bar.Baz:
using Mono.Cecil;
using Mono.Cecil.Cil;
// ...
static void SearchMethod (MethodDefinition method)
{
foreach (var instruction in method.Body.Instructions) {
if (instruction.OpCode != OpCodes.Newobj)
continue;
var constructor = (MethodReference) instruction.Operand;
if (constructor.DeclaringType.FullName != "Foo.Bar.Baz")
continue;
Console.WriteLine ("new Foo.Bar.Baz in {0}", method.FullName);
}
}
static void Main ()
{
var module = ModuleDefinition.ReadModule ("Foo.Bar.dll");
var methods = module.Types.SelectMany (t => t.Methods).Where (m => m.HasBody);
foreach (var method in methods)
SearchMethod (method);
}

Take a look at NDepend. It's a static analysis tool with a very powerful query language: CQL (Code Query Language).
Update
NDepend has a console application that can be prodded for automation (e.g. for use in a build systems) and can output reports to file.
An example query to find methods which instantiate a defined type:
SELECT METHODS WHERE CreateA "MyNamespace.MyClass"

You probably want to change your design so that they use a factory instead, such that the factory is responsible for the extra bookkeeping.
That said, you can look in the System.Reflection namespace, (from memory), where the Assembly class is used.

reflection to list return types, parameters in C# application from specific namespace

I've been working on an API that encapsulates another, trickier-to-use API. The goal is for my API to not require the user to touch any of the old API by 1) not requiring any parameters of classes in the old API and 2) not returning any instances of classes in the old API. Is there a program, perhaps a Visual Studio plugin, that can analyze my C# solution and give me a list of all the return types from publicly accessible methods in publicly accessible classes, as well as all the parameter types in such methods? Otherwise it seems like I'll have to manually go through all my classes and see if any of the old API is exposed to the user.
Edit: Since I've been using MSTest for unit testing my API anyway, I added another unit test to use reflection and Fail if any parts of the old API are exposed. However, I'm stuck with a reflection problem. I have using OldAPI in the unit test class and then I use
AppDomain.CurrentDomain.GetAssemblies().SelectMany(
assembly => assembly.GetTypes()
)
to get a list of the types in all the assemblies currently loaded. I then iterate over those in hopes of paring down the list of types to only those in the namespace OldAPI. The problem is that the namespace OldAPI does not show up. I see namespaces like Microsoft.VisualStudio.TestTools, System.Reflection, and others that are included via using statements in the test class, but no "OldAPI". Could this be because of COM stuff with the old API, so AppDomain.CurrentDomain.GetAssemblies() doesn't include the assembly even though it's included via a using statement in the class?
Solution: I got the necessary assembly by arbitrarily choosing one class I know is in OldAPI and doing the following, thanks to SLaks' comment:
Func<Type, bool> isBad = t => t.Assembly == typeof(OldAPI.SomeClass).Assembly;
Here's a snippet of my unit test for checking if any of my API's classes use any of OldAPI's classes, thanks to SLaks' answer:
MethodInfo[] badMethods = methods.Where(
m => (
isBad(m.ReturnType) ||
m.GetParameters().Any(p => isBad(p.ParameterType))
) && !isBad(m.DeclaringType)
).ToArray();
string[] badMethodNames = badMethods.Select(
m => m.DeclaringType.Name + "." + m.Name
).Distinct().ToArray();
Assert.AreEqual(0, badMethodNames.Length, "Some methods in " +
monitoredNamespaces + " projects expose OldAPI: " +
string.Join(", ", badMethodNames));

You can use LINQ, like this:
Func<Type, bool> isBad = t => t.Assembly == badAssembly;
var types = yourAssembly.GetTypes();
var methods = types.SelectMany(t => t.GetMethods()).ToArray();
var badMethods = methods.Where(m => isBad(m.ReturnType)
|| m.GetParameters().Any(p => isBad(p.ParameterType);
var properties = types.SelectMany(t => t.GetProperties()).ToArray();
var badProperties = properties.Where(p => isBad(p.PropertyType));
This would be easiest to do in LINQPad.
Note that this doesn't traverse generic types, so it'll ignore a List<BadType>.
You should probably make isBad recursive. (In which case you should turn it into a regular function)

I'm not aware of an existing tool for this, but that doesn't mean you have to do it manually -- you can very easily write your own tool to do this using Reflection. Basically you'll just need to iterate over Assembly.GetExportedTypes(); for each type, call Type.GetMethods() and Type.GetProperties() and iterate over the results; and dumping the return and parameter types for each method or property that is public.
Note that such a handwritten tool would need to run over your compiled assembly, not your C# source code. You can do something similar to source code, but it depends on the Visual Studio code model which is rather harder to use and probably not worth the effort for a one-off like this!