Develop Reference

Why C++/WinRT requires IDL files for XAML?

For C#,, XAML transpiles to .cs (*.g.cs) files and need no IDL files.
Similarly in C++, Why can't XAML be transpiled to .cpp (*.g.cpp) files and Need any IDL files at all ?
I don't understand.

There's a fair bit of confusion in the question as to how the individual pieces fit together. The main driver for the translation is the IDL file. Irrespective of whether it is authored by a developer or synthesized by the IDE, it is the IDL that produces WINMD (Windows Metadata) files describing the interfaces and runtime classes in a language-agnostic fashion.
The WINMD's are used by all tooling that needs to look up types, query for members (such as properties, events, delegates), and produce application packages.
XAML, on the other hand, isn't part of the compilation process at all. While some of its contents are verified at compile time, it usually gets translated into a compact binary representation (XBF) that's loaded and parsed at runtime to instantiate types.
The question as to why IDL's are required with C++/WinRT (and not with C# or C++/CX) is easily answered: It's simply not possible to derive enough information from C++ class definitions to unambiguously deduce the required metadata.
As an easy example, consider properties. While both C# as well as C++/CX have dedicated language constructs to describe properties, this is not the case in C++. C++/WinRT projects properties to member functions that take zero or one argument (for getters and setters, respectively). If you were to automatically deduce metadata from a C++ class definition, a tool would have to disambiguate between a property and a method. There are other challenges, too, and while Kenny Kerr has repeatedly voiced the desire to get better IDE support for C++/WinRT, the Visual Studio team doesn't seem to care much (see this comment, for example).
For the foreseeable future you should be prepared to author IDL files if you choose to use C++/WinRT.

How to use protobuf's compiled static serialization dll?

So my concern here is when I'm targeting AOT platforms so no support for dynamic code generation. protobuf-net allows to compile the serializers before hand solving the problem. But I'm not quite sure how to use this features...
So I create a RuntimeTypeModel with all my types and members and then call model.Compile(name, path); - OK... what to do with this generated dll? include it and use its serialize methods and forget about my model object? or can I just serialize from the same model object? (i.e. if I do model.Compile if I then say model.Seiralize, will the model use the serializers in the compiled dll? or...?)
Any enlightenment on the subject would be great. I found bits and bytes on the Internets but no complete answer.

This depends a bit on:
whether you need cross platform support
whether you need custom configuration
It sounds like "no" and "yes" in that order, from the question. In which case you should be able to use the overload of Compile that accepts a path and serializer type name. This emits a DLL that you can reference from your project; do that, and then simply use the serializer type:
TypeModel serializer = new YourCustomSerializer();
This then had the methods to serialize / deserialize etc.
If you need cross-platform support, the tool to compile is more complicated, and requires using a special build step. The simplest way to do this is via the "protogen" tool, which uses attribute-based configuration. If you need cross-platform compilation and custom configuration, you need to write a small tool (not big) that references the ikvm protobuf-net; this takes a bit more explaining - let me know if you need an example!

How to load an existing assembly with Roslyn, transform it and generate new .cs files

It seems like the documentation around Roslyn is a bit lacking?
I am not able to find good comprehensive documentation.
What I am trying to do essentially is copy the public surface of an existing API (.dll)
into a new assembly (need to create source code .cs files!) and at the same time make a variety of tranformations to the resulting code (think making wrapper classes).
Would really appreciate any help in how I can use Rolsyn to load the initial SyntaxTree from an existing assembly and how to do those basic tranforms (for example exclude internal classes etc)

In the current Roslyn CTP there is a Roslyn.Services.MetadataAsSource namespace which can be used to convert an type's public interface to source code. This is what we implement the F12 "metadata as source" feature with. Now, it generates only a shell of source code which won't actually compile, so you'd have to use further APIs to munge the syntax tree into what you want. Alternatively, you could use the Roslyn.Services.CodeGeneration namespace to generate source from these symbols automatically. I should warn the MetadataAsSource namespace may go away in future versions of the API.
You can import symbols from metadata by creating an otherwise empty compilation with the metadata references you care about added, and then from that compilation browsing the type hierarchy from GlobalNamespace property, or calling Compilation.GetReferencedAssemblySymbol() and then digging through that. This is actually far better than using reflection, since it'll properly express the symbol model from the "C# perspective" instead of the "CLR perspective" -- reflection won't give you information for uses of dynamic, some default parameter values, etc.

It seems like the documentation around Roslyn is a bit lacking? I am not able to find good comprehensive documentation.
Roslyn is at the Community Technology Preview stage, so it's not surprising that its documentation is lacking. You can find some sources at Roslyn API documentation.
What I am trying to do essentially is copy the public surface of an existing API (.dll) into a new assembly (need to create source code .cs files!) and at the same time make a variety of transformations to the resulting code (think making wrapper classes).
Working with assemblies this way is not something Roslyn can do. But it seems for what you want, reflection for reading the assembly combined with Roslyn for writing the new code would work. But you would need to write all the code to translate from the reflection model to Roslyn's model (e.g. Type → TypeDeclarationSyntax, MethodInfo → MethodDeclarationSyntax, etc.).

Reading MSDN pages [duplicate]

What is the necessity for the GUID attribute? why don't just let the compiler handle this automatically?!

If the compiler handled this automatically, you'd end up with one of two situations.
A new GUID every time you compiled - since GUIDs are supposed to be published, this would fail.
Collisions - if the GUID was the same every time, based on (say) a Hash of the name, multiple projects would end up using the same GUID for different purposes.
The existing approach - an explicit GUID gives developers the power to control these as required.

These are attributes that matter a great deal to COM. Which was the predecessor of .NET and had its heyday in the nineties, before Java stole the show. .NET needed to be compatible with COM to have a chance of succeeding. Or in other words, you needed to be able to write a COM server in a .NET language that a large legacy program could use.
The [ComVisible] attribute ensures that a COM client program can see and use the IEnumerable interface. Essential to allow the client program to enumerate .NET collections.
The [Guid] attribute is crucial in COM, it identifies an interface. Which is done by a guid, not a name, to ensure that it is unique across multiple applications written by different programmers. .NET has this too, but however uses a name to make it easier on humans. "System.Collections.IEnumerable, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089".
IEnumerable<>, the generic version, doesn't have a [Guid]. Generics are not compatible with COM. It doesn't much matter these days, not much visible COM around anymore, most of it has been wrapped by friendly .NET classes. But still very core in Windows, notably in the brand-new WinRT (aka Metro, aka Modern UI, aka UWP). You don't use that directly either, making COM somewhat like the assembly language of Windows programming.

You can do it (just omit the attribute) but then the compiler will generate a new GUID on each recompile even if the interface has not changed. That's unfortunate because the users of that interface don't know about the change and will retrieve the interface by it's old GUID and will therefore fail to retrieve it.

Sometimes you want to give certain classes or modules a unique identifier that is constant and hard coded inside your source.

To read this definition you would need to look up the meaning of each of those attributes. The first, ComVisibleAttribute, is described as this:
Controls accessibility of an individual managed type or member, or of all types within an assembly, to COM.
That tells us that ComVisible is something to do with COM, and lets us specify whether a particular type is visible to COM programs. Further down on the page is a link to more details on what the attribute is for and how its used by the type library exporter.
The second, GuidAttribute, is a bit less helpful at first:
Supplies an explicit System.Guid when an automatic GUID is undesirable
but again, you have to read the rest of the way down, and you will see another mention of the type library exporter.
Putting these two together, it starts to become clear that these two attributes control how IEnumerator is processed when exported to a type library. If you don't know what a type library is, this will probably not mean much to you. If you are not using COM interop, then those attributes can safely be ignored. If you are using COM interop, you would need to know the Guid to properly access the interface from unmanaged COM code.
Microsoft puts these on every interface definition in case you need them; part of the skill in reading the MSDN pages is to recognize this type of information and know when it isn't any use to you. Now that you know what those two attributes are for, you should be able to figure out if they are relevant to you, and ignore them otherwise.

Attributes in C#

I know that C# (and .NET in general) is big on attributes. However, despite the fact I have programmed in C# for many years, I haven't found myself ever using them. Would someone get me started on them, and explain where is the best to use them?
Thanks

From Pro C# 2008 and the .NET 3.5 Platform, Fourth Edition by Andrew Troelsen
Understanding Attributed Programming
One role of a .NET compiler is to generate metadata
descriptions for all defined and referenced types. In addition to this standard metadata contained
within any assembly, the .NET platform provides a way for programmers to embed additional
metadata into an assembly using attributes. In a nutshell, attributes are nothing more than code
annotations that can be applied to a given type (class, interface, structure, etc.), member (property,
method, etc.), assembly, or module.
The idea of annotating code using attributes is not new. COM IDL provided numerous predefined
attributes that allowed developers to describe the types contained within a given COM server.
However, COM attributes were little more than a set of keywords. If a COM developer needed to
create a custom attribute, he or she could do so, but it was referenced in code by a 128-bit number
(GUID), which was cumbersome at best.
Unlike COM IDL attributes (which again were simply keywords), .NET attributes are class types
that extend the abstract System.Attribute base class. As you explore the .NET namespaces, you will
find many predefined attributes that you are able to make use of in your applications. Furthermore,
you are free to build custom attributes to further qualify the behavior of your types by creating a
new type deriving from Attribute.
Understand that when you apply attributes in your code, the embedded metadata is essentially
useless until another piece of software explicitly reflects over the information. If this is not the case,
the blurb of metadata embedded within the assembly is ignored and completely harmless.
Attribute Consumers
As you would guess, the .NET 3.5 Framework SDK ships with numerous utilities that are indeed on
the lookout for various attributes. The C# compiler (csc.exe) itself has been preprogrammed to
discover the presence of various attributes during the compilation cycle. For example, if the C#
compiler encounters the [CLSCompliant] attribute, it will automatically check the attributed item to
ensure it is exposing only CLS-compliant constructs. By way of another example, if the C# compiler
discovers an item attributed with the [Obsolete] attribute, it will display a compiler warning in the
Visual Studio 2008 Error List window.
In addition to development tools, numerous methods in the .NET base class libraries are preprogrammed
to reflect over specific attributes. For example, if you wish to persist the state of an
object to file, all you are required to do is annotate your class with the [Serializable] attribute. If
the Serialize() method of the BinaryFormatter class encounters this attribute, the object is automatically
persisted to file in a compact binary format.
The .NET CLR is also on the prowl for the presence of certain attributes. Perhaps the most
famous .NET attribute is [WebMethod]. If you wish to expose a method via HTTP requests and automatically
encode the method return value as XML, simply apply [WebMethod] to the method and the
CLR handles the details. Beyond web service development, attributes are critical to the operation of
the .NET security system, Windows Communication Foundation, and COM/.NET interoperability
(and so on).
Finally, you are free to build applications that are programmed to reflect over your own custom
attributes as well as any attribute in the .NET base class libraries. By doing so, you are essentially
able to create a set of “keywords” that are understood by a specific set of assemblies.
Applying Attributes in C#
The .NET base class library provides a number of attributes in various
namespaces. Below is a snapshot of some—but by absolutely no means all—predefined
attributes.
A Tiny Sampling of Predefined Attributes
[CLSCompliant]
Enforces the annotated item to conform to the rules of the Common
Language Specification (CLS). Recall that CLS-compliant types are
guaranteed to be used seamlessly across all .NET programming languages.
[DllImport]
Allows .NET code to make calls to any unmanaged C- or C++-based code
library, including the API of the underlying operating system. Do note that
[DllImport] is not used when communicating with COM-based software.
[Obsolete]
Marks a deprecated type or member. If other programmers attempt to use
such an item, they will receive a compiler warning describing the error of
their ways.
[Serializable]
Marks a class or structure as being “serializable,” meaning it is able to persist
its current state into a stream.
[NonSerialized]
Specifies that a given field in a class or structure should not be persisted
during the serialization process.
[WebMethod]
Marks a method as being invokable via HTTP requests and instructs the CLR
to serialize the method return value as XML.
Building Custom Attributes
The first step in building a custom attribute is to create a new class deriving from System.Attribute. Example:
// A custom attribute.
public sealed class VehicleDescriptionAttribute : System.Attribute
{
private string msgData;
public VehicleDescriptionAttribute(string description)
{
msgData = description;
}
public VehicleDescriptionAttribute() { }
public string Description
{
get { return msgData; }
set { msgData = value; }
}
}
As you can see, VehicleDescriptionAttribute maintains a private internal string (msgData)
that can be set using a custom constructor and manipulated using a type property (Description).
Beyond the fact that this class derived from System.Attribute, there is nothing unique to this class
definition.
For security reasons, it is considered a .NET best practice to design all custom attributes as sealed. In
fact, Visual Studio 2008 provides a code snippet named Attribute that will dump out a new System.
Attribute-derived class into your code window.

Attributes get more use in code targeted to other programmers or between distinct parts of a program, rather than code targeted at end users.
For example, you could use attributes to import a dll, indicate how types would interact with visual studio (designer visible, intellisense helps, debugger step-through, etc), how to serialize them, indicate a type is obsolete, describe default values, descriptions, handle COM access, etc.
Those are things that are largely invisible to the end user and that a single programmer could put elsewhere in the source code. But they're useful when only the compiled binary is available and not the source.

I like to use attributes as metadata to my code. We have created some simple attributes that let us tag who wrote what code, when, and why. This lets us have both documented changes in code and in runtime. If there are any exceptions during runtime, we can inspect the callstack, look at any attributes on the methods along the way, and track down the people responsible:
[Author("Erich", "2009/04/06", Comment = "blah blah blah")]
public void MyFunction()
{
...
}
Of course, we could use our source control to look at who checked in what code, but this I've found makes the information more available in the place where you need it. Also, if we ever change source control, that information will not be lost since it is persisted in code.

Attributes are a form of declarative programming, 'similar' to creating your UI in XAML. It 'marks' pieces of code (classes, methods, properties, whatever) with an attribute so that you can later gather all those pieces marked in a specific way and then do something standard with all of them.
Eg. Consider the scenario where you have certain sections of code that you want to run once each time your app starts. In one model of programming (non-attribute) you go to your main method and explicitly call those init methods. With attributes you simply gather all methods which you've marked with your 'init' attribute, and call them via reflection.
The same pattern holds for actions like serialization, persistence and whatnot...

I believe you mean that you do not use (or frequently use) custom defined attributes ?
In my current project, I make heavy use of custom attributes, but, the fact that you need to keep in the back of your mind, is that using attributes should not be a goal on itself.
It is a tool / purpose to get to a given solution.
I sometimes use custom attributes in combination with a weaver like PostSharp, to decorate methods where some weaving should be applied to at compile-time.
In my current project, I also use attributes to decorate certain types with additional info ... But I believe I've posted about this here before:
Cool uses of Attributes or Annotations (CLR or Java)?

I use attributes for the following:
Communicating with a plug-in architecture
Telling another framework what to do with the code (NUnit, for instance)
Adding metadata for use with other code (See PropertyGrid)
Mapping objects to databases (See Castle ActiveRecord)
When writing my own APIs to allow users to communicate metadata
In framework code to tell the debugger to skip over it
Those are off the top of my head. I use them in many other places

Attributes are very good at describing some runtime behaviour of your code that is orthoganal to the code in question. For example, in a class called Customer you would model a customer, right? But you might not want to model or describe the way a Customer object is serialized.
Adding attributes to your Customer class allows you to tell some other part of the runtime how it should deal with your Customer.
MSTest and NUnit makes use of attributes to tell the test framework how it should use classes that define test fixtures.
ASP.NET MVC uses attribute to tell the mvc framework which methods on classes it should treat as controller actions.
So, any place where you have a runtime behaviour that you wish to model attributes can be useful.

Class Attribute definition is available here
ClassInterfaceAttribute : Indicates the type of class interface to be generated for a class exposed to COM, if an interface is generated at all.
ComDefaultInterfaceAttribute : Specifies a default interface to expose to COM. This class cannot be inherited.
ComVisibleAttribute: Controls accessibility of an individual managed type or member, or of all types within an assembly, to COM.