Develop Reference

Dynamic Assembly Resolution/Management

Short Version
I have an application which utilizes a plug-in infrastructure. The plug-ins have configurable properties that help them know how to do their job. The plug-ins are grouped into profiles to define how to complete a task, and the profiles are stored in XML files serialized by the DataContractSerializer. The problem is when reading the configuration files, the application deserializing has to have knowledge of all of the plug-ins defined in the configuration file. I'm looking for a way to handle the resolution of unknown plug-ins. See the proposed solution section below for a couple of the ideas I've looked into implementing, but I am open to just about anything (though I'd rather not have to reinvent the application).
Detail
Background
I've developed a sort of Business Process Automation System for internal use for the company I'm currently working for in C# 4. It makes exhaustive use of 'plug-ins' to define everything (from the tasks that are to be performed to the definition of units of work) and relies heavily on a dynamic configuration model which in turn relies on C# 4/DLR dynamic objects to fulfill jobs. It's a little heavy while executing because of its dynamic nature but it works consistently and performs well enough for our needs.
It includes a WinForms configuration UI that uses Reflection extensively to determine the configurable properties/fields of the plug-ins, as well as, the properties/fields that define each unit of work to be processed. The UI is also built on top of the BPA engine so it has a thorough understanding of the (loose) object model put in place that allows the engine to do its job, which, coincidentally, has led to several user experience improvements, such as, ad-hoc job execution and configure-time validation of user input. Again there is room for improvement, however, it seems to do its job.
The configuration UI utilizes the DataContractSerializer to serialize/deserialize the settings specified, so any plug-ins referenced by the configuration must be loaded before (or at the time of) configuration load.
Structure
The BPA engine is implemented as a shared assembly (DLL) which is referenced by the BPA service (a Windows Service), the Configuration UI (WinForms app), and a plug-in tester (Console application version of the Windows Service). Each of the three applications that reference the shared assembly only include the minimum amount of code necessary to perform their specific purpose. Additionally, all plug-ins must reference a very thin assembly which basically just defines the interface(s) that the plugin must implement.
Problem
Because of the extensibility model used in the application, there has always been a requirement that the config UI is run from the same directory (on the same PC) as the Service application. That way the UI always knows about all of the assemblies that the Service knows about so they can be deserialized without running into missing assemblies. Now that we are getting close to roll out of the system, a demand to allow the Configuration UI remotely on any PC in our network has come about from our network admins for security purposes. Typically this wouldn't be a problem if there was always a known set of assemblies to deploy, however, with the ability to extend the application using user built assemblies, there has to be a way to resolve the assemblies from which the plug-ins can be instantiated/used.
Proposed (potentially obvious) Solution
Add a WCF service to the Service application to allow the typical CRUD operations against the configurations which that instance of the service is aware of and rework the configuration UI to act more like SSMS with a Connect/Disconnect model. This doesn't really solve the problem so we would also need to expose some sort of ServiceContract from the Service application to allow querying of the assemblies it knows about/has access to. That's fine and fairly straight forward however the question arises, "When should the UI find out about the assemblies that the Service is aware of?" On connect we could send all of the assemblies from the Service to the UI to ensure that it always knows about all of the assemblies the service does but that gets messy with AppDomain management (potentially unnecessarily) and assembly version conflicts. So I suggested hooking into the AppDomain.AssemblyResolve/AppDomain.TypeResolve events to only download the assemblies that the client isn't aware of yet and only as needed. This doesn't necessarily cleanup the AppDomain management issues but it definitely helps address the version conflicts and related issues.
Question
If you've stuck with me this long I applaud and thank you, but now I'm finally getting to the actual question here. After months of research and finally coming to a conclusion I am wondering if anyone here has had to deal with a similar issue and how you dealt with the pitfalls and shortcomings? Is there a standard way of handling this that I have missed completely, or do you have any recommendations based on how you have seen this successfully handled in the past? Do you see any problems with the proposed approaches or can you offer an alternative?
I'm aware that not everyone lives in my head so please let me know if you need further clarification/explanation. Thanks!
Update
I've given MEF a fair shake and feel that it is too simplistic for my purposes. It's not that it couldn't be bent to handle the plug-in requirements of my application, the problem is doing so would be too cumbersome and dirty to make it feasible. It is a nice suggestion and it has a lot of potential, but in its current state it just isn't there yet.
Any other ideas or feedback on my proposed solutions?
Update
I don't know if the issue I'm encountering is just too localized, if I failed to properly describe what I am trying to achieve, or if this question is just too unreasonably long to be read in its entirety; but the few answers I've received have been subtly helpful enough to help me think through the problem differently and identify some shortcomings in what I am after.
In short, what I'm trying to do is take three applications which in their current state share information (configuration/assemblies) using a common directory structure, and try to make those applications work across a network with minimal impact on usability and architecture.
File shares seem like the obvious answer to this problem (as #SimonMourier proposed in the comments), but using them translates into lack of control and debugability when something goes wrong. I can see them as a viable short term solution, but long term they just don't seem feasible.

tl;dr, but I'm 90% sure you should take a look into MEF.
When I first saw it I was like "aah, another acronym", but you'll see it's very simple, and it's built in into .NET 4. Best of all, it even runs seamlessly on mono and it's a matter of less than an hour (including coffee break) between hearing about it and compiling hello worlds to get used with the features. It's really that simple.
Basically, you "export" something in an assembly and "import" it into another (all via simple attribute decorations), and you choose where to search for it (example, on the applications directory, plug-ins folder, etc).
Edit: what if you try to download and load (and possibly cache) plugins on-the-fly on configuration load?

I think that you could be overlooking a relatively simple solution that derives somewhat from the Microsoft web.config approach:
Have two sections in the config file:
Section 1 contains enough information about the plugin (i.e. name, version) to allow you to load it into an app domain.
Section 2 contains the information serialized by the plugin.
On loading the plugin, pass the information in section 2 and let the plugin deserialize it according to its needs.

Maybe you can divide this problem into two
administrator allow users to download one of predefined configuration (set of libraries) and MEF helps to inject required dependencies
each activity from user should pass through security proxy, plugin modules not allowed call BL directly. Proxy could match custom security attribute and allowed activities.
i.e.
[MyRole(Name = new[] { "Security.Action" })]
void BlockAccount(string accountId){}
[MyRole(Name = new[] { "Manager.Action" })]
void CreateAccount(string userName){}
[MyRole(Name = new[] { "Security.View", "Manager.View" })]
List<> AcountList(Predicate p){}
and allow for AD groups (some abstract description)
corp\securityOperators = "Security.*" //allow calls to all security manipulation
corp\HQmanager = "Manager.View" //allow only view access
corp\Operator = "Manager.*"

I'm not sure I completely understand the problem but I think this situation calls for "type-preserving serialization" - that is, the serialized file contains enough type information to deserialize back to the original object graph without any hints from the calling application as to what types are involved.
I've used Json.NET to do this and I can highly recommend the library for type-preserving serialization of object graphs. It looks like the NetDataContractSerializer can also do this, from the MSDN Remarks
The NetDataContractSerializer differs from the DataContractSerializer in one important way: the NetDataContractSerializer includes CLR type information in the serialized XML, whereas the DataContractSerializer does not. Therefore, the NetDataContractSerializer can be used only if both the serializing and deserializing ends share the same CLR types.
I chose Json.NET because it can serialize POCOs without any special attributes or interfaces. Both Json.NET and the NetDataContractSerializer allow you to use a custom SerializationBinder - in here you could put any logic regarding loading assemblies that may not yet be loaded.
Unfortunately, changing serialization schemes might be the "breaking-est" change to suggest because all your existing files will become incompatible. You might be able to write a conversion utility that deserializes a file using the old method and serializes the resulting object graph using the new method.

Automatically marking c# classes for serialization

I want to build a visual studio plugin that automatically annotates classes for serialization. For example for the built in binary serializer I could just add [Serializable] to the class declaration, for WCF it could add [DataContract] to the class and [DataMember] to the members and properties (I could get [KnownType] information through reflection and annotate where appropriate). If using protocol buffers it could add [ProtoContract], [ProtoMember] and [ProtoInclude] attributes and so on.
I am assuming that the classes we are going to use this on are safe to serialize (so no sockets or nonserializable stuff in there). What I want to know is what is the easier way to take an existent piece of code (or a binary if that's easier) and add those attributes while preserving the rest of the code intact. I am fine with the output being source code or binary.
It comes to mind the idea of a using a C# parser, parse everything find the interesting code elements, annotate them and write back the code. However that seems to be very complex given the relatively small amount of modifications I want to make to the code. Is there an easier way to do so?

Visual Studio already has an API for discovering and emitting code which you might take a look at. It's not exactly a joy to use but could work for this purpose.

While such a plugin would certainly be a useful thing, I would consider rather making an add-in for a tool like ReSharper instead of VS directly. The advantage is somebody already solved the huge pile of problems you haven't even dreamed of yet and so it will be a lot easier to build such a specific functionality.

it looks to me like you need to have a MSBuild task similar to this one http://kindofmagic.codeplex.com/. is that about right?

How to Make a "Dashboard" Utility that Accepts Modules/Add-Ins

I hope this question makes sense. Basically, I am looking for a set of guidelines, or even a tutorial, that will show how to make an application that can easily add and remove "modules" or "add-ins"
For example, in Microsoft Office, you will commonly see programs that you can download and install and they will just add an extra tab into Microsoft Word (for example) that will implement some new feature.
I have several applications that use basically the same data source, and I'd like to consolidate them and also leave open the possibility of adding more functionality in the future without 1. Requiring a brand new install and 2. Tweaking every piece of my code.
I'm looking for a place to start, mostly.
Thanks in advance.
**
Edit: To elaborate a little more...
The thing I have in mind specifically is an application that accesses a large set of data that is stored in text files and uses some of the data to create a few graphs and maybe some tables. I'd like the ability to add different graphs in the future using the same data. So, you can click Button_A and generate Graph_A, then a few weeks later, you can click Button_B and generate Graph_B.
It would be really nice if I could come up with a way that only required reading the data from the file(s) once, but I know that would involve having to adjust my DataReader class a bit.

One place to start would be to define an interface for your future modules, and build a utility that scans all the dll's therein, looking for classes that implement said interface.
Once you've found supporting classes you can create instances at runtime and add to your application. That's a common idiom in .NET for supporting "plug-ins"
The Activator class is a common way to create instances from a Type at runtime.
http://msdn.microsoft.com/en-us/library/system.activator.aspx
It's hard to give more details without more info in your question. Can you elaborate a bit?

Take a look at the Composite Application Library from Microsoft.
It is aimed at WPF but you could get some ideas from there.

As Adam said, the first thing to do is define the interface for your plugin modules - what can they expect to receive from the container, and what methods must the container be able to call?
As far as the container itself goes, I'm partial to MEF as a location technology; you can create catalogs and re-compose the system when new DLLs are added. I've built a similar system to this for parsing dissimilar files, and the composition capabilities of MEF are awesome for runtime discovery.

.jar to .cs class/file transform technique or utility?

I am looking for a way to transform some classes from Java to .Net in a code gen way.
Not at run time, but re-generate a handful of business objects as needed (not often).
the catch is, i want to have full control of how they end up. So while java classes have get and set methods, i will create a property out of them.
the only way i can think of for now, is to read the file using c#, get the necessary members, and maybe feed them to a code gen template. at the very lease, i could throw them into a database, and something like code smith could generate .net classes based on a template and the updated database table.
any other ideas/utilities?
my solution was to use reflection to write out java class members as xml, and than plugging that into codesmith template to create .net classes. the neat part is that it can be automated as a step on build server

Have a look at IKVM.

Creating/compling .net data class within an application

Is there a pattern, Xml structure, architecture technique we can use to create simple data holder class code that we can deserialise to/from and do that at runtime?
We're in the early design stage of a .Net project and one of our goals is to make the resulting system extensible through configuration without needing a developer. We need to support new sources of data, typcially delivered as Xml messages. Currently we convert/deserialise the messages into simple classes and then use an already existing language which can manipulate those classes as we need.
That works well when you have a developer to map the Xml to simple class, create the class and then write the deserialisation, but it's not extensible for for an administrator. Our target user audience is high end DBA and/or network admin - people who can handle Xml but may not know C#.

You don't have to write any classes or deserialization routines. If you have a schema, you can use the XSD.exe tool from Visual Studio to automatically make Classes, and use built in .NET XML Serialization/Deserialization.
Now how to have that happen without a recompile each time...
It's not ideal, but this should work:
Assume your DBA can write a schema for the XML.
You could write a tool that takes the schema, runs it through XSD, Add's some wrapper code on top of it, and creates a dll which can be used from within your application.
This could be a manual process (ie the admins email you the schema) or you can distribute the tool as part of your application.
Also, you can infer a schema from an existing XML document.

Perhaps DataTable? Or just use an xml DOM (XmlDocument or XDocument) as data-storage? Neither is ideal, of course - but there is little point creating a type at runtime just for this if your real code will ever see it. What purpose would the extra class type serve? Among other issues you'd have to use lots of reflection just to talk to it.
The other option is a custom property-bag and IXmlSerializable, but that is effort.