This question is strictly about the inefficient architecture of an existing system that needs to be rebuilt. It solicits validation from fellow developers who have had experience with managing such awkward systems. I have tried to abstract it as best as I could below.
The application caters to a very complex need and it delivers very well. The problem is that internal plumbing makes code management and scalability a nightmare. The little information I can share about context includes the fact that we need to treat code as a data commodity. In other words, the system can only function if implemented classes are added to it on a continuing basis.
What the application delivers to end-users is not data, but an [Action] that requires a code execution context. So the application has to execute some code on the target system in order to deliver what the user expects. Now these expectations are not known at compile-time and new ones need to be added almost on a daily basis. That means, developers keep adding [Actions] to the system regularly.
The existing system links to these [Action] classes statically! Not only does that make code management a nightmare, but also requires a recompile every time an action is added.
My first instinct was to have the system dynamically link to assemblies at runtime where each assembly would contain a bunch of actions. This would be akin to adding extensibility capabilities to the application. I thought about the MEF framework but it just did not feel right.
The only alternative I can think of is storing each action in the database as either source code or a compiled module. Each has its own trade-offs such as storing as source is less secure but gives me more control over code review and continued maintenance. Storing as compiled has the benefit of server-side assembly signing.
I would appreciate some advice about how to structure a system like this.
I don't think you need a more flexible architecture, but a more flexible software process. Adding new functionality on a daily basis is what most developers do. That doesn't a valid argument for a plugin system.
You don't need a plugin architecture. You need a good software development methodology, such as the agile processes (such as Scrum and XP), and make sure you be able to do this:
Let developers build new components in braches.
After thorough testing, merge new functionality to the main branch.
This way the main branch always has production quality and you can roll out new versions each day using continuous integration and continuous delivery.
We have an ordering system that is currently quite hard coded. We would like to make this flexible, and allow our Business Analyst to create workflows himself. The application will be installed for separate clients so there will be separate workflows for each client.
We are a bit uncertain about the correct technology here. One option may be WF but this seems to be directed more towards developer use.
We were thinking to create 'activities' and through a screen the business analyst can arrange these accordingly in a workflow. This information would be saved in some XML format and then we would load the info and execute the activities.
What is the correct technology here please? perhaps a third party tool or is WF ok?
I'm not sure I can give you the correct technology for your situation, but give you a few pointers if you want to go down the WF4 route. It will require investing some time in learning WF4 in general and some of the complex features of Re-hosting a Workflow. To put it simply Re-hosting the Workflow in an application will let you provide an environment where your BA can work that does not force him/her to work in Visual Studio.
Take some time and invest in learning Wf4 and Re-hosting:
http://msdn.microsoft.com/en-us/library/cc835242.aspx
http://blogs.msdn.com/b/asgisv/archive/2010/02/06/rehosting-the-workflow-designer-is-so-much-easier-in-net4-a-boon-for-the-isv-community.aspx
I am currently writing a .net application with c# and want to check a number of rules and based on passing or failing the rules, perform an action. So I am looking to implement a generic solution that I can reuse adhering to good oop principles. This has lead me to the conclusion that I need to write a rules engine.
I have good knowledge of c# but this is the first time I have needed to write a rules engine so as part of my research in to the design and development of such, I am looking for any tips regarding the creation of such an engine. What would be great further would be any examples out there that I could look at? Any c#/.net rules engine applications? What layer in a typical 3 tier architecture should such reside at? I had a quick look on the codeplex and google code but none jumped out at me! So some direction would be great.
Actually .NET has a top-notch rules engine meant to be used with workflows (as it is designed to be) but can be used outside of workflows easily: You should see "Windows Workflow Foundation Rules Engine" and inspect the System.Workflow.Activities.Rules namespace.
Learning how to use rules outside of workflows takes only a bit of googling.
Edit: If you want to inspect the architecture, here is two open-source prebuilt engines:
NxBRE
.NET Application Block for Validation and Business Rules
Building and implementing your own rules engine can be a very difficult task with many things to consider. The biggest problem you will face is trying to decide which rules to fire and when. Not giving this the proper care can lead to performance problems within the implementation. I would highly recommend focusing in on the business problem and providing your subject matter experts (SME’s) with the ability to define and maintain their own rules. There are many good commercial products that do this; the one I have successfully implemented multiple times is www.inrule.com. They have a nice set of products that can help solve simple and complex problems. Hopefully this helps.
I have to create a workflow in C# capable of moving an object (persisted as a database record) through an approval workflow where people are required to perform some sort of action or validation.
We initially looked at Windows Workflow Foundation but shied away from it because it seemed so infrastructure-heavy (and besides we don't really like Microsoft products). We looked at ObjectFlow because it's lightweight, but I'm having trouble figuring out how to persist & resume workflow states. It almost seems like it's too lightweight.
Does anyone have a particular favorite framework for doing workflow? I'm open to ideas (even to WWF, if you can explain why it's your favorite).
Now at the end of 2022 I recommend Elsa Workflows library which is free and open source.In this case state machine workflows works. I have used it in multiple projects witch success. It is flexible, has a web workflow designer and acceptable documents
As the question #gsharp linked to says, WF 4 isn't entirely easy to use. However, ObjectFlow has an easy fluent interface that is light and built with solid design principles. Given the apparent lack of decent workflow frameworks, I decided to pitch in and extend ObjectFlow with an IStatefulWorkflow that contains a .Yield() method capable of yielding workflow processing to the calling method so that it's state can be persisted.
The end result of my work will be a new release at codeplex in a couple weeks. Until then, you can follow my progress at github.
Have you looked at Drools.Net?
Drools.NET is a Business Rules Engine (BRE) based on Charles Forgy's Rete algorithm. Developers can now exploit a powerful Rule Engine through a completely managed .NET code base! Drools.NET is based on Jboss Rules, and comes with all the features of that Rules Engine.
I have recently developed a C# Workflow library that leverages the fluent syntax and provides compile time validation between workflow steps. A Workflow's steps can be consolidated in a single location providing maintainable code. The library is very light weight and performant.
https://www.nuget.org/packages/NetWorkflow
https://github.com/Tmarndt1/NetWorkflow
I am responsible for a team of developers who will are about to start development of a light weight insurance claims system. The system involves a lot of manual tasks and business workflows and we are looking at using Windows Workflow (.NET 4.0).
An example of the business domain is as follows:
A policy holder calls the contact centre to lodge a claim. This “event” fires two sub tasks which are manually actioned in parallel and may take a lengthy time to complete;
Check customer for fraud – A manual process whereby an operator calls various credit companies to check and assess the potential of a fraudulent customer. From here the sub task can enter a number of sub-statuses (Check in progress, Failed Reference Check, Passed Reference Check, etc)
Send item to repairs centre – A manual process where the item for which the policy holder lodged the claim is sent the repairs centre to be fixed. From here the sub task can enter a number of sub-statuses (Awaiting Repair, In Progress, Repaired, Posted, etc).
The claim can only proceed once the status of each sub task has reached a predefined status (based on the business rules).
On the surface it seems that Workflow is indeed the best technology choice; however I do have a few concerns in using WF 4.0.
Skill set – Looking at the average developer skill set I do not see many developers who understand or know Workflow.
Maintainability – There seems to be little support within the community for WF 4.0 projects and this coupled with the lack of skill set raise concerns around maintainability.
Barrier to entry – I have a feeling that Windows Workflow has a steep learning curve and it’s not always that easy to pick up.
New product – As Workflow has been completely rewritten for .NET 4.0 I see the product as a first generation product and may not have the necessary stability.
Reputation – Previous versions of Workflow were not well received, considered difficult to develop with and resulted in poor business uptake.
So my question is should we use Windows Workflow (WF) 4.0 for this situation or is there an alternative technology (e.g., Simple State Machine, etc) or even a better workflow engine to use?
I have done several WF4 projects so lets see if I can add any useful info to the other answers.
From the description of your business problem it sounds like WF4 is a good match, so no problems there.
Regarding your concerns you are right. Basically WF4 is a new product and is lacking some important features and has some rough edges. There is a learning curve, you do have to do some things differently. The main point is long running and serialization, which is something the average developer is not used to and requires some thought to get right as I hear far too often that people have problems serializing an entities framework data context.
Most of the time using workflow services hosted in IIS/WAS is the best route when doing these long running type of workflows. That makes solving the versioning problem not to hard either, just have the first message return the workflow version and make that a part of each subsequent message. Next put the WCF router in between that routes the message to the correct endpoint based on the version. The basic is never to change an existing workflow, always create a new one.
So what is my advise to you?
Don't take a big gamble on a unknown, and for you unproven, piece of technology. Do a small, non critical, piece of the application using WF4. That way if it works you can expand on it but if it fails you can rip it out and replace it with more traditional .NET code. That way you get real experience with WF4 instead of having to base a decision on second hand information and you learn a new and powerful technology in the process. If possible take a course on WF4 as that will save you a lot of time in getting up to speed (shameless self plug here).
About the Simple State Machine. I have not used it but I was under the impression it was for short running, in memory, state machines. One of the main benefits of WF4 is the long running aspects.
I have come to this dilemma couple of times and I had chosen not to use Work Flow foundation. Some of considerations (similar to yours) were
Involved work flows were lot simpler (a combination of state machine and sequential actions) and doing it in WF seems to overkill for efforts involved.
Learning curve for developers to understand and to use WF effectively was considered high. Status transition table describing valid transitions and actions to be taken are used for additional flexibility and developers were comfortable with it, easily understanding the concept and purpose.
Chances of business process changes were slim and rudimentary changes were easily possible with help of transition table. A change in transition would mean a database script while change in actions would result in new release/patch. However, probability of such occurrence was deemed to be low.
Looking back after 13-14 months, I still think that decision of not using WF was correct. IMO, WF makes sense where there is strong likely hood that work flow can change and/or business rules can change. WF allows to isolate workflow in separate file and so making it configurable by users will be simpler.
We have been using WF 4.0 the last couple of months. I have to say it's challenging to think the Workflow way. However, I can tell you it's worth it. We knew very little when we started. We've bought a beginner and professional book for WF 4.0 that helped. I, myself, watched many videos online and followed PDC 2009 for their breaking news about WF 4.0 and how it's different from the previous somewhat sucky versions.
One major thing that we had to propose a solution for is the way we can deal with In/Our Arguments in a workflow without bounding our custom activities to certain data types and how to pass parameters between activities. I have come up with a good solution for that, and the workflow experience that we have so far is not bad at all. Actually, we have a workflow-intensive application that is getting bigger and bigger and I really cannot imagine myself solving it in a different environment. I love the visual effect that it has: it keeps me away from the details of if/else etc constructs and makes the business rules apparent in a way that doesn't make you forced to dive into lines of code to know what's going on or how to fix some bug.
By the way, the project that we worked on is very similar to what you described and it's a medium-sized project.
You can tell from my words that I like it and I do recommend it although is incorporates some risks as it's a new technology and you have to come up with some innovative ideas.
my 2 cents...
I did three projects in WF 3.5 and I have to say it is not easy. It force you to think in the whole new way especially when persistance is used. Updating the application which contains hundreds of incomplete persisted workflow is challenging. Single breaking change in serialization crashes them all. Introducing multiple versions of the same library to support new and old running workflows is common. It was challenging.
I haven't tryed WF 4.0 yet but based on experience from BizTalk and WF 3.5 I think it will be similar.
Anyway the best approach you can take is to do Proof-of-Concept. Take single WF from your requirments and try to implment it in WF 4.0. You will spend some time with it but you will prove if you are able to do that in WF 4.0 and if there are any visible benefits.
If you decide to use WF 4.0 I insist that you check possibility to run WF as WCF service hosted in Windows AppFabric. AppFabric provides some out of the box functionality for hosting WFs.
I think it does not really make sense today to talk about Workflow in WF4 as a technology choice for this kind of problem. What is really appropriate, as mentioned by Ladislav Mrnka above, is WCF WF Services hosted in AppFabric.
My experience with this is that it pays great dividends and is very enjoyable, but problems arise in the beginning because it is not properly appreciated that for many programmers this is a methodology shift more than a technology shift. On the other hand, generalists and those with a problem-solving mindset saw WCF WF AppFabric as a set of exciting opportunities. So if the mix of people on the project are fairly conservative C# devs attached to their daily set of OO and patterns, it will be hard to introduce. If the team is more innovative, then adoption will be much easier because the potential and new doorways multiply with each discovery.
Two main conceptual problems programmers had in moving to this technology was:
a) Message correlation and messaged exchange patterns
b) Workflows and unit testing
In standard systems in C# for example a workflow is rarely explicit and therefore rarely unit tested. The overall workflow is left for testing by acceptance scenarios or integration. Introduce an explicit WF as a software artifact and suddenly standard devs want to try and unit test it, which is usually not worth doing.
The message correlation aspect of it is a bit of mindset shift for those not familiar with message exchange patterns. Most devs have dealt with in process and remote calls, web service and SOAP, and usually focussed on one or two of those. To abstract above it all and work with a general message based system can be confusing at first.
On the positive side though, the end result is something that saves a lot of time and creates a lot of opportunities. One main thing is that the worfklow, if visually clear, is something that can be worked on by end user, developer and analyst together, eliminating unnecessary steps in the development lifecycle and focusing the parties on one artifact. Further, it discourages islands of functionality in dedicated apps, with dedicated glue layers, by encouraging a suite of business processes in WF per business domain. Further, with AppFabric, the plumbing for persistence, logging, and waking up scheduled activities is all done for you. WF4 performance is outstanding too.
My recommendation would be to find the most innovative or explorative team member do the initial scouting to discover the tricky parts, get the core functions working, and have that initial person be responsible for then compartmentalising the remaining work.
In order to do an insurance claim system of any complexity that involves roles and "sub-tasks" you really need an BPM solution, not just workflow. Workflow Foundation 4.0 is slick but it really doesn't not come close to the functionalities of a BPM product.
BPM solutions, like Metastorm BPM, Global360, and K2.NET, provide human centric workflow, tasks, roles, and system integration that can model and streamline the business processes like your insurance claim system. Use ASP.NET to build the interface that integrates with the BPM workflow engine as their built in designers are usually limited and force you to use their custom built web control which usually are not as full featured as the ASP.NET web controls.
Go with the technology your team knows and feels comfortable with. Workflow Foundation is not a product that you can use straight away - it's rather a set of pieces you can embed in your application in order to build a workflow system. IMHO the workflow logic is the least important piece of technology, first of all you have to concentrate on the GUI because business owners will not see anything but the GUI. But if your system is a success then you have to be prepared for neverending change requests and new requirements so you have to implement your business logic so that it's easy to change and easy to divide into separate processes to suit different user needs (sometimes contradicting). BPM helps in this task because it allows you to have separate, multiple versions of business processes suiting various business needs. You don't need full fledged BPM engine for that but it's useful to code your business logic so that it can be versioned and divided into individual business processes - the worst thing to have is an unmantainable and intertangled blob of code that handles 'everything' and that no one can understand. There are many ideas for that - state machines, DSLs (domain specific languages), scripts etc - you decide what the implementation should be. But you should always think in terms of business processes and organize your logic accordingly so that it reflects these processes.
And be prepared for coexistence of many variants of business logic and data structures - this is the most difficult design task imho.
I'm in a situation where I have to use 4.0 as .NET 4.5 isn't accredited for use in our prod environment yet. I had major pain understanding generally how to get long running workflows going to suit our business need but eventually found an elegant solution. It's not something which just anyone coming later to support can just pick up with ease because there's so much to think about, but I do believe in WF as a tool for managing workflow states.
One big thing I take issue with WF 4.0 though is Maurice's comment:
The basic is never to change an existing workflow, always create a new one
That's great if you just want a new version, but what if you have 50,000 persisted workflows and realise at some point that there's a bug in the workflow? You need to be able to update the xamlx and still be coupled to the existing instances. I've tried ungzipping the various metadata columns in the SQL Server instances table to find something that ties the instance to the workflow definition without any luck.
I did write a synchronisation application for importing data from an old system into our new WF 4.0 driven one. We basically load the data into the system, then run the process which goes about automatically calling into the workflow steps and calling validation methods, essentially mocking user interaction. This only really worked well with us due to the architecture we implemented for access to the workflow service host. It's great as a one off, where after running you can go through and do checks to ensure consistency of the data migration process, but having to use this approach for potentially hundreds of thousands of cases once a system is live isn't really an approach that instills confidence and over burdens the process of integration simple bug fixes.
My recommendation is that you avoid WF 4.0 altogether and just go straight to 4.5 if you're environment supports it. The Dynamic Updates and Side by Side Versioning it provides caters for bug fixing and WF versioning all out of the box. I've still yet to investigate exactly how as 4.5 still isn't accredited for use by our client, but eagerly awaiting this opportunity.
What I'm desperately hoping for is that our client doesn't request changes to policy (and therefore workflow adjustments) and that the current workflows hold up without any bugs. The latter being a vain and empty hope as bugs always pop up.
I really can't understand what was going through the WF dev team's heads to release a system where out of the box you can't fix bugs easily. They should have developed a technique for re-binding an instance to new xamlx.