I'm currently trying to convince management of the need to port one of our applications to .NET. The application has grown to be a bit of a monster in Access (backend in SQL), with 700 linked tables, 650 forms/subforms, 130 modules and 850 queries.
I pretty much know all the major benefits of doing this, but now need to look at how this can be achieved technically, so I can put a project plan together.
So, my plan was to convert the queries into stored procedures and/or views on the backend and re-write the forms in WPF or WinForms.
Now, the code is where I come unstuck. Is it possible to packaged up the code behind and modules into dlls and consume them whilst it is slowly ported to VB/C#?
What we can't be left with is half an application in VB/C# and half in Access, it must 'appear' to all be one application, even half way through the migration.
Thanks in advance.
EDIT: Just some more info about what we do and why we're looking at moving away from Access.
We are essentially an ISV and the Access application is our main product. This application has been developed over a period of 15 years, by many, many developers on an ad hoc basis. There is no documentation for this application.
We also have problems with getting branching in SCC to work properly, so we've currently got 4 or 5 code bases for the half a dozen clients we have. On top of that, all the testing we do is completely manual, which you can imagine is very labour intensive, and only scratches the surface of what really needs to be tested.
We're currently looking to expand, and have a number of sales leads that are in the final stages. I'm worried that with these new sales, we're going to be swamped with support and testing, and that this application is going to become even more entangled an buggy.
I'll also add to this the fact that we're just about to enter the spec phase of a brand new product, which is almost certainly going to be built in .NET. If we were to rewrite the Access application in .NET, then the people we use for that can go straight on to this new development. If we were to stay in Access, then we'd have to get some new Access people in, whom would have to be retrained once we start the new development.
So essentially it has come down to two choices, major refactoring work in Access to try and 'organise' the code a bit better, and those of you who have suggested culling parts are most probably right; I'm sure there are parts that are no longer used. However, I fear that if we stay in Access we still won't be able to build in effective testing and we still won't have proper SCC branching, which will lead to support continuing to be a nightmare, and any future developments on this product makings things worse. Either way there is a lot of work that we're about to embark on, which is either going to be done in Acces, or .NET.
I been involved in a lot of migration projects where one is converting from one platform to another. I've also seen spectacular cost overruns, and spectacular under estimations of how difficult these types of projects can be.
In some of the projects and platforms I've created and that I had built for about $25,000, the cost of replacing this application and rewriting it resulted in the other team of people taking over this project and the resulting cost was in excess of $750,000.
You're also making an assumption that the current system needs to be replaced. You MUST have clear in your mind what the actual goals of moving and replacing the current platform and software are. Simply rewriting something and moving the functionality over to another platform yields you very little, except spending a lot of money that really doesn't benefit your business of all (but hey those developers will take the money, if they convince you of the need of doing this)
You might want to take a read of this wonderful article by Joel on software (Joel by the way developed and created this forum stack overflow – and I was a moderator of some of his discussion forums for almost 10 years),. In this article, Joel warns and gives caution against out of the blue simply rewriting perfectly good software that does not rust or wear out.
Things You Should Never Do, Part I
by Joel Spolsky
They did. They did it by making the single worst strategic mistake that any software company can make:
They decided to rewrite the code from scratch.
Article here: http://www.joelonsoftware.com/articles/fog0000000069.html
Joel continues to note that in the past 10 years, that article still remains one of his more popular ones (and somewhat controversial)
It makes no sense to take a perfectly good running application that's been running great for 10 years and doing its job, and then simply rewrite it in another platform, especially if you don't have the Manpower and expertise and personnel available to maintain this new system. And this is especially more so, if the new systems not going to accomplish anything more than what the previous system was doing. In fact if you did have that Manpower, they'd likely already have STARTED converting this system already over time. I mean why all of sudden out of the blue did someone throw a light switch and all of a sudden realize that new developers need to be brought in to rewrite a system that's already been running fine?
I'll also point out having been in this business for a long time (both published and technical editor of access books) I also done migration projects from mainframe systems to desktops . And, I also done migration of desktop database systems to large mainframe systems .
I can only say that it is rare to see an application with that many tables. In fact this issue raises alarm bells right off the bat.
Because of such a large number of tables here, I have to think that there's likely either very many processes, and multiple applications cobble together here that represents this whole system . If this is not the case, then of course rewriting in .net does not make sense unless you address the un normalized nature of the system. The fact that the data is already in SQL server helps, but that just might mean that you had the horsepower and capacity and infrastructure to scale something that was poorly designed and the first place
A very big major portion of software flexibility comes from having properly normalized data models. The problem you have is that you have the data in SQL server, and it's very tempting to rewrite parts of the forms and functionality as .net forms, and continue to use the current existing data models. Unfortunately this put you in a bit of a rock and hard place, because you want to continue to use existing data, and start rewriting functionality in .net. However rewriting of functionality in .net without addressing the data models is a very bad idea.
In an ironic twist of fate, this is a catch 22, because likely if that system had really fantastic great designed data models, you might not even have the need to redesign and move this into .net. Access and SQL server can scale out to 100's of users with ease anway. And, access supports the use of class objects, and even source code control.
In other words keep in mind that people might be asking to rewrite this in .net because they believe the application will then magically have increased flexibility, and be able to be changed faster then that of their changing business rules. In fact often the opposite occurs, because access is a very RAD tool. This means that the frontline people can often make modifications due to business rules changing, faster than the IT Dept and their team of developers working away on the next great version of the application. And worse you don't want to saddle that IT Dept and those developers with a poor data model.
I mean, are you to now hire the IT department to build every single spreadsheet and excel for the people because are current business processes are not flexible enough? It would be wonderful if the IT Dept to go around to everybody's desk, hold their hand, and build the excel sheets CORRECTLY for everybody, but it's not practical in the real world. So in addition to taking access away from these people, you might as well take excel away from them also.
I am just pointing out that my spider sense suggests to me that the data models here are going to be a real challenge. Remember, I would always take poor code and great designed data models over the reverse (Great code, but terrible data models). The reason for this is with great data designs, then the code and applications practically write themselves. And with great data models, then the ease of which you can change for the ever changing business rules again favors great data designs over that of great code. You can also RE factor the code overtime WHEN you have good data models. So, with good data models you can move forms and functionality and the UI over into .net, and you can do this seamlessly and easily WHEN the existing data models make sense to keep.
Also it makes no sense to move to these new technologies and less you're going to introduce the possibility of introducing things like self serve web portals for the existing business processes. So, today we can now allow customers to manipulate and use some of that information that is currently locked up in the system. This might be simple as them checking the status of their orders instead of wasting valuable customer phone time. Or it might be something simple like how a major package company in Canada saved an estimated $10,000,000 in the first year of implementing their package tracking system. Or might be something as simple as allowing the customer to look up their account balance.
So right now in the marketplace, these self serve customer web portal systems allow customers to enter and use and get at their information instead of the calling up some employed within the organization who then turns around and launches the application and then manipulates the information for that customer while on the phone. Might just as well let the customer do this work! So from order status, to balances owing, to banking, or what ever it is, the real cherry model ticket today is to allow the customer to use at a cell serve web portal that represents all that valuable information that that internal application is creating .
As mentioned, you have to ask, is where is the Manpower and personnel going to come to build and maintain this application? Obviously the existing system with enormous numbers of forms and tables you are throwing out must somehow been created, and represents some significant investments of time and efforts . The key concept you have to ask, is where were those significant investments and resources coming from to build that existing application? Who is going to maintain the new system then? In other words you need to design the new system to reduce maintenance costs. (new versions of my software can reduce maintenance by as much as 10 or 15 hours per year for customer ).
At the end of the day, good software development and good designs are good designs. Using Access, or VB6, or vb.net don't matter if the system is meeting your business needs now.
I should also point out that the new version of access 2010, can create .web forms. They are XAML (zammel .net forms). I am pointing this out, because changing the front end skin from access to .net yields you VERY little unless the underlying data structures and designs are also modified to take advantages of new possible business processes that can be accomplished with new technologies (such as those cell serve web portals). Simply repaint the font ends with .net forms is really very much amounts to a waste of money in my humble opinion UNLESS other issues are being addressed such as the data model, or some type of web portal that'll improve flexibility here.
You have some great advice here already. Keep in mind this really comes down to what are the goals and reasons for these people desiring this software to be rewritten in .net. Those new goals and desires better not be based on the pretext of simply remaking the forms you have now into .net as that will really accomplish nothing at all, and will not improve their ability to address their changing business needs that the system they are currently using obviously had been doing in the past.
Good luck on this I don't think this is the kind of question that can be answered in a simple forum post, but at least you have lots to chew on here so you can get the ball rolling.
Instead of telling you how hard it is -- which I'm sure you already realize -- I'll try to toss out some hints:
1) Start by moving everything you can out of MS Access and into MS SQL. This means tables, stored procedures, views, etc. If you get this step right, your MS Access app will be a front-end for a real database, which is already a win.
2) Consider giving up before you start. Instead of porting everything, it might make more sense to recognize which features can just be left alone, while new ones get WCF or MVC front-ends.
3) It's tempting to port from VBA to VB.NET, on the theory that it's more similar, but I don't recommend this.
I'm working in department which is mainly responsible for replacing old Access applications with .NET solutions. In my company Access applications are used for simple scenarios to fulfill business needs of single employee or small group of employees.
Sometimes Access application grows, group of users grows or too many changes are needed. In that case department using that Access application can start a project to recreate application. When this begins we can be sure that new application will be far away from current Access application.
First the business analyst is assigned to the project. His responsibility is to map current solution and to discuss problems of the current solution and expectations for the new solution. I haven't seen a project where "customer" wanted only replacing of the current solution. Everytime the customer also wants some new features and extensions which were not possible in Access.
Business analyst creates some initial description of expected solution which is passed to architects. Architects decide which type of application will be build, what type of HW infrastructure will be needed and how the application will be connected to other systems (if needed). After this initial phase IT have big picture about application and about needed changes. Here some initial estimate is done so project can be planned and resources can be allocated. This estimate is boundary for the project. Than my team starts to do the job.
We are using agile approach so our customer (internal team) incrementally sees new features in the application. First we gather some initial set (backlog) of user stories (special form of requirements) and we estimate these user stories and we let the customer to prioritize them. We choose subset of user stories for iteration (usually 2-4 weeks). New user stories can be added to backlog any time but our selected user stories can't change during iteration. After the iteration we present customer working part of the software. Based on the working part customer can decide to change priorities in backlog or create new user stories. We repeat this approach until customer says stop or unit budget is consumed. The important point is that not all user stories have to be done. Project has been planned with some budget and some low priority user stories don't have to be overtake.
From the technical point of view it is project as any other except few differencies:
You have initial database and you always have to be sure that already implemented part in your new solution also have working migration of exisiting data.
You have existing UI. Users can like or can hate the UI. Make sure that you understand it so that you create UI which is not worse than existing one. I created applications where UI had to be completely different and I created applications where UI had to be exactly the same so that users didn't need additional training.
Try to add some new features so that new application is reasonable. It is always easier to explain needs for the new application if you can describe new needed features.
650 forms/subforms is large by any standards. That represents a major conversion project, and a 'slow' port will be a nightmare.
I would suggest developing a new .NET application 'spike' that contains the basic functionality that is absolutely required and then build upon it. At the same time, freeze the Access application from all but essential fixes.
There are a few tools that will convert MS Access forms to .NET, but they will likely fail on complex forms with sub-forms.
'Effortlessly' Convert Access Forms to VB Objects
So if a user is in Access and they take an action to open a form that happens to be a different executable written in C#, won't it 'appear' to be the same application?
There has to be a user group that would love a separate application that only has the 5-10 forms they use.
Getting rid of tables/forms/features that are not used is an increase in functionality. I don't know the level of documentation for this app, but start there. When users find out they have to document areas of an application and justify the need for parts they don't use, they'll volunteer to have it removed.
Related
Recently, our Development team came up with an idea of reorganizing our processes. Aside from the new methodology that needs to be implemented, our consultant suggested that we should start our development from UI layer and release it weekly for user testing even if the backbone is not yet completed. His idea is that, we need to release a module as quickly as possible in weekly bases. According to him, the users do not really mind what's behind the application, their concern is only about the output of the application (i totally agree with Him on this). Although I am not totally convinced with the idea of starting from UI Layer as I tend to start coding either from the application's domain or the Database instead of doing the UI Design. I don't know if I am just being bias here.
Any thoughts, ideas or suggestions would be very much appreciated.
This is a very broad question, not knowing your development/sale structures.
I've expierienced this kind of points very often, resulting in a not-maintainable whatever (The result had nothing to with software anymore)
Modern development will mostly fail without an strategy like Scrum or Agile.
Consultants and sales-department often push in that direction because they want to sell stuff and get therefore money in their own pocket due to provision. Usually they have nothing bad in their mind. They simply just dont know it better and probably have never heard about Softwarearchitecture before.
Anyway, following some points on pros and cons:
Pro
Sales-Dudes can sell products better because they have something to show to the customer
Users/Customers can get an early feel for the Application and can point out what they like and what you have to improve
Cons
Hardly you will be able to follow architectonical patterns (MVVM just to name one)
Mosty you produce non-maintainable-code
In most cases your application will not be extensible (Plugins)
Sales-Dudes and Users usually dont know what they even want, ending up in applications like this
Your code will be really hard to test. So TDD will not be possible
These are only a few examples stuck in my mind. I could extend this list to 4 pages why it is NOT a reasonable idea to develop by UI-First.
I know this is mostly based on my opinion, but its also full of Expierence i made by myself, resulting in developing the same Application over and over again.
In our company we develop and sell a VB6 application and we think it's about time to migrate it to .NET.
The main reasons are:
We expect VB6 runtime support to end at some point in time, and we do not want to start the migration just then since it's probably gonna be a lengthy process.
There is just 1 1/2 VB6 developers left. The half one being me.
More and more customers asking for features like cloud and mobile device support.
I know that rewriting an application from scratch is the least recommended way for migrating to .NET. I totally aggree with that! Throwing away over a decade of code feels just wrong and would be such a waste of money spent, that I have a hard time recommending and justifying it towards our management.
But right now I don't see another way to do it.
Let me tell you a little bit about the application:
Like I said it has been developed for over a decade. There have been numerous developers working on it, most of them rather unexperienced at that time. We have one developer left from the initial team. That application has been his first and biggest software project and by now he realizes that many of the architectural decisions made over last 15 years have been horribly wrong, others were right at that time but have not been refactored to meet changes made in other parts of the application and so have become wrong at some point in time. This application seems to be a showcase example of code rot.
We are talking about an application of about 150 KSLOC, all in one single executable. It uses about 15 external DLLs, some of them third party ActiveX controls, some of them are our own .NET assemblies.
Adding new features to the application is still possible and being done, but takes ages compared to our other .NET applications. The reason is that every little change in the codebase requires changes all over the place. The only reason why changes are possible at all is because that one developer simply knows most the dependencies and quirks of the application. As you might have guessed the rate of unexpected side effects and bugs is quite high.
My first thought about migrating that application was to first clean up and refactor, then migrate/convert possibly using tools from Artinsoft/Microsoft/WhoEver and then refactor again to get a nice and clean .NET application.
But I see some problems:
There seems to be no way of refactoring the old application. There is no automated testing whatsoever, not even a formal method for manual testing. Every little change requires manual testing by experienced users who just know where defects might hide.
on the other hand I have established a process and set of tools for testing of our .NET applications which gives us a solid base for making refactorings
Converting that code to .NET without major refactoring feels like: Garbage in, garbage out. Even though I hate calling the old application garbage because somehow it works and has proven itself useful.
Our management has a habit of explicitly demanding quick and dirty solutions, disregarding the effects it has on the productivity and against all recommendations from the development team which has at some point started to deny the existence of quick and dirty solutions in order to be able to do things right. That does not mean that we polish features, but we do include the time to write tests and do refactoring in our estimates. So knowing this, I suspect that once the code is converted to .NET and fixed to the point where the application starts and seems to work, the refactoring-phase will be canceled and the application will be shipped to some customers.
So. What I think is that, despite the fact that rewriting from scratch will take a lot of time and resources, it might still be our only option.
Am I missing an option? Do you see possibilities of not having to rewrite that application?
I suggest that you take a step back and read this paper by Brian Foote & Joseph Yoder (University of Illinois). It provides some architectural insight into the problem you have and options to solve it. It's titled 'Big Ball of Mud' (please don't laugh, it is a serious paper). Here is the abstract:
While much attention has been focused on high-level software
architectural patterns, what is, in effect, the de-facto standard
software architecture is seldom discussed. This paper examines the
most frequently deployed architecture: the BIG BALL OF MUD. A BIG BALL
OF MUD is a casually, even haphazardly, structured system. Its
organization, if one can call it that, is dictated more by expediency
than design. Yet, its enduring popularity cannot merely be indicative
of a general disregard for architecture.
These patterns explore the forces that encourage the emergence of a
BIG BALL OF MUD, and the undeniable effectiveness of this approach to
software architecture. In order to become so popular, it must be doing
something right. If more high-minded architectural approaches are to
compete, we must understand what the forces that lead to a BIG BALL OF
MUD are, and examine alternative ways to resolve them.
A number of additional patterns emerge out of the BIG BALL OF MUD. We
discuss them in turn. Two principal questions underlie these patterns:
Why are so many existing systems architecturally undistinguished, and
what can we do to improve them?
BTW, I think your best option is to use the current application as your Requirements and rewrite everything in VB.NET or C# using a proper design.
There are four main options when you have an application like this:
Do nothing: this is always an option, as everybody knows, if it ain't broke don't fix it. However this might not be an option for several reasons such as needing to comply with some security requirements at the company, or simply because one of the components doesn't work in new platforms.
Rewrite: This would be the dream, right? being able to get rid of all the bad practices and duplicated code and so on? Well, it might be that way, however you have to think all the risks involved in developing a new application from scratch. Do you have all the formal requirements? what about test cases? do your team know every little detail in the code or would you need to go line by line trying to figure out how why that if is there? Also, how many bugs do
Buy something off-the-shelf: Since you are an ISV this won't be an option.
Migrate: Of course you'll be bound by the programming practices you used for the original development but you'll get to a new platform faster, all your business logic will be automatically migrated, you can actually hire developers for the new platform and you can get rid of the legacy elements. From here you can also take advantage of all the tools available to refactor code, continuous integration, unit testing, etc.
Also, with an automatic migration you can actually go further than just WinForms. There are also tools that can take your C# code all the way to the web using a modern architecture.
Of course, I work for Mobilize.Net (previously Artinsoft) and this is my biased perspective.
We've been working on this for around 15 years and have seen dozens of clients who come to us after trying to re-write their application and fail after months or even years of struggling without being able to deliver a working application.
I would like to ask you what experience you have with developing and deploying one application that in general has some standard features, but the application can also have customer specific features.
For example:
Customer 1 have the standard features but also want a search function.
Customer 2 have the standard features only.
Customer 3 have the standard features and also want an employee calendar.
How would you solve this?
Would you have one project where you deploy all the application from and then have some kind of config file to determind which features are avaliable in the specific application?
Would you have one project for each customer? This is how I'm doing it now, but the problem here is that if there are bugs that need to be fixed in the standard features I have to fix them in every project.
Any other suggestions are very welcome.
The application is developed in Delphi and C#.
My company solves that problem by giving all customers all features. This keeps development simpler and allows us to spend more time working on improving the product and not have to spend time dealing with the complexities of optional features.
We sometimes meet mild resistance from clients who want a cheaper version with less functionality but that's never been a sales problem.
On the other hand if you sell clients cheaper less functional versions, they are liable to try to get away with these cheaper versions. This can then lead to them not liking the software as much they should because they bought the cheap crippled version. I strongly believe in getting the best product possible to the user.
This advice may not be appropriate to your personal situation, but you did say that any opinions would be welcome.
One version per customer is not a good idea, IMHO. It will stuck your sales one day or later.
Better let all features be released to all customers, i.e. just maintain one software, but locked by a password, for instance. You can give a unique licence number at the software installation in order to identify the customer (put its name in the licence), then compute some passwords according to this licence number, to unlock some features, on request - when paid. This password can be easily automated via a web site, with minimal cost for you.
Or you can also let all functions available for testing, but lock the printing or the saving - just to let the customer think about spending some money to have this "nice added feature".
Sometimes, having all features tend to create a "Gasworks" application. You'll probably need a separate setup application, to customize the application to your customer's needs. Worth thinking about this architecture.
Even with a revision control, multiple versions are a nightmare to maintain: for instance, just back-port all hotfixes to a previous version takes a lot of time. If you don't have to (because of regulatory purpose / certified versions e.g.), don't "branch" your software.
No definitely do not have one project per customer, you could have one solution per customer where you agregate all projects given setup need.
Just to give you in an alternative to plugin architecture, which is right way to go, but also usualy fairly complex.
Option1.
Put common functionality in main project (Core)
Additional stuff like calendar put in separated DLL projects ( one per functionality)
Create VS SOLUTIONS, where you agregate all projects for specific setup + Core. So customer1 will have Customer1Silution with Core and all additional projects he need, customer2 its solution with Core and its additional stuff.
Option2.
Have one big setup for every one and based on its configuration/license enable/ disable access to user to a additional functionality.
Depends on your resources like time, experience, people you work with, clients , you can chose an option more appropriate to you.
1 plugin based: may be the best one but it, complex and it will take a time you become familiar with it, if you never did before something similiar.
Option 1 easy and fast, but if the clients quantity and configyration become defer you will jump into scale problems.
Option 2 is an average between those two, but keep an eye on your setup dimension.
Considering tha fact that you refer projects and nit DLLs in your solutions, if you fix a problem in Core in one solution it will affect also all other solutions.
you have several options:
put the "standard features" into separate module(s) which can be used/linked by the other versions
use a "plugin-architecture" to load the optional features dynamically
In addition to what others have said there is another option: Conditional defines.
With conditional compiling you can wrap feature specific code with IFDEFS (IFDEF EmployeeCalendar, IFDEF SearchFunction...). Then for each client you copy the project file only and set the conditional defines according to features you want to include.
If a client wants/pays for additional feature you just add it to Conditional defines in that client's project file.
This is similar to the modules approach (BPL/DLL) but avoids the added cost of having to deploy/manage extra files. The drawback is that the feature set is fixed at compile time.
With BPL/DLL you could dynamically load additional modules at run time, but if that is not important in your case, then Conditional defines might be a good choice.
Of course if your features are not easily separable then you can end up with a lot of IFDEFs in the code, but then your problem is clear separation of features, and it would be the problem with modules too.
I'm trying to get my head around the thought process when designing a large scale application.
Let's say I have a client who needs a new customer website and he is estimating 40,000 orders per day with an already 25,000 user base. When desiging the application, how do you about determining if a distributed architecutre is needed? Should I use a web farm? etc.
I've mostly build 2 tier (physical) applications in the past and I really want to improve my understanding.
Any insight would be great!
Load Test your new application from the get-go.
Since doing a big-design up-front will never give you the results that you expect from it (15+ years of experience) the best thing to do is to design for change and let the right architecture emerge from your requirements.
Given your description, adopt an agile methodology for this project, and use its practices to guide your project into a success. One of those vital practices is to have a 'Definition of Done' for all the work that you do. Clearly in your DoD you will have the item:
Needs to pass the Load Test (40,000 orders; 25,000 users per day).
As you would then start development, one of the first things to do is ofcourse to set up the environment to be able to run such a load test. If that never happens, you already know very early in the project that you will be in trouble.
Having the load test done as many times as needed (at least once every sprint) you will know that the scale requirement is/can be handled by your architecture.
HtH
It's going to depend on a lot of other factors than just the number of orders per day. Where will it be hosted? What does that physical architecture look like? What else does the application do besides ecommerce? Does it need to integrate with other applications (besides the payment gateways of course)? Etc.
A simple two tier application in the right cloud hosting environment (say VMware for instance) that can scale dynamically would work just fine for an ecommerce website. A simple two tier application in the right on-premises hosting environment (load balanced web farm) should also work fine for an ecommerce website. It's the difference between scaling up (potentially hidden with virtualization, which ends up being a scale out of sorts) and scaling out (adding more servers).
A distributed architecture would allow you to distribute the system load (say order processing) to 1:M servers that sit (perhaps) behind a load balancer. This is a very common approach, and would also work very well for an ecommerce website.
In my opinion, there isn't one architecture or system design that fits every mold. The closest architecture to fit every mold (again, my opinion) would be a service oriented architecture. If all business processes and logic are services (and designed correctly), then no matter how your requirements change, no matter what your hosting environment looks like or changes to, and no matter what integration requirements you have, your system can handle it with little or no changes.
Since you are coding .Net, put the application in Azure (da cLOUD man, DA CLOUD! ;).
It will help you get the processing power that you'll need. The application will work fin as long as you do not do any stupid mistakes.
Scaling can be addressed, at least partially, with load balancing.
Concurrency might be your true problem. Distributed transactions is a heavy handed tool, but it won't solve every use case without additional thought.
Some financial companies have additional security requirements regarding the web servers having direct access to the database.
I am responsible for a team of developers who will are about to start development of a light weight insurance claims system. The system involves a lot of manual tasks and business workflows and we are looking at using Windows Workflow (.NET 4.0).
An example of the business domain is as follows:
A policy holder calls the contact centre to lodge a claim. This “event” fires two sub tasks which are manually actioned in parallel and may take a lengthy time to complete;
Check customer for fraud – A manual process whereby an operator calls various credit companies to check and assess the potential of a fraudulent customer. From here the sub task can enter a number of sub-statuses (Check in progress, Failed Reference Check, Passed Reference Check, etc)
Send item to repairs centre – A manual process where the item for which the policy holder lodged the claim is sent the repairs centre to be fixed. From here the sub task can enter a number of sub-statuses (Awaiting Repair, In Progress, Repaired, Posted, etc).
The claim can only proceed once the status of each sub task has reached a predefined status (based on the business rules).
On the surface it seems that Workflow is indeed the best technology choice; however I do have a few concerns in using WF 4.0.
Skill set – Looking at the average developer skill set I do not see many developers who understand or know Workflow.
Maintainability – There seems to be little support within the community for WF 4.0 projects and this coupled with the lack of skill set raise concerns around maintainability.
Barrier to entry – I have a feeling that Windows Workflow has a steep learning curve and it’s not always that easy to pick up.
New product – As Workflow has been completely rewritten for .NET 4.0 I see the product as a first generation product and may not have the necessary stability.
Reputation – Previous versions of Workflow were not well received, considered difficult to develop with and resulted in poor business uptake.
So my question is should we use Windows Workflow (WF) 4.0 for this situation or is there an alternative technology (e.g., Simple State Machine, etc) or even a better workflow engine to use?
I have done several WF4 projects so lets see if I can add any useful info to the other answers.
From the description of your business problem it sounds like WF4 is a good match, so no problems there.
Regarding your concerns you are right. Basically WF4 is a new product and is lacking some important features and has some rough edges. There is a learning curve, you do have to do some things differently. The main point is long running and serialization, which is something the average developer is not used to and requires some thought to get right as I hear far too often that people have problems serializing an entities framework data context.
Most of the time using workflow services hosted in IIS/WAS is the best route when doing these long running type of workflows. That makes solving the versioning problem not to hard either, just have the first message return the workflow version and make that a part of each subsequent message. Next put the WCF router in between that routes the message to the correct endpoint based on the version. The basic is never to change an existing workflow, always create a new one.
So what is my advise to you?
Don't take a big gamble on a unknown, and for you unproven, piece of technology. Do a small, non critical, piece of the application using WF4. That way if it works you can expand on it but if it fails you can rip it out and replace it with more traditional .NET code. That way you get real experience with WF4 instead of having to base a decision on second hand information and you learn a new and powerful technology in the process. If possible take a course on WF4 as that will save you a lot of time in getting up to speed (shameless self plug here).
About the Simple State Machine. I have not used it but I was under the impression it was for short running, in memory, state machines. One of the main benefits of WF4 is the long running aspects.
I have come to this dilemma couple of times and I had chosen not to use Work Flow foundation. Some of considerations (similar to yours) were
Involved work flows were lot simpler (a combination of state machine and sequential actions) and doing it in WF seems to overkill for efforts involved.
Learning curve for developers to understand and to use WF effectively was considered high. Status transition table describing valid transitions and actions to be taken are used for additional flexibility and developers were comfortable with it, easily understanding the concept and purpose.
Chances of business process changes were slim and rudimentary changes were easily possible with help of transition table. A change in transition would mean a database script while change in actions would result in new release/patch. However, probability of such occurrence was deemed to be low.
Looking back after 13-14 months, I still think that decision of not using WF was correct. IMO, WF makes sense where there is strong likely hood that work flow can change and/or business rules can change. WF allows to isolate workflow in separate file and so making it configurable by users will be simpler.
We have been using WF 4.0 the last couple of months. I have to say it's challenging to think the Workflow way. However, I can tell you it's worth it. We knew very little when we started. We've bought a beginner and professional book for WF 4.0 that helped. I, myself, watched many videos online and followed PDC 2009 for their breaking news about WF 4.0 and how it's different from the previous somewhat sucky versions.
One major thing that we had to propose a solution for is the way we can deal with In/Our Arguments in a workflow without bounding our custom activities to certain data types and how to pass parameters between activities. I have come up with a good solution for that, and the workflow experience that we have so far is not bad at all. Actually, we have a workflow-intensive application that is getting bigger and bigger and I really cannot imagine myself solving it in a different environment. I love the visual effect that it has: it keeps me away from the details of if/else etc constructs and makes the business rules apparent in a way that doesn't make you forced to dive into lines of code to know what's going on or how to fix some bug.
By the way, the project that we worked on is very similar to what you described and it's a medium-sized project.
You can tell from my words that I like it and I do recommend it although is incorporates some risks as it's a new technology and you have to come up with some innovative ideas.
my 2 cents...
I did three projects in WF 3.5 and I have to say it is not easy. It force you to think in the whole new way especially when persistance is used. Updating the application which contains hundreds of incomplete persisted workflow is challenging. Single breaking change in serialization crashes them all. Introducing multiple versions of the same library to support new and old running workflows is common. It was challenging.
I haven't tryed WF 4.0 yet but based on experience from BizTalk and WF 3.5 I think it will be similar.
Anyway the best approach you can take is to do Proof-of-Concept. Take single WF from your requirments and try to implment it in WF 4.0. You will spend some time with it but you will prove if you are able to do that in WF 4.0 and if there are any visible benefits.
If you decide to use WF 4.0 I insist that you check possibility to run WF as WCF service hosted in Windows AppFabric. AppFabric provides some out of the box functionality for hosting WFs.
I think it does not really make sense today to talk about Workflow in WF4 as a technology choice for this kind of problem. What is really appropriate, as mentioned by Ladislav Mrnka above, is WCF WF Services hosted in AppFabric.
My experience with this is that it pays great dividends and is very enjoyable, but problems arise in the beginning because it is not properly appreciated that for many programmers this is a methodology shift more than a technology shift. On the other hand, generalists and those with a problem-solving mindset saw WCF WF AppFabric as a set of exciting opportunities. So if the mix of people on the project are fairly conservative C# devs attached to their daily set of OO and patterns, it will be hard to introduce. If the team is more innovative, then adoption will be much easier because the potential and new doorways multiply with each discovery.
Two main conceptual problems programmers had in moving to this technology was:
a) Message correlation and messaged exchange patterns
b) Workflows and unit testing
In standard systems in C# for example a workflow is rarely explicit and therefore rarely unit tested. The overall workflow is left for testing by acceptance scenarios or integration. Introduce an explicit WF as a software artifact and suddenly standard devs want to try and unit test it, which is usually not worth doing.
The message correlation aspect of it is a bit of mindset shift for those not familiar with message exchange patterns. Most devs have dealt with in process and remote calls, web service and SOAP, and usually focussed on one or two of those. To abstract above it all and work with a general message based system can be confusing at first.
On the positive side though, the end result is something that saves a lot of time and creates a lot of opportunities. One main thing is that the worfklow, if visually clear, is something that can be worked on by end user, developer and analyst together, eliminating unnecessary steps in the development lifecycle and focusing the parties on one artifact. Further, it discourages islands of functionality in dedicated apps, with dedicated glue layers, by encouraging a suite of business processes in WF per business domain. Further, with AppFabric, the plumbing for persistence, logging, and waking up scheduled activities is all done for you. WF4 performance is outstanding too.
My recommendation would be to find the most innovative or explorative team member do the initial scouting to discover the tricky parts, get the core functions working, and have that initial person be responsible for then compartmentalising the remaining work.
In order to do an insurance claim system of any complexity that involves roles and "sub-tasks" you really need an BPM solution, not just workflow. Workflow Foundation 4.0 is slick but it really doesn't not come close to the functionalities of a BPM product.
BPM solutions, like Metastorm BPM, Global360, and K2.NET, provide human centric workflow, tasks, roles, and system integration that can model and streamline the business processes like your insurance claim system. Use ASP.NET to build the interface that integrates with the BPM workflow engine as their built in designers are usually limited and force you to use their custom built web control which usually are not as full featured as the ASP.NET web controls.
Go with the technology your team knows and feels comfortable with. Workflow Foundation is not a product that you can use straight away - it's rather a set of pieces you can embed in your application in order to build a workflow system. IMHO the workflow logic is the least important piece of technology, first of all you have to concentrate on the GUI because business owners will not see anything but the GUI. But if your system is a success then you have to be prepared for neverending change requests and new requirements so you have to implement your business logic so that it's easy to change and easy to divide into separate processes to suit different user needs (sometimes contradicting). BPM helps in this task because it allows you to have separate, multiple versions of business processes suiting various business needs. You don't need full fledged BPM engine for that but it's useful to code your business logic so that it can be versioned and divided into individual business processes - the worst thing to have is an unmantainable and intertangled blob of code that handles 'everything' and that no one can understand. There are many ideas for that - state machines, DSLs (domain specific languages), scripts etc - you decide what the implementation should be. But you should always think in terms of business processes and organize your logic accordingly so that it reflects these processes.
And be prepared for coexistence of many variants of business logic and data structures - this is the most difficult design task imho.
I'm in a situation where I have to use 4.0 as .NET 4.5 isn't accredited for use in our prod environment yet. I had major pain understanding generally how to get long running workflows going to suit our business need but eventually found an elegant solution. It's not something which just anyone coming later to support can just pick up with ease because there's so much to think about, but I do believe in WF as a tool for managing workflow states.
One big thing I take issue with WF 4.0 though is Maurice's comment:
The basic is never to change an existing workflow, always create a new one
That's great if you just want a new version, but what if you have 50,000 persisted workflows and realise at some point that there's a bug in the workflow? You need to be able to update the xamlx and still be coupled to the existing instances. I've tried ungzipping the various metadata columns in the SQL Server instances table to find something that ties the instance to the workflow definition without any luck.
I did write a synchronisation application for importing data from an old system into our new WF 4.0 driven one. We basically load the data into the system, then run the process which goes about automatically calling into the workflow steps and calling validation methods, essentially mocking user interaction. This only really worked well with us due to the architecture we implemented for access to the workflow service host. It's great as a one off, where after running you can go through and do checks to ensure consistency of the data migration process, but having to use this approach for potentially hundreds of thousands of cases once a system is live isn't really an approach that instills confidence and over burdens the process of integration simple bug fixes.
My recommendation is that you avoid WF 4.0 altogether and just go straight to 4.5 if you're environment supports it. The Dynamic Updates and Side by Side Versioning it provides caters for bug fixing and WF versioning all out of the box. I've still yet to investigate exactly how as 4.5 still isn't accredited for use by our client, but eagerly awaiting this opportunity.
What I'm desperately hoping for is that our client doesn't request changes to policy (and therefore workflow adjustments) and that the current workflows hold up without any bugs. The latter being a vain and empty hope as bugs always pop up.
I really can't understand what was going through the WF dev team's heads to release a system where out of the box you can't fix bugs easily. They should have developed a technique for re-binding an instance to new xamlx.