I was told about http://fitnesse.org, as a testing Framework quite good to develop tests for my HTML and DOM. So, I made a proof of concept but it seems like it isn't the appropriate tool for that. It appears to be oriented to documentation than deep testing.
Te question is: is it a good tool to test the different screens the user will navigate through my asp.net mvc application ? Is there a way to test this, through FitNesse tables? In case of non being, what could be a good choice?
Thank you!
FitNesse is best at testing a level below the GUI. It is a good fit for testing your ASP.NET services directly (which is what we do) but it is not intended for testing at a GUI level - e.g. by automating a browser.
To test your services using FitNesse, you write a 'fixture' (in your case probably a .NET class library) and this fixture acts as a thin UI to your services.
The Fixture has a public interface which you can write FitNesse tests against.
Automated testing at this level is effective because your public ASP.NET services will be stable. This means you can trust your tests - when they are red it is probably because your code is broken.
Testing at the GUI level - typically by automating a browser and comparing screen shots - can prove to be counter-effective, because your client is likely to be volatile:
Change the layout slightly and your tests are broken; change one of your css styles and your tests are broken; a browser or operating system update can break the tests.
Checking tests that are marked red to see if they are really broken can cost a lot of time and is error-prone. We test a Windows desktop application this way using a home made solution based on Ranorex, and the cost-effectiveness of these tests is questionable.
Ideally, you will have a combination of both GUI and service-level tests - but we are also searching for the correct solution for our HTML client.
Have you possibly tried WaitN?
Related
I know the question comes up every now and then. But because it hasn't been considered for a long time, I would like to ask it again. What would you recommend to create and run automated tests of a WPF application? Recording would be good too, of course, but it is not mandatory. Can you recommend something?
Besides the possible automation bots - or better: UI testing by code - I actually haven't been working on that lately. So, therefore, I would like to approach this from a slightly different angle: How to test a WPF application?
In general; your testing capabilities - well, actually the ease of it - depend on how the program is structured.
Some examples:
Dependency Injection: works very well in combination with unit tests.
The MVVM pattern (suited for WPF): works very well with Dependency Injection and Unit Tests - And allows to isolate UI tests.
Separation of layers - quite obvious, but often forgotten
Of course there are also test concepts to take into account:
Unit tests
Integration test
Consumer Contract tests
UI tests
End to end test.
My guess is, you're referring to a tool to do the UI testing, as I said, it has been a while, so I can't give you an answer there - but what I can give you: good test-ability depends on a well defined architecture.
Here's something on UWP; I am pretty sure there was something similar for WPF: https://learn.microsoft.com/en-us/visualstudio/test/test-uwp-app-with-coded-ui-test?view=vs-2019
I hope this helps.
Take a look at Appium and WinAppDriver. It's a platform to test Web apps, mobile apps and desktop apps using the same API.
I have used Specflow together with Appium to do UI testing in a recent project.
If you want to go for UI testing using recording (no coding required) there a multiple options. Haven't used any of them, but there is: Microfocus Silk, SmartBear TestComplete and many more.
Some of these tools also provide you with Test Case Management as well.
I am trying to understand acceptance tests, but I am confused where it starts or what kind of test/s is/are involved.
Do I have to use automating GUI test frameworks or do I have to use unit tests? What is the boundary of an acceptance test?
Edit: My question is about automated acceptance tests.
Acceptance testing is done after the entire application/software is developed and integrated.
Acceptance testing is done mainly to test whether the application meets the user requirements.
There are mainly 2 types of acceptance testing.
Alpha testing
Beta testing
Acceptance testing is mainly done by the client (person who asked for the software to be developed) and end users.
Alpha testing is done by the client. He is helped by the developer.
Here the client looks at the software to make sure all his requirements are fulfilled.
Beta testing is done after Alpha testing is completed.
Here the application is released to a set of people who behave as end users and use the applications.
Unit test are not to be confused with acceptance tests.
Acceptance tests are basically requirements, written as tests so that:
It is clear when requirements are met;
Actual testing is easier to plan, and to run.
Unit tests are automated test for small bits of code, used to keep an eye on all the little bits without the need for constant (and much hated) manual checks.
You can go down the UI road. Selenium or WatiR are solid tools that can be used to run ui-based test suite.
If you are Dot.Net developer, you can use WatiN, but the problem with it is that it seems to be pretty much dead, as it had no new version since April 2011.
I did manage to have some decent test suite work for me a while back, integrating SpecFlow (more on that later) and watiN, and it worked fine.
However, as time went by, i realized that when I was doing UI based tests, all i was doing was loading a page, clicking something, and than checking the results in the DB. Sometimes, i also checked that the screen also showed me the expected message, but that's all. That conclusion drove me off UI based testing.
What i started doing, is make sure the UI is built on rules and idioms. The tooling now-days (asp.net mvc, razor templates or better yet - knockout.js) allows us to do so without too much pain. When the UI is built methodically, and not by everybody throwing whatever field they like on the page, most of the time all you need to test is the methods that build it, and not the result. Ofcurse, if i do want to test it (and in some scenarios, you will), it is much easier (and faster) to test it with tools like QUnit,
So my way current of practicing ATDD :
Use specflow to get business requirements into test code.
Test only "code behind".
Use knockoutJS for the UI plumbing (using lots of custom bindings)
Create standards for Models that are returned to the view.
Treat UI behavior tests as unit tests.
Here is a good starting point for specflow: http://www.infoq.com/articles/Spec-Flow
I was browsing for a while the internet and this site and instead of finding some ways to unit test my existing code the only finding was to separate logic and interaction with the user (MVC approach). Although this is great for new projects it is time-consuming and as a result too expensive to invest for existing ones. Is there a way to create specific unit tests, ideally automated, for existing GUI projects that unfortunately connect directly to databases or other systems to get data and the data are manipulated before it is shown? Currently we have two projects the one being MFC, the other C# .net 2.0 Thanks a lot.
Unit testing won't cut in here considering you can't change your existing code (not to mention you don't really unit test UI). You should look for some kind of GUI testing automation/scripting tools. Like Sikuli. Quoting literally the first paragraph from their website:
Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots).
It doesn't get any simplier than that. You "tell" the tool which parts of your UI it should observe/interact, it records and replays it. Skimming through this presentation will give you idea of what exactly you can do (might also check their video). Probably won't solve all your problems, but might be alternative worth considering.
Unit testing an already existing project is always a challenge. however i point you to some open source tools that will help you to automate unit testing
C++
Boost unit test framework
Google Mock
C#
NUnit
NMock
It is possible to produce some level of automated testing, but not unit tests.
Unit tests, by definition, test small units of logic decoupled from the system as a whole. I'd recommend new code be written in the way you described (mvc etc) to be unit testable.
With your existing code, unit testing will obviously require refactoring, which I appreciate is not in your timeframe. You will need to work with what you've got an look at a way to perform more whole-system automated testing, probably driven through the UI. The fact these are not Unit tests is by-the-by, there are helpful tests to have even if you have unit tests. Its helpful to know the distinction though when you are searching for resources.
You are probably best searching for automated ui testing. With the .net apps, you may find something like White useful
If you are lucky enough to have a Premium (at least) version of Visual Studio 2010, then you could consider writing Coded UI Tests.
A UI Test is basically an automated sequence of actions (mouse, keyboard...) on a GUI. These are very high level tests (or functional tests), not unit tests, but it can help testing a GUI application that is already existing.
For instance you can easily automate CRUD actions (which imply a database) and check (assert) that actions have produce the expected result in the UI (new created item in a list...).
Write UI testing can be very time consuming because there are various aspect you had to test. Thanks God there are a lot of frameworks to achieve that result but you always have to write some code.
I assume you already did unit testing (Visual Studio itself comes with a not so bad unit test framework) so what you want to check is not algorithms but UI automation/results. What does it mean? Everything that is code must be tested by code (database operations and algorithms, for example). Even some UI controls can be somehow tested by code (example: if I simulate a user click I'll get that event fired when this condition is true). Trust me, UI testing is Black Art and often you'll get failed tests even if everything is OK.
Simple stress scenario
For a simple scenario, for example to stress your application to reproduce a bug repeating the same operation many times, you can use a macro recorder (such as WinMacro). You register user inputs and then you run that macro in a loop. If there's a subtle bug you have many chances to reproduce (and/or to find) it when that actions are repeated 5000 times in a night. That done you'll get data from your logs.
Simple scenario
If your application can be somehow automated (it may be easy for .NET application using VSA) you can prepare some "good" macro to automate an operation, put results in a file and compare them with a known good results data file.
Simple tip: for MFC application you can write your own "macro" with a text file where each line is a Windows message with its parameter; read it, parse it and SendMessage() them to your application to simulate user inputs, menu clicks and so on. Grab - for example - text box value and compare with something known. WinSpy++ is your friend.
Complex scenario
For anything else (is my custom control drawing everything in the right way? when user click that button then UI colors changes?) you have to use a more complex tool. There are several tools to automate UI testing, built-in in Visual Studio 2010 (not in every edition) what you need to create coded UI tests. What does it mean? You write code to automate your application and then you write more code to check its results (sometimes even comparing bitmaps with known results. It may be tedious and a lot but virtually you can test everything, even if the application hasn't been designed for UI testing.
Start reading this from MSDN.
There is a plethora of commercial tools too (even if I never used it in any project) so I not write any link, I guess you'll have a lot of results in Google.
Mocking is usually the best approach for simulating integration points, but unfortunately most Mocking frameworks fall short if the code is too interconnected and bury dependencies in private methods etc.
It might be worth looking into a framework called Moles (made by Microsoft) that have very few limitations for what you can Mock. It even handles private methods!
Perhaps you could use it to mock your db calls to test your data manipulation?
There are several tutorials online.
Here's one that might get you started:
http://www.unit-testing.net/CurrentArticle/How-To-Mock-Code-Dependencies-Using-Moles.html
I have an old WinForm application written in .NET 2.0. The application doesn't follow any pattern or layer pattern. My client now want to introduce unit testing Framework. As it is live application so it very risky to re-write the whole code again. What approach should i follow ?
Thanks in Advance
There's a book I've seen recommended often on this exact subject, I haven't read it but it seems appropriate for your problem, though I don't know if the fact that it's WinForms complicates this or not.
"Working Effectively with Legacy Code"
http://books.google.ie/books?id=CQlRAAAAMAAJ&q=dealing+with+legacy+code&dq=dealing+with+legacy+code&hl=en&ei=LnVXTrviCtSu8QPEwei0DA&sa=X&oi=book_result&ct=result&resnum=1&ved=0CC0Q6AEwAA
In WinForms it's really hard to unit test because the code behind is linked tightly to the GUI itself. Some automation tests will probably be the best you can get on a WinForms application without going out of your way to change the program.
If your client want a testable solution, I would suggest making it in WPF and use a MVVM framework like Caliburn.Micro that emphasizes unit testing.
Unfortunately, that means rewriting the entire application.
Short answer
Either rewrite application, or don't bother to much with the unit tests.
Since WinForms applications typically have a UI that is strongly coupled with the underlying "data model", some form of automated UI testing is probably your best bet - i.e. an external application emulates user's clicks and other interactions with the UI and checks whether your program is in expected state after that.
You may want to take a look at:
http://smartbear.com/products/qa-tools/automated-testing/supported-testing-types/functional-testing/
To do just the UI tests (since its a .net based) you could use .Net light weight Test automation feature. Details here http://msdn.microsoft.com/en-us/magazine/cc163864.aspx
You dont have to use an external tool for automating it. But again, to test the underlying layers or business logic you have to extend the test cases.
I would recommend you move the most critical parts to normal C# classes and unit test those. And then only do UI testing on the complicated parts of the GUI. I have worked on a legacy project but with ASP.NET Webforms instead of Windows Forms and what I noticed is that some parts of the system changed often and were good candidates for refactoring and unit testing while other parts never ever changed and just were not worth the effort of testing.
If this is a large project then this could take a long time. A huge part of the work is making the code testable and introducing some sort of MVP (Model-View-Presenter) pattern to be able to separate the GUI code from the business logic.
I would highly recommend Working Effectively with Legacy Code as recommended by Eoin Carroll. It describes techniques to work with legacy code and also provides motivation (you could be in for some tough times) by showing that it can be done.
Also take a look at these two StackOverflow questions(here and here) for discussions on WinForms and MVP.
That depends on what you want to test, and how the app is written.
GUI is hard to test, but check out some of the other answers. If, as I suspect, you just want to test the business layer, that's easier:
If the app has the business and GUI decoupled, then you can easily use NUnit, either by integrating it into the project from your IDE, or in NUnit's own way, then write tests to cover what functionality the business layer exposes to the GUI.
If the app isn't decoupled, and you have a lot of the logic in the GUI, then you really need to refactor and decouple. Without doing this you will be restricted to testing via the GUI, which is difficult and not really full-proof, and it means trivial changes to the GUI may invalidate your testing of the business logic.
The product I have been working on has been in development for the past six years. It started as a generic data entry portal into an insanely complex part WPF/part legacy application. The system has been developed for all these years without a single Unit test in its fold. Now, the point has been raised for a comprehensive unit testing framework. I have been recruited recently to work on this product and have been tasked to get the 'Testing' in order. Since the team that worked on the product for the last six years adopted 'Agile', the project lacks any documentation of the business rules or any design documents.
I have been trying to write unit tests for some of the modules. But I am not sure what to Mock, how to setup my Test fixture and eventually what to Test for, since a casual glance of the methods does not reveal its intentions. Also, it has come to my attention that the code was not developed with a particular methodology in mind.
Given the situation, I was wondering if the good people of Stackoverflow could provide me with some advise on how to salvage this situation. I have heard about the book 'Working with Legacy Code' that has something to say about this general situation but I was thinking about getting some pointers from individuals who have encountered similar situations within the technology stack(C#,VB,C++,.NET 3.5,WCF,SQL Server 2005).
In my opinion the best way is to start by "stabilizing" the current code functionallity using integration tests. Try to create tests that has start point which is not likely to change later. Using the integration tests you can gain confidence that refactoring that'll come later for the unit tests are not breaking anything.
The next step is to unit test the code. If you're free to refactor the code you can start separating logic to classes (e.g. extra logic in view layer) and add unit tests to them. Using this process you also get to know better the code of the product.
It is very recommended read Working with Legacy Code, many of the problems you're going to encounter already have solutions :)
Unit testing legacy code can be a challenge sometimes, depending on the existing code and on how much you can change the code. You can use some tools, for example for writing integration tests you can use White framework to automate the GUI. Another tool you can use for writing the unit tests without forcing major changes in the code is Typemock Isolator (disclaimer - I work at Typemock), it allows faking most of dependencies without changing the production code. There are many other tools which can ease the process, try to find and make best use of them :)
#sc_ray: I know this may sound pretty obvious, but I believe before you start writing tests against the existing code base, you should focus on making sure you are using an MVVM approach when interacting with your UI. Being a legacy app does not mean you have code updating the UI directly with if statements, but the older the project the easier it is for people to bypass more modern software development styles. All I'm saying is that I would make sure I am optimizing the use of binding, commanding, and all the wonderful infrastructure facilities WPF provides. Otherwise important pieces of your business logic would not be able to be tested and you could be potentially writing tests against less relevant code...