Is there a way to do automated testing and continous integration of C#, WPF projects? I thought about something like Jenkins but as far as I know Jenkins does not support C# projects. It should be a tool to do test driven development with the possibility to do automated testing, also for the GUI. Maybe the build tool form gitlab is an option?
Many thanks in advance!
We do use Jenkins with our C# projects. You may use the MSBuild plugin to build the projects, or use a "Windows Batch Command" like
"C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe" YourSolution.sln /tv:4.0 /p:Configuration=Release /p:TreatWarningsAsErrors="true" /p:CheckForOverflowUnderflow="true" /p:WarningLevel=4 /v:m /t:rebuild
Note: with this command line, I overwrite project specific settings for warnings and arithmetic overflow.
There are also plugins for Unit Tests. We use MSTest. Since I integrated the OpenCover Code Coverage Report Generator, I must use a long command line:
"C:\Program Files (x86)\OpenCover\OpenCover.Console.exe" "-target:C:\vs\Common7\IDE\MSTest.exe" "-targetargs:/nologo /testcontainer:Tests\Project1Tests.dll /testcontainer:Tests\Project2Tests.dll /resultsfile:testresult.trx /category:"^!SqlTests^&^!Perfomance"" -output:coverage.xml
"C:\Program Files (x86)\OpenCover\OpenCoverToCoberturaConverter.exe" -input:coverage.xml -output:outputCobertura.xml -sources:%WORKSPACE%
"C:\Program Files (x86)\ReportGenerator\ReportGenerator.exe" -reports:coverage.xml -targetDir:CodeCoverageHTML
Sadly, you mstest does not accept wild cards for the test projects, so you end up with a terribly long line. Also note that the above command line excludes test categories "SqlTests" and "Performance". Then the output is converted to a format accepted by other plugins.
You may start some virtual machines after the build and the unit tests, and install there your programs by some scripts complete with some test data and do some automated tests of the system.
For the GUI proper, we do not yet have a test strategy.
There are a lot of options to pick from if you are going to use out of the box MS Test. Otherwise you need to check if the service provider can support xUnit runner or other similar testing frameworks that could be used in your solution.
TFS / VisualStudio Online
TeamCity
Jekings with MSBuild
Bamboo
Appveyor
For UI Automation you could check White Framework. It is by far the nicest one in my opinion if compared to features and ease of use.
Related
I'm using Selenium with C# to create my automated tests. Now i have to use some sort of reporting tools to save the test cases whether they passed,failed...etc.
I've seen a lot of tools like Allure, and Jenkins. But they require an XML file. and i can't find it when i run my tests using Visual 2013. Why is that ?
how to obtain these XML files in details if i'm missing something ?
Is there any way i could achieve this with minimum effort ?
EDIT:
How can i run my tests using Nunit console runner ? where can i find it? i downloaded the .zip for nunit but i couldn't find the runner?
The NUnit Visual Studio Adapter does not currently produce XML results, although there is an issue on GitHub to add the ability. Your best bet is to run your tests using the NUnit Console runner when you want to create reports. It always produces an XML result file.
You can install the NUnit Console by adding the NUnit.Console NuGet package to your test project. The console will be in a directory like packages\NUnit.ConsoleRunner.3.4.0\tools in your solution root.
Another option is to install the MSI for the console runner. It will then be under C:\Program Files (x86)\NUnit.org\nunit-console
For creating reports, ReportUnit creates excellent HTML reports from your test results.
To run the tests, I would recommend setting up a simple command line build that builds your solution, runs the tests and then produces the report. For day to day development, just running your tests in Visual Studio or on the command line will likely be enough. Most developers find the Visual Studio Test Explorer Window to be a poor UI, but usable for seeing passed and failed tests and running/debugging them.
If you want to setup a command line build, one good tool is Cake Build. It will take a bit of time to setup, but it is an excellent way to run your build tasks as your project gets larger.
NUnit testing can be run with the nunit-console.exe application which is installed with nunit under {Project_root}/lib/nunit/nunit-console.exe. It downloads with NuGet when NUnit installs.
It can be passed a list of testing binaries, or testing project files, or an nunit project (listing multiple if needed).
{PathToProject}\lib\nunit\nunit-console.exe {PathToTestDll}\Project1.Tests.dll {PathToTestDll}\Project2.Tests.dll /xml=nunit-result.xml
or create an NUnit Project with the NUnit Project Editor if you want to group all your test projects into a single config file.
{PathToProject}\lib\nunit\nunit-console.exe {PathToNUnitProject}\Project.Tests.nunit /xml=nunit-result.xml
Allure doesn't require xml. At lease now :)
https://github.com/unickq/allure-nunit
You just use [AllureNUnit] attribure
Over the last 6 months our test team have been using selenium webdriver to test our web based products. We have had great success with it and continue to use it on a daily basis. We use visual studio since we are a .net shop to write our c# unit tests. We don't use any other testing frameworks for .net.
We have up until recently been running our automation tests manually through the test explorer window in visual studio (2013), checking on the results and then logging them into a spread sheet. I know this isn't ideal but we don't have that many tests so this has suited us fine thus far. However, the number of automation tests we will be required to write and maintain is due to rapidly increase over the next few months.
I have recently been playing around with creating batch files for calling vstest.console.exe and its various commands and then adding those logs to a server. Not ideal. I know I can still do so much more, specifically integrate some sort of CI server.
We are already using team foundation server and have various virtual servers (all running windows 8.1) at our disposal so I thought about taking advantage of this so I began looking into Jenkins. Trouble is, I'm not finding much information regarding Jenkinks and c#. It looks primarily geared to a java setup. Am I missing something? What little information I have found is seriously outdated and didn't work for me.
I got as far as setting it up and installing the vstest.console.exe plugin but couldn't get a simple test to run. A current step by step guide that doesn’t pre-date 2012 would be great :)
Do you guys think Jenkins is the way to go for c# and the .net framework? Is there a "standard" used within the c# community? I have heard of cruise control and I’m going to check that out. Is it a viable alternative? Easier to use with .net?
Here is essentially what we need:
Continue writing our tests inside visual studio and creating c# unit tests
Schedule a run of our unit tests on a remote / local server
Write out a result / log file - nice reporting features on fails / passes would be great
Email said file to qa / dev teams
I'm hoping some of you guys have been down this road once and can share some insights
It is possible to use Jenkins to run tests via batch scripts, reporting back to Jenkins via the NUnit or MSTest plugins. To do this, simply call the test runner from a Jenkins-executed script (see links below). The primary reason for doing this in my shop is that Jenkins is used to automate the build process, and automated tests are run every time new code is promoted. If you don't use Jenkins for build automation and reporting - i.e. you just want scheduling - the most basic solution would be Task Scheduler (as John O indicated). Plus, if you are using MSTest rather than NUnit then, as others have suggested, it is better to have a look at TFS.
If you really want to use Jenkins with MSTest, the following links may be useful from a configuration perspective:
Error trying to run mstest on jenkins - 2012
Example of running MSTest from Jenkins from above link:
"C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\MSTest.exe" /testcontainer:Example.Tests\bin\Debug\Example.Tests.dll /resultsfile:TestResults.trx
Jenkins MSTestRunner plugin is unable to launch mstest.exe - 2014
If you can/want to use NUnit, check out the following:
How do you run NUnit tests from Jenkins?
Execute NUnit with Jenkins
Actually, for reporting purposes, logging to Excel or something similar isn't a horrible idea (particularly if your organization uses Sharepoint). Worst comes to the worst and you can't get anything to work, then one solution would be automating this reporting process by using Excel's COM Object Model to directly write results to the spreadsheet.
Would still suggest that TFS is your best bet, however.
I am going to give you some information on how the test can be handle from CI not exactly how to configure Jenkins or other CI server. This might be a partial answer to your question but at least something. Keep that in mind selenium tests always need a live site to point to and before you kick off the tests site deployment should happen(either manually or automatic). I prefer running the test suite once at night and let it go for whole night. You can simply accomplish this with the help of NUnit console, batch and windows task scheduler. See my answer here how to do that. Sure enough this can be accomplish from any other CI server. With NUnit Command switch you can export result automatically also.
I'm totally new to .net. Actually the main language I work on is Java. So I'm more familiar with CI pipe in Java.
In our java develop environment, we have
1. a SVN repository to store code
2. a bamboo server to trigger build(code compile and UT, a tool named maven is used here) and also test automation
3. a sonar server which will analyze code coverage, code rule, code complexity and etc...
Compare to .net, now we only have
1. a TFS repository to store code
But how about the other two things? Is there any CI tool for .net to automatically watch code repository to build on demand and also analyze the code quality?
Thanks a lot.
I like TeamCity. It can monitor a TFS server and kick off an MSBuild command when it detects new commits to your repository. It can do code coverage via unit test integration as well.
Ok, so here's my current setup and my problem at the moment. I have a growing set of projects in a Visual Studio solution. The solution contains about 15 projects (give or take a few) and a quickly growing code-base. I realize that I should have had a continuous build system set-up before I got to this, but its never too late I suppose. So after doing a little research, I believe that my perfect setup would be:
NUnit 2.5.x (we are already tied to this... so a necessity)
Integration with CruiseControl.Net (open to other options, but only free ones with Git support)
Integration with a code-coverage tool (NCover, DotCover) would be nice
Integration to run shell commands (for JSLint and compression tools, etc.)
What I am missing is a tool to run the automated build. I looked at NAnt, but it's support for running MSBuild (to build the project) seemed fairly outdated (we're using VS2010) and utilizing the solution files in our build process would be a HUGE time saver. I also looked at MSBuild (for the obvious reasons) but the process that I found for running NUnit tests only supports 2.4.x (MSBuild extensions project).
I am curious how everyone else has organized their continuous build systems. NUnit if fairly popular, so I must not be the only one who is wondering about this.
my first question is how may build projects will you have?
Teamcity Professional is free for 20 build configurations per server and will make your like sooooo much easier, has dotcover build in, and is really easy to setup, run your tests, etc. and it's by far the most fully baked CI server out there.
Jenkins is the next runner up, it's a fork of Hudson and is very flexible with plugins
to do just about anything making it a little more flexible then Teamcity but it's not as easy to set up, code coverage is a pain to set up and has some annoying quirks, but is completely free.
unless you have some really strong reason to use CruiseControl.Net, don't bother, for it's time it was very powerful but it's now sadly out dated and painful to use.
As far as setting up builds, both Teamcity and Jenkins support MSBuild, NAnt, Rake, etc. they also support multiple build steps like would do in an msbuild or Nant file. What I have done in the past is just use the .sln file to do the build with one build step, used the build in task for unit tests, then used the built in task for code coverage then used a another build task for pushing the files.
I have used TeamCity, Jenkins, TFS, and I tried to used CruiseControl.Net but found it painfully clunky. By far Teamcity is the best, with Jenkins a close second, I would not willing use TFS even if I had it.
If you have any questions please feel free to contact me.
You can use NAnt to build your Visual Studio 2010 solutions. I do it all the time. I provided sample NAnt script in my answer here: <msbuild> task or msbuild.exe with NAnt?
If all you need is compile and run tests you can't go wrong with TeamCity it has great support for NUnit/VS and a bunch of reports built in.
If you need to run a more complex build script I suggest you use FinalBuilder for creating the build script and TeamCity command runner to execute that script.
By importing the test result into TeamCity you can still get the reports and there is a simple way to output build status from FinalBuilder to TeamCity:
How can I output messages from FinalBuilder that will be captured by TeamCity?
Outputting build status from FinalBuilder to TeamCity
Is there a tool for automated test discovery for .NET. I am using the Visual Studio unit testing stuff and wanted functionality similar to Python Nose. I need to have a tool automatically discover all the unit test available and run for example the ones "marked" as unit and in different scenarios run the tests "marked" as Integration and so on. I have found an individual that has created his own implementation of the MSBuild test task and an considering creating my own with annotation attributes to do what Nose does but wanted to see if anyone was aware of an existing tool that could work.
Thanks
Visual Studio integrated test framework does exactly that when running from the IDE.
If you need a command line tool that does exactly the same functionality (finds all the tests in a specific directory/solution) I guess you have to write something.
Because MSTest command line needs at least the assembly to be specified. I suggest you write a short script that iterates all the assemblies and find if they have tests in them and then run each assembly using MSTest.exe
Update:
I've just published a new CodePlex project called #Nose that does exactly what you need. Currently it only supports NUnit but I plan on adding VSTest as well.
Try ReSharper from JetBrains. It does what you want with unit tests plus a whole lot more.