I have this strange problem. When I use the OpenCover from Visual Studio all tests pass and everything seems fine.
Image from Visual Studio here: http://i.stack.imgur.com/awmWy.png
But when I want to automate the tests using the OpenCover.Console.exe than some tests fail. I have noticed that the 2 failing tests read from config file.
Either the test or the code that is tested reads from config file using ConfigurationManager.AppSettings["someConfigValue"].
I have the same values in the main project and the test project appsettings.
The two failing tests when executing from console image here: http://i.stack.imgur.com/a77Tt.png
In addition I provide the command used for OpenCover.Console.exe.
"OpenCover.Console.exe"
-register
-target:"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\MSTest.exe"
-targetargs:"/noisolation /testcontainer:\"E:\TEAM\TestProject.Tests\bin\Debug\TestProject.Tests.dll" /resultsfile:TestResults.trx"
-mergebyhash
-output:CodeCoverage.xml
UPDATE [04 September 2015]
Since I did not find any particular solution, I came up with a workaround.
Practically I have taken as granted that configuration file and all its data does not belong to unit tests, but to integration tests.
So in my code I do not use ConfigurationManager.AppSettings["someConfigValue"] any more. What I did is created a ConfigurationService and in the IConfigurationService I have all the methods I need for getting the values from the config file.
Having this injected in the BL of the solution, I use Mocks for unit testing purposes.
Related
I have a project with classes and methods that I want to test. I have another project with the test methods that will test the methods of my main project.
I run the tests with opencover and I generate the reports with reportgenerator, with this commands that I have in a .bet file:
..\tools\OpenCover.Console.exe -register:user -target:"C:\myDllWithTests.dll" -output:"c:\coverage\opencovertests.xml"
.\ReportGenerator.exe "-reports:c:\coverage\opencovertests.xml" "-targetdir:c:\coverage\opencovertests.xml\reports"
I am using MSTest for testing.
The problem is that in the html report, I see that the code that is covered is the tests methods, not the methods in my test main project.
How I could add the main methods in the result?
Thanks.
In target argument for OpenCover pass the path to MSTest (e.g. "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\mstest.exe") and specify your test assemblies (e.g. "C:\myDllWithTests.dll") in targetargs argument.
To remove test assemblies from code coverage statistics, specify them in filter argument.
Below is OpenCover command that works fine for me. Here code under test is placed in SampleApp.dll and test code is placed in SampleApp.Tests.dll.
.\OpenCover.Console.exe -register:user -mergebyhash -target:"c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\MSTest.exe" -targetargs:"/testcontainer:\"d:\test\SampleApp\SampleApp.Tests\bin\Debug\SampleApp.Tests.dll\"" -output:UTResults.xml -filter:"+[SampleApp*]* -[SampleApp.Tests]*"
Result report contains only stats for SampleApp.dll assembly, SampleApp.Tests.dll is excluded:
Check this answer for some more details.
There is also a great article by Allen Conway on using OpenCover & ReportGenerator for .Net projects.
This might be quite a late answer here, but I've spent an hour or two playing with this and found the following will fix this. It's worth noting that I had the original bat script from another project that I know works, and just changed the DLL file name, so I know the script was OK.
The additional check to make is to:-
Right click the project that has the source code you want visible in the coverage report (not the unit test project) and click Properties
Select Build > Output > Advanced
Set Debugging information to Full
Rebuild solution and re-run bat file.
Works for me in Visual Studio 2019 with .NET Framework 4.7.2 projects.
I am helping a team whose builds have started failing due to test failures.
The failures are being caused by missing connection string configuration. I checked the usual issues in respect of the config file to ensure that the connection string was specified with exactly the right name.
In the end I obtained the full path of the config file to check that the one on the build server contained the exact configuration that was expected.
AppDomain.CurrentDomain.SetupInformation.ConfigurationFile
The path did not point to the TestProject.exe.config file, but instead pointed to the vstest.executionengine.x86.exe.Config at the following location:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.executionengine.x86.exe.Config
This file contains no connection strings at all.
When I write out all of the available connection strings from configuration, I get the default connection string:
Name: LocalSqlServer Connection: data source=.\SQLEXPRESS;Integrated
Security=SSPI;AttachDBFilename=|DataDirectory|aspnetdb.mdf;User
Instance=true . Aborting test execution.
This is coming from the machine.config file (kudos petelids).
So the big question is:
Why is the vstest.executionengine.x86.exe.Config being used rather than the app.config (at runtime TestProject.exe.config)? (I can guess that this is because the process running is the test runner, but I think it is fair to say that you would expect the test runner to let the test project use its own config file, which is what normally happens).
I think it is fair to say that you would expect the test runner to let the test project use its own config file, which is what normally happens
That's perfectly true assumption when working with NUnit, xUnit, etc. but not with Microsoft Test Runners (MSTest and VSTest). They simply ignore target assembly config file and use their own config.
There are two solution to this problem:
Change MSTest to NUnit or xUnit
Use custom config file (not the default one)
MSTest runs tests in isolated mode by default.
The solution to this issue was to add a new test project and move the tests into it.
The new project behaved like all of the other projects, running the tests in an isolated process and using the app.config file from the test project.
I'm putting this down to a quirk.
I have a test suite, comprised of both unit tests and integration tests, in a project using C# on .NET 4.0 with Visual Studio 2010. The test suite uses MSTest. When I run all tests in solution (either by hitting the button in the testing toolbar or using the Ctrl-R A shortcut chord) all of the tests, integration and unit, pass successfully.
When I either attempt to run the same tests from the command line with mstest (explicitly using the only .testsettings file present) or attempt to run them from the Test List Editor or using the .vsmdi file the integration tests fail.
The integration tests test the UI and so have dependencies on deployment items and such, whereas the unit tests do not. However, I cannot seem to pin down what is actually different between these two methods of running the tests.
When I inspect the appropriate Out directories from the test run, not all of the files are present.
What would cause some of the files that deploy correctly in one situation from Visual Studio to not deploy correctly in another?
The static content started being copied shortly after I wrote the comments above. The other major issue I ran into was that the integration test project referenced libraries that were dependencies of the system-under-test (with copy-local set to true) in order to ensure that the DLLs would be present when they were needed. For some reason, these stubbornly refused to copy when the tests were run through Test List or mstest.
What I eventually did to work around it was include [DeploymentItem] attributes for the DLLs that I needed. This got things working no matter how the tests were run. What I am still unclear on, that may have answered the underlying solution, or provided a better solution, is how Test View/mstest differ from the regular test runner (assuming that the correct .settings file was passed to mstest.). I'm putting these notes/workarounds in an answer, but I'll leave the question open in case anyone can address the underlying cause for how the different test execution paths differ.
I have a load test which contains only a single unit test.
The unit test is of a function in C# which calls C++ code using C++/CLI wrapper.
It runs well without any exception.
The projects are
1)Business logic ->BusinessLogic.lib
2)Wrapper(takes Business Logic.lib)->Wrapper.dll
3)C# project(takes Wrapper.dll)
The load test on running says that Wrapper.dll is not found whereas the full application runs properly and even unit test doesn't report any such error
The exact error is
Test method TestProject1.testTest.getstateTest threw exception: System.IO.FileNotFoundException: Could not load file or assembly 'ManagedBL.dll' or one of its dependencies. The specified module could not be found
where TestProject1 is the Test Project,testTest is the Test Class and getstateTest is the unit test
Is your load test part of a testing framework (NUnit et al) or is it part of your application?
If the load test is inside a testing framework, then check that the Wrapper.dll gets copied into the "staging" folder where the test framework outputs the DLL and executes the test. Furthermore, please specify exactly what the error message is when you run the load test.
Update
OK, so there are several things that could cause this issue:
Visual Studio does not copy the ManagedBL.dll into the test staging directory.
Visual Studio does not copy the BusinessLogic.lib into the staging directory.
Visual Studio does not copy some other library dependency used by the BusinessLogic.lib (i.e. is the BusinessLogic.lib linking into any other native libraries?)
Technically, VS should copy the ManagedBL.dll if you have added it to the C# project's references; however, do check that it gets properly copied anyway (should be in the TestResults folder).
To fix 2 and 3 you might have to do something like this: How to copy native libraries to the unit test staging directory in Visual Studio 2010
Finally, if all else fails, I would highly recommend that you get the Process Monitor and run it while your test is loading and use the filters to only show the information relative to your test process. Process Monitor should be able to tell you when your process fails to find a file/library.
I would like to run automated Silverlight unit tests from a Hudson build server. It seems there are two options:
Use Statlight, although it seems to be designed for TeamCity rather than Hudson, so it would involve a bit of hacking to get it to work.
Use NUnit Silverlight tests.
Can anyone recomend either of these options? Or is there a better alternative?
You can try using Lighthouse Silverlight Unit Test Runner, it works with every Build Server including Hudson, TeamCity and CCNet because it by default produces NUnit compatible xml results file:
http://lighthouse.codeplex.com/
In our company we are using NUnit with Hudson for automatized unit testing. It is simple to setup and execute.
Just download and unzip latest nunit somewhere on Hudson host.
Add Windows batch command as last buildstep with content like:
C:\NUnit\bin\net-2.0\nunit-console.exe "%WORKSPACE%\src\Test\AllTests.nunit" /config=Release /xml="%WORKSPACE%\src\Test\TestResults.xml"
This will execute tests as defined in "AllTests.nunit" file. It is possible tu point just to one assembly (.dll).
To populate test results within Hudson Job page, you would need to install Hudson NUnit plugin. Its possible directly from Hudson plugin management.
After instalation there will be new Post build action: Publish NUnit test result report.
If you check it, you've got line to enter path to test result report. Corresponding path for example above is:
src/Test/TestResults.xml
Hope it helps you to decide ;-)