DotTrace can be used to profile tests on TeamCity. But this example works with NUnit, while my tests are written in Visual Studio Tests. When running my tests I use the build-in test engine VSTest, no such option is available for the DotTrace profiler plugin. Instead I need to specify the path for the test engine executable.
I did find an executable VSTest.exe but it's path includes "Microsoft SDKs\Windows Phone\v8.0\ExtensionSDKs", so unsurprisingly it doesn't work. Saying it can't find a start.bat file, that does indeed not exist in that .exe file.
What is the executable path that TeamCity uses for VSTest?
Then I can configure the performance test to use that test engine. I know the tests can be executed using DotTrace, having run it in Visual Studio, just don't want to install Visual Studio and the DotTrace plugin there when there's a TeamCity plugin.
According to this, you can install Visual Studio Test Agent on your TeamCity Agent and then specify vstest.console.exe as a profiling target in the XML config.
Nevertheless, I suggest you to try a new dotTrace plugin version (if it asks you for credentials, specify guest with no password). It's not yet officially announced but fully functional.
The main difference is that the plugin is now integrated into 'unit testing' build steps, e.g. Visual Stiduo Tests, NUnit, MSpec, MSTest. So, it doesn't require a configuration file. Thus, after you install the plugin, all unit testing steps will get the additional option Run build step under dotTrace profiler. All you need to do is add the Visual Studio Tests step, check this option there and specify performance thresholds as usual.
Related
I'm using Selenium with C# to create my automated tests. Now i have to use some sort of reporting tools to save the test cases whether they passed,failed...etc.
I've seen a lot of tools like Allure, and Jenkins. But they require an XML file. and i can't find it when i run my tests using Visual 2013. Why is that ?
how to obtain these XML files in details if i'm missing something ?
Is there any way i could achieve this with minimum effort ?
EDIT:
How can i run my tests using Nunit console runner ? where can i find it? i downloaded the .zip for nunit but i couldn't find the runner?
The NUnit Visual Studio Adapter does not currently produce XML results, although there is an issue on GitHub to add the ability. Your best bet is to run your tests using the NUnit Console runner when you want to create reports. It always produces an XML result file.
You can install the NUnit Console by adding the NUnit.Console NuGet package to your test project. The console will be in a directory like packages\NUnit.ConsoleRunner.3.4.0\tools in your solution root.
Another option is to install the MSI for the console runner. It will then be under C:\Program Files (x86)\NUnit.org\nunit-console
For creating reports, ReportUnit creates excellent HTML reports from your test results.
To run the tests, I would recommend setting up a simple command line build that builds your solution, runs the tests and then produces the report. For day to day development, just running your tests in Visual Studio or on the command line will likely be enough. Most developers find the Visual Studio Test Explorer Window to be a poor UI, but usable for seeing passed and failed tests and running/debugging them.
If you want to setup a command line build, one good tool is Cake Build. It will take a bit of time to setup, but it is an excellent way to run your build tasks as your project gets larger.
NUnit testing can be run with the nunit-console.exe application which is installed with nunit under {Project_root}/lib/nunit/nunit-console.exe. It downloads with NuGet when NUnit installs.
It can be passed a list of testing binaries, or testing project files, or an nunit project (listing multiple if needed).
{PathToProject}\lib\nunit\nunit-console.exe {PathToTestDll}\Project1.Tests.dll {PathToTestDll}\Project2.Tests.dll /xml=nunit-result.xml
or create an NUnit Project with the NUnit Project Editor if you want to group all your test projects into a single config file.
{PathToProject}\lib\nunit\nunit-console.exe {PathToNUnitProject}\Project.Tests.nunit /xml=nunit-result.xml
Allure doesn't require xml. At lease now :)
https://github.com/unickq/allure-nunit
You just use [AllureNUnit] attribure
Does MSTest have standalone GUI similar to nUnit that lets me use it and run test without visual studio? What is the official site for MSTest where I can learn more about how to use it?
MSTest can be used without installing Visual Studio. You will need to install Visual Studio Test Agent, which is a free download from Microsoft.
I think this approach is better from a licensing perspective than manually copying MSTest.exe and its dependencies onto the build server.
See this blog for reference:
http://blogs.msdn.com/b/anutthara/archive/2009/12/16/running-tests-in-mstest-without-installing-the-vs-ide.aspx
It doesn't have a GUI (apart from Visual Studio) but there's a command line tool: MSTest.exe
Here is the official documentation on running MSTest tests.
You can do this with mstest.exe, but the trick is in getting it to work without installing visual studio. This involves the copying of several files and registry entries. I have blogged about it here.
Use Gallio as your test runner... then its not so much of a drama when you enventually drop MsTest and move to a real test framework.
Use VSTest.console.exe part of Microsoft.TestPlatform
Required steps:
Download the test platform from https://www.nuget.org/packages/Microsoft.TestPlatform/
Unzip
In the unzipped folder, copy the \tools\net451\Common7\IDE\Extensions\TestPlatform folder to the machine that has no Visual Studio installed
From cmd.exe run VSTest.console.exe MyTest.dll
More details here:https://learn.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2017#general-command-line-options
You can also use this tool from codeplex: http://testrunner.codeplex.com...
Uset "dotnet test", it is compatible with all versions, eg:
'dotnet test yourassembly.dll -l console -v detailed'
I am trying to use OpenCover for a calculator application, written in C#.
I have opened it in Visual Studio 2013 and installed the OpenCover and Report Generation packages using the package manager console. I cannot understand the other commands to continue. Specifically what does it mean by -target<target application>?
I cannot understand a single step to continue.
When you install OpenCover it would/should have presented you with a readme.txt that provides a link to the wiki http://www.github.com/OpenCover/OpenCover/wiki/usage - there is also a usage.pdf installed with the tool.
-target means the target application, this can be the app you have just written or a coverage tool such as nunit that would run parts of your application code whilst it is running the supplied tests.
If you're happy with any code coverage without it specifically needing to be OpenCover, then see my answer here.
Using the AxoCover plugin makes it very quick and simple to get your NUnit tests displaying their code coverage.
I've got integration tests which I launch from Visual Studio 2008 Development Edition (they are executed as MSTest unit-tests, even though they are actually integration tests). I would like to do some profiling while the tests are being carried out.
In my case, I have an MSTest "ordered" test I'm running. I'd like to collect sampling data while each of the ordered tests is executed.
How do I do this?
p.s. Perhaps I can accomplish this from the command line, if I manually launch mstest from a command prompt, while trying to follow command line directions. Unfortunately, when I try to launch VsPerfCLREnv from the Visual Studio command prompt, it is "file not found." What the heck?
I just got profiling working in VS2008. In case it helps anyone, here is how I got it to work:
Ensure you are in Debug configuration mode
Click Analyse > Launch Performance Wizard
Select Profile a dynamic link library
Enter the following:
Dll path: The dll you want to instrument, located in the bin\Debug of your unit test project
Executable path: C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe
Start Option Arguments: /testmetadata:TestList.vsmdi /runconfig:TestRunConfig1.testrunconfig
Adjust this based on the name of your vsmdi and testrunconfig files.
Working directory: Your solution root path
Choose Instrumentatation. I haven't tried Sampling, so I cannot comment on that.
Run the new performance project
Using this approach I identified the bottleneck in my code, and the unit tests are now 15x faster, and the users are seeing a 10x speed improvement.
Unfortunately Ordered Tests don't have the 'Create Performance Session' option available:
If you had a simple unit test you would be able to create a performance session.
To use command-line profiling you can navigate to \Program Files\Microsoft Visual Studio 9.0\Team Tools\Performance Tools where you will see vsperfclrenv.cmd and other tools mentioned on the profiler blog.
I have Gallio/MbUnit installed and am using VS 2010 RC and I want to be able to run a single unit test or just all unit tests inside of a TestFixture and not all the tests in the entire project everytime I debug. How do you do this in VS 2010?
Install testdriven.net, there is a free personal version
http://www.testdriven.net/
As Simon said, TD.Net is just working well.
If you work with VS2010 Premium or Ultimate, you may also use the built-in test run feature of VS2010. It should find and run your MbUnit tests as well (if the Gallio add-in was installed)
You may also run your tests by using Resharper 5, but it seems that the latest beta of Resharper 5 has broken the Gallio test driver. It should be fixed soon however.
Apart from the VS2010 IDE, you may also run your tests with Echo (a powerful console application located in the bin folder of the Gallio installation directory), Icarus (Winform application with a nice UI; same location), or from PowerShell (by using the Run-Gallio snap-in). You can also invoke the test runner from one of the numerous extensions of Gallio for MSBuild, NAnt, TeamCity, etc.
I didn't see it earlier but VS 2010 has a Test List Editor which allows me to add the specific test to a a list and run it from there so that only the one test is run. I know that TD.Net probably has a right click/run test option for a test method but I didn't want to buy it and install yet another tool.
Another alternative, how we do it is select the specification inside visual studio, press F5, and it runs the specification via a Rake task.
Example:
class DiffuserObserverSpecification
: AutoMockSpecification<DiffuserObserver, IDiffuserObserver>
class When_diffuser_observer_is_created : DiffuserObserverSpecification
class When_diffuser_observer_has_injected_diffuser_change
: DiffuserObserverSpecification
So to execute the behaviour expectations of all DiffuserObserverSpecification, we select the DiffuserObserverSpecification and press F5.
And to execute just the behaviour for When_diffuser_observer_is_created, select that and press F5.
After one month you forget what I debugger looks like.