I've got integration tests which I launch from Visual Studio 2008 Development Edition (they are executed as MSTest unit-tests, even though they are actually integration tests). I would like to do some profiling while the tests are being carried out.
In my case, I have an MSTest "ordered" test I'm running. I'd like to collect sampling data while each of the ordered tests is executed.
How do I do this?
p.s. Perhaps I can accomplish this from the command line, if I manually launch mstest from a command prompt, while trying to follow command line directions. Unfortunately, when I try to launch VsPerfCLREnv from the Visual Studio command prompt, it is "file not found." What the heck?
I just got profiling working in VS2008. In case it helps anyone, here is how I got it to work:
Ensure you are in Debug configuration mode
Click Analyse > Launch Performance Wizard
Select Profile a dynamic link library
Enter the following:
Dll path: The dll you want to instrument, located in the bin\Debug of your unit test project
Executable path: C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe
Start Option Arguments: /testmetadata:TestList.vsmdi /runconfig:TestRunConfig1.testrunconfig
Adjust this based on the name of your vsmdi and testrunconfig files.
Working directory: Your solution root path
Choose Instrumentatation. I haven't tried Sampling, so I cannot comment on that.
Run the new performance project
Using this approach I identified the bottleneck in my code, and the unit tests are now 15x faster, and the users are seeing a 10x speed improvement.
Unfortunately Ordered Tests don't have the 'Create Performance Session' option available:
If you had a simple unit test you would be able to create a performance session.
To use command-line profiling you can navigate to \Program Files\Microsoft Visual Studio 9.0\Team Tools\Performance Tools where you will see vsperfclrenv.cmd and other tools mentioned on the profiler blog.
Related
I have a C# project in Visual Studio 2017 Professional. It has some unit tests: methods declared with [Test] which display a little green tick icon in the editor, where I can right-click and 'Run' or 'Debug' the test. When I do so, a window opens with all tests in the solution and I can click a green 'play' triangle to run them all.
I can also run my program under the profiler starting from Debug -> Performance Profiler.
So far so good. Now, how do I run a unit test under the profiler?
When I first asked this question I looked for profiling unit tests and saw it was a feature in Visual Studio Professional. But in fact, I'm not using VS's own unit testing support, but rather NUnit, driven by Resharper. Sorry to throw everyone off the scent.
Using VS2017 Professional and the Diagnostic Tools you can profile code between two break points!
Thus, you can debug your unit test (xunit, nunit, mstest, whatever!) with any tool (VS Test Explorer, ReSharper, etc) like this:
enable diagnostic tools while debugging (see answer from tm1)
Set a break point at the start of your unit test, another one at the end (or wherever you like)
Debug the unit test with your favorite tool, you hit the first break point.
Open the Diagnostic Tools window (Debug > Windows > Show Diagnostic Tools)
Start Recording a CPU profile (small record button below all those graphs)
Continue (F5) to the next break point
VS will now create a snapshot for this period and you can dig into the relevant methods. Voila!
Just for completeness, an alternative, commercial solution is Jetbrain's DotTrace, which is part of ReSharper Ultimate at the point of writing. Here you could directly profile your unit test without manual break points.
If you can debug and stop at a break-point, you should also be able to get CPU and memory usage according to this page, given you enable diagnostics tools while debugging.
Edit
According to #patrick-stalph the 3-step proposed solution does not work in Visual Studio 2017 Professional.
Disclaimer
I'm using Visual Studio Enterprise 2017, so I cannot check, whether this works in Professional, or not.
Open Test Explorer e.g. Ctrl + Q, then type Test Explorer and hit Enter
You should see your tests in Test Explorer. If you cannot, Build.
Choose a test, right click it, and choose Profile Test
Workaround
I must note, that I'm using xunit and autofixture, and I haven't seen this feature working. See this issue, why I haven't.
Hence, based on the answer of Jeremy Liberman, I was able to work around using the steps below:
Install xunit.runner.console NuGet package to the test project
Click Analyze > Performance Profiler... from the Toolbar.
Change Target to Executable
Click Start
Follow the Wizard, till you reach Choose the application that you want to profile (.EXE, .DLL, Website)
Choose An executable (.EXE file)
Based on Run tests with the xUnit.net console runner, I've specified the following values:
Full path: c:\<...>\src\packages\xunit.runner.console.2.4.0\tools\net472\xunit.console.exe (fit this to your environment)
Command-line arguments: <Product>.Tests\bin\Debug\<Product>.Tests.dll
Working directory: c:\<...>\src\ (the root of the Solution)
Next, Finish
Note, that this profiles all of the tests in the test assembly. You can choose, which tests you want to run by providing more Command-line arguments to xunit.runner.console. To learn more about the console runner options, run the console runner with no command line options.
I'm using Selenium with C# to create my automated tests. Now i have to use some sort of reporting tools to save the test cases whether they passed,failed...etc.
I've seen a lot of tools like Allure, and Jenkins. But they require an XML file. and i can't find it when i run my tests using Visual 2013. Why is that ?
how to obtain these XML files in details if i'm missing something ?
Is there any way i could achieve this with minimum effort ?
EDIT:
How can i run my tests using Nunit console runner ? where can i find it? i downloaded the .zip for nunit but i couldn't find the runner?
The NUnit Visual Studio Adapter does not currently produce XML results, although there is an issue on GitHub to add the ability. Your best bet is to run your tests using the NUnit Console runner when you want to create reports. It always produces an XML result file.
You can install the NUnit Console by adding the NUnit.Console NuGet package to your test project. The console will be in a directory like packages\NUnit.ConsoleRunner.3.4.0\tools in your solution root.
Another option is to install the MSI for the console runner. It will then be under C:\Program Files (x86)\NUnit.org\nunit-console
For creating reports, ReportUnit creates excellent HTML reports from your test results.
To run the tests, I would recommend setting up a simple command line build that builds your solution, runs the tests and then produces the report. For day to day development, just running your tests in Visual Studio or on the command line will likely be enough. Most developers find the Visual Studio Test Explorer Window to be a poor UI, but usable for seeing passed and failed tests and running/debugging them.
If you want to setup a command line build, one good tool is Cake Build. It will take a bit of time to setup, but it is an excellent way to run your build tasks as your project gets larger.
NUnit testing can be run with the nunit-console.exe application which is installed with nunit under {Project_root}/lib/nunit/nunit-console.exe. It downloads with NuGet when NUnit installs.
It can be passed a list of testing binaries, or testing project files, or an nunit project (listing multiple if needed).
{PathToProject}\lib\nunit\nunit-console.exe {PathToTestDll}\Project1.Tests.dll {PathToTestDll}\Project2.Tests.dll /xml=nunit-result.xml
or create an NUnit Project with the NUnit Project Editor if you want to group all your test projects into a single config file.
{PathToProject}\lib\nunit\nunit-console.exe {PathToNUnitProject}\Project.Tests.nunit /xml=nunit-result.xml
Allure doesn't require xml. At lease now :)
https://github.com/unickq/allure-nunit
You just use [AllureNUnit] attribure
Does MSTest have standalone GUI similar to nUnit that lets me use it and run test without visual studio? What is the official site for MSTest where I can learn more about how to use it?
MSTest can be used without installing Visual Studio. You will need to install Visual Studio Test Agent, which is a free download from Microsoft.
I think this approach is better from a licensing perspective than manually copying MSTest.exe and its dependencies onto the build server.
See this blog for reference:
http://blogs.msdn.com/b/anutthara/archive/2009/12/16/running-tests-in-mstest-without-installing-the-vs-ide.aspx
It doesn't have a GUI (apart from Visual Studio) but there's a command line tool: MSTest.exe
Here is the official documentation on running MSTest tests.
You can do this with mstest.exe, but the trick is in getting it to work without installing visual studio. This involves the copying of several files and registry entries. I have blogged about it here.
Use Gallio as your test runner... then its not so much of a drama when you enventually drop MsTest and move to a real test framework.
Use VSTest.console.exe part of Microsoft.TestPlatform
Required steps:
Download the test platform from https://www.nuget.org/packages/Microsoft.TestPlatform/
Unzip
In the unzipped folder, copy the \tools\net451\Common7\IDE\Extensions\TestPlatform folder to the machine that has no Visual Studio installed
From cmd.exe run VSTest.console.exe MyTest.dll
More details here:https://learn.microsoft.com/en-us/visualstudio/test/vstest-console-options?view=vs-2017#general-command-line-options
You can also use this tool from codeplex: http://testrunner.codeplex.com...
Uset "dotnet test", it is compatible with all versions, eg:
'dotnet test yourassembly.dll -l console -v detailed'
DotTrace can be used to profile tests on TeamCity. But this example works with NUnit, while my tests are written in Visual Studio Tests. When running my tests I use the build-in test engine VSTest, no such option is available for the DotTrace profiler plugin. Instead I need to specify the path for the test engine executable.
I did find an executable VSTest.exe but it's path includes "Microsoft SDKs\Windows Phone\v8.0\ExtensionSDKs", so unsurprisingly it doesn't work. Saying it can't find a start.bat file, that does indeed not exist in that .exe file.
What is the executable path that TeamCity uses for VSTest?
Then I can configure the performance test to use that test engine. I know the tests can be executed using DotTrace, having run it in Visual Studio, just don't want to install Visual Studio and the DotTrace plugin there when there's a TeamCity plugin.
According to this, you can install Visual Studio Test Agent on your TeamCity Agent and then specify vstest.console.exe as a profiling target in the XML config.
Nevertheless, I suggest you to try a new dotTrace plugin version (if it asks you for credentials, specify guest with no password). It's not yet officially announced but fully functional.
The main difference is that the plugin is now integrated into 'unit testing' build steps, e.g. Visual Stiduo Tests, NUnit, MSpec, MSTest. So, it doesn't require a configuration file. Thus, after you install the plugin, all unit testing steps will get the additional option Run build step under dotTrace profiler. All you need to do is add the Visual Studio Tests step, check this option there and specify performance thresholds as usual.
I have been working on UI Automation to perform unit testing using NUnit in Visual Studio 2012. The NUnit is integrated in my VS2012. I need to store the results of the test such as pass/fail status, the error message and the other available details in the test explorer somewhere in a human readable form. I know that VS2010 we used to get the .trx file which could be used further. How do I get the result in VS2012? Is it even possible?
I can at least assure you that it's possible with TFS, as we did this in our company. But I don't remember every detail. The basic steps are:
Create a build definition to run NUnit tests (use the conrole runner for this)
Transform the result XML (TestResult.xml) to VS format (.trx) using a tool. There are several around, try to g00g1e it.
Publish the transformed results to TFS (done in the build script).
It seems that it is not possible to somehow log all test results, when run directly in VS. In this case the only chance you have, is to run the tests externally using the NUnit runners (GUI or Console) and store the TestResults.xml output file.