I have been unable to get my CodedUI test project to run. The goal is to have it run on a separate machine (virtual machine). I've configured the test controller and agent on the VM. I've also set it to run as interactive process (details below). When I run my build it partially succeeds and the following test error is reported:
Error calling Initialization method for test class XXXX.CodedUI:
Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To
run tests that interact with the desktop, you must set up the test
agent to run as an interactive process. For more information, see "How
to: Set Up Your Test Agent to Run Tests That Interact with the
Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
I've performed the following steps:
I've installed a Test Controller and Test Agent to VM (separate from TFS server)
My Test Controller was set "Register test controller with Team Project Collection". I've removed the checkbox! This caused a different error when running my build so I removed the checkbox.
My Test Agent is set to the test controller - set to interactive process - screen saver disabled
I have added a Test settings file to my solution.
Test Execution Method: Remote Execution
Controller shows my controller
Manage Test Controllers shows my Test Agent as Ready!!
Resources I used:
http://msdn.microsoft.com/en-us/library/ee291332.aspx
http://msdn.microsoft.com/en-us/library/dd695837.aspx
After performing all the steps I'm still getting the same error. What could the problem be? My test settings seems to correctly point to the test controller. The test controller and agent seems to be configured correctly. Do I need to use Lab Management to have these tests run on a VM?
Thanks!
You need to configure a Standard Environment in Lab management so that tfs knows where you are sending the tests.
http://nakedalm.com/standard-environments-for-automated-deployment-and-testing/
Should only take you a few minutes to setup.
Alternative set up is the following (without Lab Management environment):
1) do NOT register test controller with a Team Project Collection
2) have your agent set up to interact with desktop
3) in your solution - create testsettings file for the coded UI test run, point it to your test controller
4) in your build definition - set up automated tests to use this newly created test settings file
I am trying to run coded UI test in MTM when build is triggering in TFS2015. set up is done with test agent and test controller and created LabEnvironment. When I trigger build then build is success but test case is not executed showing below error .Enabled interactive window in test agent.
MTM, Test agent , test controller, TFS installed in my same machine (Local machine)
I'm purposely running test cases during the build. This is going to be a build that kicks off my CodedUI tests.
Error calling Initialization method for test class CodedUITestProject.CodedUITest1: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012) If you are running the tests as part of your team build, you must also set up the build age
Related
I am currently working on C# selenium automation tests. My setup is simple: running tests on a test server (accessible via remote desktop), I've setup task scheduler trigger to run them every day via this command:
dotnet test --logger:"html;LogFileName=C:\inetpub\wwwroot\mywebsite\TestResults\TestResults.html" C:\AutomationSuite\Testsuite.dll
The test results output file is stored in a wwwroot foolder and is uploaded to a website that is accessible via vpn.
Visible output:
Now, my question is: are there any extensions or convertors for my test results to be more appealing to the eye? Like in a pie chart or something like that (running on azure devops the output of test run is more appealing).
Is it possible or am I digging too deep?
Much appreciated
Managed to set it up using AllureReport
I just started using docker and wanted to also use gitlab-ci for builds and tests. After painfully long and numerous attempts I finally managed to prep my images and gitlab-ci.yml. My project consists of WebApi, database project and test project.
My current flow is (using docker/compose and dind in gitlab-ci):
Build stage:
build webapi and load it to container registry
build database and load it to container registry (image mcr.microsoft.com/mssql/server:2017-latest)
Test stage:
run docker-compose on compose file that prepares testing:
start database container from image in container registry build in build stage
build test project and start the container
docer execute on test container "dotnet test 'Tests.csproj'"
One more important piece of information - the test project is using Respawn to reset the database to clean state for integration tests. It basically goes through the object graph and deletes data before each tests (also handles nicely the relationships). For sql server it uses SqlConnection (abstracted to DbConnection in later stages of the code) to connect to the database.
If I run this on my local machine (manually create the database image and then run docker-compose from 2a, and manually run docker-compose -f Tests/docker-compose.ci.yml -p dockercomposeForTests --no-ansi up -d) the tests are running fine.
However from gitlab.com, tests are hanging on the Respawn. They stop exactly on DbConnection.CreateCommand(). At this stage it seems to me, that there is some kind of a problem either in my setup or in the container, but afters days of trying to resolve this issue, I do not have any more ideas.
I know the database is up and the tests are connecting to it. First, Respawn is opening a working connection. Second, I also have a Hangfire starting and connecting to exactly the same database (uses the same connection string as Respawn), and hangfire is reporting connectivity (at first I had some problems with it but I left it because it helps me to diagnose if connection is alive and working; at some point I will probably get rid of it so it won't slow down the tests).
Can we publish the details of failed test cases statistics in HTML report of SpecRun?
Example: Analytics of failed test cases in previous executions.
If one test scenario is failing from the past 5 runs, In Error Details section of specrun report, 5 needs to be displayed against that test case. So that we can analyse whether this test case failure is new in current run or its there from previous builds.
Can we make any settings to run only failed test cases in the previous run with SpecFlow+SpenRun ?
To publish historical failed tests and displayed them against that test case and run only failed test cases might not be achieveable in azure devops.
However you can achieve rerun failed test cases via visual studio test task. You can configure the threshold to rerun failed tests and the maximum attempts under Execution options in visual studio test task.
All backendtests works fine locally and on one of the TeamCity agents, but always fails when another TeamCity agent tries to run them.
I logged onto all agents, and can't find any differences between them.
Other issues I have found with 'Unable to find testhost.dll' error, have been solved with referencing Microsoft.Net.Test.SDK, but that doesn't work.
In the logs for the agent working:
[10:52:07][vstest] Starting test execution, please wait...
[10:52:09][vstest] NUnit Adapter 3.13.0.0: Test execution started
In the logs for one of the agents not working:
[12:27:56][vstest] Starting test execution, please wait...
[12:27:57][vstest] Unable to find D:\TeamCity\BuildAgent5\work\b75e42d21fae163\tests\UnitTests\bin\Release\netcoreapp2.2\testhost.dll. Please publish your test project and retry.
[12:27:57][vstest] Test Run Aborted.
Everything before this looks simular. I think the agents that are not working can't fint NUnit Adapter, but I have no idea why. I checked the folder over and all agents had both NUnit3.TestAdapter.dll and NUnit3.TestAdapter.pbd in the same versions.
Upon running the vstest.console.exe command mentioned below,
vstest.console.exe C:\Products\Engineering\ACOE\EEDemo\ParallelExecution\ParallelExecution\bin\Debug\ParallelExecution.dll /Settings:C:\Products\Engineering\ACOE\EEDemo\ParallelExecution\Parallel1.runsettings /Parallel /Platform:"x64" /TestCaseFilter:"Name~Test" /Logger:TfsPublisher; Collection=http://rdtfs01:8080/tfs/Engineering; TeamProject="ACoE"; Platform="Any CPU"; Flavor="Release"; RunTitle="Sample Project"; BuildName="20170928.1"
The output obtained is
"Publish completed successfully. Test Results: mtm://rdtfs01:8080/tfs/Engineering/p:ACoE/Testing/testrun/open?id=5360."
As a next step, if we try opening the test result file using the above generated link, it externally opens the report in MTM however does not link with any of the test plan. (Please find the image attached).
However if we next close the report and open
Microsoft Test Manager >> Testing Center >> Test >> Analyze Test Runs
we do not find the report.
This is because of the test run results have not been associated with the TestPlanId in the Database ( Please find the DB screenshot attached ).
Is there a way where we can associate testplan with the output result file generated, so that if an end-user opens the
Microsoft Test Manager >> selects test plan >> Testing Center >> Test Tab >> Analyze Test Runs
will allow user to see the current results?
Seems you are using vstest.console.exe command line and use it with /logger:TfsPublisher which lets you publish the results to the tfs server against a build & test runs show up on the build page.
Published results report can be viewed in Microsoft Test Manager or through build reports from Visual Studio and Web access. Test results can be published through TfsPublisher only for the current test run. More details please refer this blog: Publishing test results through command line test runner
For tests that you run from a test plan by using Microsoft Test
Manager you can review your test results, assign a reason for a
failed test, and assign a resolution.
For tests that you run from the Visual Studio, you can save and then
reopen your test results to analyze them. You can also publish these
test results to Team Foundation Server. However, these test results
are not used in the pre-defined test reports and cannot be associated
with a test plan.
Source Link: Reviewing Test Results
Since you are using the vstest.console.exe to run the tests and publish test results to TFS, it's just using Visual Studio, so it's not able to assoicate the result with Test plan in MTM.