I am currently working on C# selenium automation tests. My setup is simple: running tests on a test server (accessible via remote desktop), I've setup task scheduler trigger to run them every day via this command:
dotnet test --logger:"html;LogFileName=C:\inetpub\wwwroot\mywebsite\TestResults\TestResults.html" C:\AutomationSuite\Testsuite.dll
The test results output file is stored in a wwwroot foolder and is uploaded to a website that is accessible via vpn.
Visible output:
Now, my question is: are there any extensions or convertors for my test results to be more appealing to the eye? Like in a pie chart or something like that (running on azure devops the output of test run is more appealing).
Is it possible or am I digging too deep?
Much appreciated
Managed to set it up using AllureReport
Related
I just started using docker and wanted to also use gitlab-ci for builds and tests. After painfully long and numerous attempts I finally managed to prep my images and gitlab-ci.yml. My project consists of WebApi, database project and test project.
My current flow is (using docker/compose and dind in gitlab-ci):
Build stage:
build webapi and load it to container registry
build database and load it to container registry (image mcr.microsoft.com/mssql/server:2017-latest)
Test stage:
run docker-compose on compose file that prepares testing:
start database container from image in container registry build in build stage
build test project and start the container
docer execute on test container "dotnet test 'Tests.csproj'"
One more important piece of information - the test project is using Respawn to reset the database to clean state for integration tests. It basically goes through the object graph and deletes data before each tests (also handles nicely the relationships). For sql server it uses SqlConnection (abstracted to DbConnection in later stages of the code) to connect to the database.
If I run this on my local machine (manually create the database image and then run docker-compose from 2a, and manually run docker-compose -f Tests/docker-compose.ci.yml -p dockercomposeForTests --no-ansi up -d) the tests are running fine.
However from gitlab.com, tests are hanging on the Respawn. They stop exactly on DbConnection.CreateCommand(). At this stage it seems to me, that there is some kind of a problem either in my setup or in the container, but afters days of trying to resolve this issue, I do not have any more ideas.
I know the database is up and the tests are connecting to it. First, Respawn is opening a working connection. Second, I also have a Hangfire starting and connecting to exactly the same database (uses the same connection string as Respawn), and hangfire is reporting connectivity (at first I had some problems with it but I left it because it helps me to diagnose if connection is alive and working; at some point I will probably get rid of it so it won't slow down the tests).
Upon running the vstest.console.exe command mentioned below,
vstest.console.exe C:\Products\Engineering\ACOE\EEDemo\ParallelExecution\ParallelExecution\bin\Debug\ParallelExecution.dll /Settings:C:\Products\Engineering\ACOE\EEDemo\ParallelExecution\Parallel1.runsettings /Parallel /Platform:"x64" /TestCaseFilter:"Name~Test" /Logger:TfsPublisher; Collection=http://rdtfs01:8080/tfs/Engineering; TeamProject="ACoE"; Platform="Any CPU"; Flavor="Release"; RunTitle="Sample Project"; BuildName="20170928.1"
The output obtained is
"Publish completed successfully. Test Results: mtm://rdtfs01:8080/tfs/Engineering/p:ACoE/Testing/testrun/open?id=5360."
As a next step, if we try opening the test result file using the above generated link, it externally opens the report in MTM however does not link with any of the test plan. (Please find the image attached).
However if we next close the report and open
Microsoft Test Manager >> Testing Center >> Test >> Analyze Test Runs
we do not find the report.
This is because of the test run results have not been associated with the TestPlanId in the Database ( Please find the DB screenshot attached ).
Is there a way where we can associate testplan with the output result file generated, so that if an end-user opens the
Microsoft Test Manager >> selects test plan >> Testing Center >> Test Tab >> Analyze Test Runs
will allow user to see the current results?
Seems you are using vstest.console.exe command line and use it with /logger:TfsPublisher which lets you publish the results to the tfs server against a build & test runs show up on the build page.
Published results report can be viewed in Microsoft Test Manager or through build reports from Visual Studio and Web access. Test results can be published through TfsPublisher only for the current test run. More details please refer this blog: Publishing test results through command line test runner
For tests that you run from a test plan by using Microsoft Test
Manager you can review your test results, assign a reason for a
failed test, and assign a resolution.
For tests that you run from the Visual Studio, you can save and then
reopen your test results to analyze them. You can also publish these
test results to Team Foundation Server. However, these test results
are not used in the pre-defined test reports and cannot be associated
with a test plan.
Source Link: Reviewing Test Results
Since you are using the vstest.console.exe to run the tests and publish test results to TFS, it's just using Visual Studio, so it's not able to assoicate the result with Test plan in MTM.
1. Background
I'm currently working on the following build/deployment pipeline:
Github (https://github.com/devedse/DeveMazeGeneratorCore)
Travis Build (https://travis-ci.org/devedse/DeveMazeGeneratorCore/jobs/196910720)
dotnet restore
dotnet build
dotnet publish
docker create image
docker publish image to hub
Docker image hub (https://hub.docker.com/r/devedse/devemazegeneratorcore/)
Use Azure Web App on Linux to execute deployment (http://devemazegeneratorcoredocker.azurewebsites.net/api/mazes/MazePath/512/512)
.
2. Problem
Whenever I push something to the Github repository, a build is kicked off and step 1-3 are being executed correctly.
However, the website on Azure is unreachable.
I used SCM to browse to the debug console (https://devemazegeneratorcoredocker.scm.azurewebsites.net/DebugConsole/Default.cshtml (for future reference)) and executed the following commands to find the log files that were generated by Docker:
and then used the following commands to read them:
cat docker_128_err.log
cat docker_128_out.log
The out log showed the following results (which seem correct):
Login Succeeded
latest: Pulling from devedse/devemazegeneratorcore
5040bd298390: Already exists
fce5728aad85: Already exists
76610ec20bf5: Already exists
51ee4768b31d: Already exists
4dc55ff439a1: Already exists
9cb727c7d7a0: Already exists
2bea08464ad0: Pulling fs layer
2bea08464ad0: Verifying Checksum
2bea08464ad0: Download complete
2bea08464ad0: Pull complete
Digest: sha256:647f3db3daa3330b7eb109a1c604e5bd403c2c7089b3c18c5e9249a9805d3a4d
Status: Downloaded newer image for devedse/devemazegeneratorcore:latest
Login Succeeded
latest: Pulling from devedse/devemazegeneratorcore
Digest: sha256:647f3db3daa3330b7eb109a1c604e5bd403c2c7089b3c18c5e9249a9805d3a4d
Status: Image is up to date for devedse/devemazegeneratorcore:latest
The error log however, shows the following errors:
2017-01-31T13:11:46.757760723Z No executable found matching command "dotnet-/home/DeveMazeGeneratorCoreWebPublish/DeveMazeGeneratorWeb.dll"
The strange thing is, is that whenever I run the image locally, it all works fine:
docker run -it --rm -p 0.0.0.0:5001:80 devedse/devemazegeneratorcore:latest
Somehow there seems to be a difference in running a Docker image on a Linux machine in Azure, compared to my local Docker installation which runs the Docker images on the default VM that's being installed when you install Docker for Windows.
3. Configuration files used:
.travis.yml: (https://github.com/devedse/DeveMazeGeneratorCore/blob/master/.travis.yml)
Dockerfile: (https://github.com/devedse/DeveMazeGeneratorCore/blob/master/Scripts/Docker/Dockerfile)
4. Summary
So summarizing, it seems that running Docker on Azure is being executed in a different manner then when doing this locally. Does anyone have an idea on what this could be/how to solve it?
Again, (just for easy reference), the error:
2017-01-31T13:11:46.757760723Z No executable found matching command "dotnet-/home/DeveMazeGeneratorCoreWebPublish/DeveMazeGeneratorWeb.dll"
Modify your image to put your application bits somewhere other than /home.
/home is where Azure App Service on Linux bind-mounts the persistent site volume, which is a disk that is shared across instances and is persisted between restarts.
You don't have to use it (you may not have any use for it when running your own image), but anything in your image's /home will disappear at runtime.
I am working with HDInsight .NET SDK to use C# sdk with pig. I am getting error when specifying the c# application path.
here's how am defining the C# app in pig script
DEFINE pigudf `PigUDF.exe` SHIP('wasb://book#storage.blob.core.windows.net/PIG/app/PigUDF.exe');
am getting error "invalid ship specification" 'wasb://bookstore#storage160203122480.blob.core.windows.net/PIG/app/PigUDF.exe' doesn't exists, however PigUDF.exe does exists at the given path.
If I run the same query from HDInsight cluster console with both pig script file and c# app stored locally on cluster, it runs successfully.. i.e the below works on hdinsight cluster console
DEFINE pigudf `PigUDF.exe` SHIP('C:/PigUDF.exe');
where pigudf.exe is locally stored on cluster.
I even tried running it through HDInsight tools for visual studio, but I get same error.
Any help here will be appreciated.
thanks,
Saleem
Try using http:// instead of wasb://. The wasb protocol is used to access Windows Azure blob storage.
DEFINE pigudf `PigUDF.exe` SHIP('http://book#storage.blob.core.windows.net/PIG/app/PigUDF.exe');
You can copy your udf to local, update its permissions, ship it, and eventually remove it
fs -copyToLocal wasb://<container>#account/udf.exe
sh cacls udf.exe /E /G <group>:F
define myUdf `udf.exe`ship('udf.exe')
-- run your computation...
sh del udf.exe
I have been unable to get my CodedUI test project to run. The goal is to have it run on a separate machine (virtual machine). I've configured the test controller and agent on the VM. I've also set it to run as interactive process (details below). When I run my build it partially succeeds and the following test error is reported:
Error calling Initialization method for test class XXXX.CodedUI:
Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To
run tests that interact with the desktop, you must set up the test
agent to run as an interactive process. For more information, see "How
to: Set Up Your Test Agent to Run Tests That Interact with the
Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012)
I've performed the following steps:
I've installed a Test Controller and Test Agent to VM (separate from TFS server)
My Test Controller was set "Register test controller with Team Project Collection". I've removed the checkbox! This caused a different error when running my build so I removed the checkbox.
My Test Agent is set to the test controller - set to interactive process - screen saver disabled
I have added a Test settings file to my solution.
Test Execution Method: Remote Execution
Controller shows my controller
Manage Test Controllers shows my Test Agent as Ready!!
Resources I used:
http://msdn.microsoft.com/en-us/library/ee291332.aspx
http://msdn.microsoft.com/en-us/library/dd695837.aspx
After performing all the steps I'm still getting the same error. What could the problem be? My test settings seems to correctly point to the test controller. The test controller and agent seems to be configured correctly. Do I need to use Lab Management to have these tests run on a VM?
Thanks!
You need to configure a Standard Environment in Lab management so that tfs knows where you are sending the tests.
http://nakedalm.com/standard-environments-for-automated-deployment-and-testing/
Should only take you a few minutes to setup.
Alternative set up is the following (without Lab Management environment):
1) do NOT register test controller with a Team Project Collection
2) have your agent set up to interact with desktop
3) in your solution - create testsettings file for the coded UI test run, point it to your test controller
4) in your build definition - set up automated tests to use this newly created test settings file
I am trying to run coded UI test in MTM when build is triggering in TFS2015. set up is done with test agent and test controller and created LabEnvironment. When I trigger build then build is success but test case is not executed showing below error .Enabled interactive window in test agent.
MTM, Test agent , test controller, TFS installed in my same machine (Local machine)
I'm purposely running test cases during the build. This is going to be a build that kicks off my CodedUI tests.
Error calling Initialization method for test class CodedUITestProject.CodedUITest1: Microsoft.VisualStudio.TestTools.UITest.Extension.UITestException: To run tests that interact with the desktop, you must set up the test agent to run as an interactive process. For more information, see "How to: Set Up Your Test Agent to Run Tests That Interact with the Desktop" (http://go.microsoft.com/fwlink/?LinkId=255012) If you are running the tests as part of your team build, you must also set up the build age