I am stuck trying to automate unit tests runs with MSTest and deployment of app.config. I read multiple posts and blogs, tried multiple things and yet still app.config doesn't seem to be picked up during MSTest execution. Having a dll that contains all my unit tests built with msbuild, here is what I've tried...
Attempt 1
Copied app.config to the same location with MyTests.dll as MyTests.dll.config (on one of the msdn forums it was said it'd be picked up automagically)
Added [DeploymentItem("MyTests.dll.config")] attribute to each test
Ran MSTest.exe /noisolation /testcontainer:d:\MyTestTests.dll /test:MyTest
Attempt 2
Created local.testrunconfig file with the following content (below)
Ran mstest with /runconfig and no isolation, but nothing was executed: MSTest.exe /runconfig:d:\local.testrunconfig /testcontainer:d:\MyTestTests.dll /test:MyTest
Result:
Loading d:\local.testrunconfig...
d:\local.testrunconfig
d:\local.testrunconfig
... and nothing happens: no errors, no tests are executed!
EDIT/RESOLUTION: By default, MSTest executes tests in separate processes. In this case, config file is automatically picked up if it is named like "dllname.dll.config". However, it is hard to debug tests running in separate processes if they run outside of VS. /noisolation switch is used to make MSTest run all tests in one process. However, in this case test config file is NOT picked up. Instead, MSTest.exe.config file is used, which is located in the same directory as MSTest. To resolve this issue, configuration file can be loaded pragmatically like this:
ExeConfigurationFileMap map = new ExeConfigurationFileMap();
map.ExeConfigFilename = #"path to config file";
Configuration config =
ConfigurationManager.OpenMappedExeConfiguration(map, ConfigurationUserLevel.None);
Kateroh,
My setup looks like this (I'm using msbuild from a TFSbuild.proj):
Build sln
Copy all from output to %TEMP% (black magic :D)
Don't know why but mstest is looking for references in %TEMP%.
Run mstest with parms:
"C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe" /nologo /testcontainer:$(TestDir)\mylib.dll /resultsfile:$(TestResultFile) /runconfig:$(SlnDir)\AutomaticBuildTest.testrunconfig /searchpathroot:$(TestDir) /publish:mytfsserver /publishbuild:$(BuildDefinition) /flavor:Debug /platform:AnyCPU /teamproject:mytfsproject
where AutomaticBuildTest.testrunconfig is bellow
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="AutomaticBuildTest" id="eda99352-93e1-402e-9517-d04fffa66b35" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<!--<Deployment enabled="false" />-->
<Deployment enabled="true">
<DeploymentItem filename="D:\sa12\78\bin\Debug" />
</Deployment>
<NamingScheme baseName="BC2ALibraryTest" appendTimeStamp="false" useDefault="false" />
<!-- http://blogs.msdn.com/b/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx -->
<Execution location="Local" hostProcessPlatform="MSIL">
<!--http://msdn.microsoft.com/en-us/library/ms404663.aspx-->
<ExecutionThread apartmentState="MTA" />
<Hosts skipUnhostableTests="false" />
<TestTypeSpecific>
<UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
<AssemblyResolution applicationBaseDirectory="D:\sa12\78\bin\Debug">
<TestDirectory useLoadContext="false" />
</AssemblyResolution>
</UnitTestRunConfig>
</TestTypeSpecific>
<AgentRule name="LocalMachineDefaultRole">
</AgentRule>
</Execution>
</TestSettings>
Your application doesn't use app.config. It uses application.exe.config. That's what you need to deploy.
Sorry, didn't see you were testing a DLL.
This is not how .NET configuration works. Each DLL does not use its own config file. It can only use the config file of the .exe it is running in (actually, of the AppDomain it's running in).
You would somehow need to get MSTEST to add your .dll.config to its own configuration, assuming it's actually the host process. I don't see how to do that from the command line.
I usually use a Unit Test Project in Visual Studio, so I just deploy the config file from that project. Works fine.
The problem is, as it turns out, with our over-complicated build environment and the fact that we are using and x-copiable version of MSTest (produced locally). The following command succeeded when I ran against VS2008 "proper" MSTest:
"%ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe" /testcontainer:d:\MyTests.dll /test:MyTests /resultsfile:results.trx
Thanks everyone for the answers! The check goes to you, Marius, you made me learn new things with your tesrunconfig.
Related
I have a stage with the follow tasks:
There are 2 test suites with 8 test cases.
I want to execute only test cases with tag "download" in feature file
When I locally execute command
dotnet test <name>.dll --filter TestCategory=download
everything is fine, but when I try to add command-line options in "Other console options" in task
test cases are not filtered and all are executed.
What am I doing wrong and what can help me to filter tests?
The other console options are not honored when running from a test plan. You either need to change it to select tests using an assembly (which matches your local copy) or you need to filter the tests in the suite using a filter criteria not tied in source code.
If you expand the info of that option:
These options are not supported and will be ignored when running tests
using the ‘Multi agent’ parallel setting of an agent job or when
running tests using ‘Test plan’ option. The options can be specified
using a settings file instead.
Matt is correct. The other console options in Visual Studio test task doesn't support running from a test plan.
To solve this issue, you could use .runsettings file.
Now, versions after VS 16.6 preview 3 can support adding testcasefilter to .runsettings directly.
Here is an example:
.runsettings file
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<!-- Configurations that affect the Test Framework -->
<RunConfiguration>
<MaxCpuCount>1</MaxCpuCount>
<!-- Path relative to directory that contains .runsettings file-->
<ResultsDirectory>.\TestResults</ResultsDirectory>
<TestCaseFilter>TestCategory=xxx</TestCaseFilter>
</RunConfiguration>
....
Pipeline Settings:
You could install the VS 16.7.1 in Visual Studio Test Platform Installer task.
Then you could set the Test Platform version and runsettings file in the visual studio test task.
Here is a ticket about testcasefilter in runsettings file.
I am trying to run an all included exe (created using Costura ) on a windows VM. As per an answer in this SO question , I have installed test agent and build tools and trying to run exe through this command
C:\Users\..\Desktop>"C:\Program Files (x86)\Microsoft Visual Studio\2017\TestAgent\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" MyExeName.exe /Tests: MyTestName
on VM , I start the agent and give above command but I am receiving error No test is available in C:\Users\..\Desktop\MyExeName.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
The exe runs fine on local machine using same command and start tests. On local I have VS2019 and all the code. On VM , I can not have VS and code base as a requirement.
EDIT:
I noticed (through ILSpy) that in dot exe under references, there is no reference to this Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll , however the dll is present in resources.
I could reproduce this error... I did some experiments on my csproj file and it turns out you have to add an Import tag in your console csproj file. This has nothing to do with your test agent or test class.
Check your package.config (ps. this is 2019, so check your version) MSTest.TestAdapter has to be added.
<package id="MSTest.TestAdapter" version="1.3.2" targetFramework="net48" />
Secondly, add the Import tag to your csproj file. (and again check your version)
<Import Project="..\packages\MSTest.TestAdapter.1.3.2\build\net45\MSTest.TestAdapter.props" Condition="Exists('..\packages\MSTest.TestAdapter.1.3.2\build\net45\MSTest.TestAdapter.props')" />
When importing this project a few more items are being added to your project.
And this results into this output
See screenshots below for test case 1 (not working) and 2 (working)
Test 1:
Test 2:
I hope this is working on your end!!
Solution and project:
I have an ASP.NET project that outputs a class library as a dll which is then deployed to IIS. Per standard builds a config file is also output along with the dll as xxx.dll.config and it contains many useful environment variables such as connection strings (i.e. it conforms to the application settings architecture requirements).
In VS 2012 the contents of xxx.dll.config file were sourced from app.config in the project root directory.
In VS 2015 it appears xxx.dll.config's contents are instead sourced from web.config in the project root directory.
Steps to reproduce:
Create a ASP.NET application with VS2015.
Add a app.config file in this application.
Build this application.
Open the bin folder and find the application.dll.config file and open it. The content is the same with Web.config.
Do the same operations from Step 1 to Step 4 in Visual Studio 2012, the content is the same with App.config.
This is causing me a bit of a headache as I was relying on app.config to hold variables (connection strings, auth providers) for my functional tests while web.config held variables for deployed code.
Most of the reading I've done on AppDomains and their configuration variables seems to indicate that web.config shouldn't factor into the build output unless explicitly instructed and even then it isn't exactly easy to transform a web.config file, nevermind have it end up as the xxx.dll.config file.
I could rework my test fixtures to set the environment variables directly from the tests but for things like ASP.Identity that requires writing quite a few new classes to load a RoleManager and associated providers which I'd like to avoid.
If anyone could suggest how to engineer the VS build to output a config file that is easily used by the assembly AND the nunit tests when run via the nunit runner or Resharper that would be awesome.
To restore the original behaviour of Visual Studio 2015 back to what I was accustomed to in Visual Studio 2012 I added a post build event:
Go to Project-> Properties
Select Build Events
Add the following to the "Post-build event command line" :
copy /Y "$(ProjectDir)\App.config" "$(TargetDir)\$(TargetName).dll.config"
The .dll.config is now once again the App.config instead of Web.config
I am helping a team whose builds have started failing due to test failures.
The failures are being caused by missing connection string configuration. I checked the usual issues in respect of the config file to ensure that the connection string was specified with exactly the right name.
In the end I obtained the full path of the config file to check that the one on the build server contained the exact configuration that was expected.
AppDomain.CurrentDomain.SetupInformation.ConfigurationFile
The path did not point to the TestProject.exe.config file, but instead pointed to the vstest.executionengine.x86.exe.Config at the following location:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.executionengine.x86.exe.Config
This file contains no connection strings at all.
When I write out all of the available connection strings from configuration, I get the default connection string:
Name: LocalSqlServer Connection: data source=.\SQLEXPRESS;Integrated
Security=SSPI;AttachDBFilename=|DataDirectory|aspnetdb.mdf;User
Instance=true . Aborting test execution.
This is coming from the machine.config file (kudos petelids).
So the big question is:
Why is the vstest.executionengine.x86.exe.Config being used rather than the app.config (at runtime TestProject.exe.config)? (I can guess that this is because the process running is the test runner, but I think it is fair to say that you would expect the test runner to let the test project use its own config file, which is what normally happens).
I think it is fair to say that you would expect the test runner to let the test project use its own config file, which is what normally happens
That's perfectly true assumption when working with NUnit, xUnit, etc. but not with Microsoft Test Runners (MSTest and VSTest). They simply ignore target assembly config file and use their own config.
There are two solution to this problem:
Change MSTest to NUnit or xUnit
Use custom config file (not the default one)
MSTest runs tests in isolated mode by default.
The solution to this issue was to add a new test project and move the tests into it.
The new project behaved like all of the other projects, running the tests in an isolated process and using the app.config file from the test project.
I'm putting this down to a quirk.
The setup is this:
.NET 4.5
Visual Studio 2012
Azure 2.0
one WebRole which hosts 2 sites:
one MVC4
one WCF project
When I published the site to the cloud, it looked like it was working in debug mode.
I published it by right-clicking the cloud project, clicking publish and selecting the config version and the "release" build version when it prompted.
To check what happened, I opened up the .cspkg file by changing the extension to .zip, and checked the .cssx file in it (also changed the extension to .zip). This allowed me to see the sitesroot/0 and sitesroot/1 folders which had my website and service. These contained source code, even the .csproj files. To compare, I checked another solution on the cloud, which just had the bin folder and the only non-compiled code were the views and .js scripts and stylesheets.
I've made sure that the compilation element in the individual projects' configs looked like this:
<compilation debug="false">
Anyway, whatever I do it looks like it builds a package in debug mode. I'm completely in a loss, and I don't know what to look at anymore, since everything looks fine.
This question shows a related problem:
VS publish to azure uses debug instead of release
But they're using TFS to automate the build and publish process, while I'm doing it manually.
Any tips or pointers would come in handy. Or at least a list of things to check and compare with other projects.
After a bit(lot) more digging, I've found the reason. It's the multiple sites on one web role that cause issues. Since the site code just gets copied into the sitesroot folder.
I've managed to solve the issue following the article here:
http://michaelcollier.wordpress.com/2013/01/14/multiple-sites-in-a-web-role/
What needs to be done is:
setup prebuild and postbuild events on each of the sites to take care of copying the built dlls into a custom build folder
Pre (clears the custom build folder):
rmdir "$(ProjectDir)..\YOUR-AZURE-PROJECT\Sites\$(ProjectName)" /S /Q
Post (copies dlls to the custom build folder):
%WinDir%\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe "$(ProjectPath)" /T:PipelinePreDeployCopyAllFilesToOneFolder /P:Configuration=$(ConfigurationName);PreBuildEvent="";PostBuildEvent="";PackageAsSingleFile=false;_PackageTempDir="$(ProjectDir)..\YOUR-AZURE-PROJECT\Sites\$(ProjectName)"
change the sites elements in ServiceDefinition.csdef to point to the custom build folder
<Sites>
<Site name="WebSite" physicalDirectory="..\..\Sites\WebSite\">
<Bindings>
<Binding name="Endpoint1" endpointName="Endpoint2" />
</Bindings>
</Site>
</Sites>