I need to organise the launch Unit-Tests in MS Test Manager. Can It be done? I create controller but I can't specify path for my solution or tests dll.
I'm not sure that launch Unit-Tests in Microsoft test manager are possible, but my boss requires it.
What you should probably do is setup your build server to run the tests. Then use test manager to trigger a build, which will in turn run the tests.
There's not much value in running the tests again on previously built code as it won't have changed. The tests should run on the code as it builds and then again when the code changes and is consequently rebuilt.
Related
I'm trying to run unit tests from the command line.
For this, I need the test dlls, as per this:
https://msdn.microsoft.com/en-us/library/ms182490.aspx
In the Prerequisites section on that page, it says I need to "Run a Unit Test and Fix Your Code"
Question 1: Do I need to run tests from VS to be able to run them from command line? Surely not?
Now to the more important bit. When I build my solution, regardless of whether it's via VS or command line using MSBuild, I do not get any dlls generated for my tests.
I know this is for C++ but thought might still be relevant https://social.msdn.microsoft.com/Forums/vstudio/en-US/a89c2173-90e6-47b2-af8e-48865969cbca/msbuild15-does-not-create-a-dll-file-after-building-the-c-native-unit-test-project?forum=msbuild. (not that it helped of course).
Question 2: Why don't my unit test dlls get generated when building?
UWP is a bit different from other projects when it comes to Unit Testing. Universal Windows Apps run in a sandbox to make sure they don't do things they don't have permissions for. For this reason, the unit test project is not a simple DLL only, but in fact a full-fledged UWP app (and the generated executable has appx extension), that is launched and performs the tests.
That said, you can still launch the unit test project from the console using a special command as you can see in this SO answer.
vstest.console.exe /Platform:x64 AppPackages\UnitTestProject1_1.0.0.0_x64_Debug_Test\UnitTestProject1_1.0.0.0_x64_Debug.appx
We have 10K+ unit tests in C# solution which are passed when running in local and in TFS.
Now, we are setting up Jenkins for our solution stack and we are facing issue of around 250 Unitests failing consistently.
The same unit tests are passed when i tried running them in jenkins setup server by using Visual Studio and Commnad prompt(MSTest).
What do you think is the issue? Any leads to look at this issue will be helpful.
Edit 1:
I did the research and not able to find anything as the problem itself is strange one. If you are not clear please raise questions instead of Down voting.
Edit 2:
I am able to find out the issue. It is with the unittests dll config file. When i executed MSTest in server by removing the config file, i am seeing same set of tests failing which are failing with Jenkins setup.
I guess we need to modify the steps configure in Jenkins portal to load the unittests dll config file.
My guess would be you have tests that are not actually Unit tests, but instead are integration tests or worse, and they fail non-deterministically. Other than that, you're asking people to do the impossible here without source code posted.
Either post source, or hire a consultant who knows about Jenkins IMO.
There might be issues with conditional compilation symbols (e.g. DEBUG vs. RELEASE code): in VS you normally run the tests on a DEBUG build, on the CI server on a release build.
Also look at some global state not being cleaned up correctly. Some threads which may still be running after a test has seemingly finished may corrupt later tests even when those later tests are located in a different test dll. That can sometimes be detected if test failure depends on the order of tests run.
Another issue often faced is dependency on test data in files: the files may be missing in the virtualized environment where the test is actually run. Use the Deployment attribute.
I had a similar problem with NCrunch. Maybe you can check in VS your "Build"-tag the Platform target. It should be the same like Jenkins is configured. For example "Platform target" is x64, it should be used in Jenkins as well.
I have a suite of SpecFlow tests written in C# using MSTest as the framework. I understand its possible to upload the DLL to MTM and run them through there.
However I was wondering if it was possible for me to execute these tests outside of Visual Studio. For example via a dashboard. I wouldn't want the results uploaded to MTM or TFS as we have a few projects that are not connected to them.
My framework uploads results to a database so thats good enough for me in terms of logging.
I gather I would have to do something with the DLL thats generated from building the project that contains SpecFlow, but I want to avoid using reflection if at all possible.
Does anyone have any ideas on how I could go about this?
Tests written using Specflow are simply tests in selected testing framework (MSTest in your case) so you can run them using test runner for that framework.
In case of MSTest you can use MSTest.exe (located in Visual Studio directory, probably also part of TFS installation). Take a look here http://msdn.microsoft.com/en-US/library/ms182489.aspx on how to run tests (easiest way is to use /testcontainer option).
Please note that MSTest is not the best test framework and running outside of Visual Studio is not the same as running them inside of it. There are problems with config file and separate files needed by tests. If you hit these problems, you can switch to other test framework (I use NUnit).
In my project we use UnitTests (MsTest). We run them manually and in the build script.
But for some environments where we deploy manually.
We are in the need of having some smoke tests. At the moment I made this smoke tests "manually" (Login, create an user, create a licence, create a product, etc).
I know I can run unit tests from command line or from visual studio but...
Do you know if is possible to run these mstests (or probably a small subset) from an .aspx page and show the results?
EDITED: wondering if mstests or something is able to run the tests and have something visual to show in an .aspx page similar to a list of tests passed and tests failed...
I am thinking on running only a small subset of tests (5-6)
If you can run them from command line then you can run them from aspx file. Check Process class.
What approach would you take while developing a custom MSBuild Task in a test driven way?
Are there any available test harnesses suitable for test drive development of a Microsoft.Build.Utilities.ToolTask extension?
I was considering attempting to use NUnit or MSUnit and check files generated and where they are placed, though this I forsee this as being a little clunky.
it's not really TDD way but look at the Tool MS Build Sidekicks
This tool really helps us to develop our nightly/daily builds (with database creation, structure compare, CodeAnalysis, test execution, clickonce deployment ...)
You can analyse and debug the build types on the build machine and on the local development machine.
Build scripts are not designed to be tested.. but
You can create some SmokeTests of your build to see if everything went ok. If you are deploying a website you can have some smoke tests to see:
Login page could be opened
Login page works (You can make a correct login and a failed one)
Core funcionality works (Once you accessed to your site you can perform some basic action like
opening product page or similar)
Those smoke test should be able to be called from command line, so you can call them from task AfterDropBuild to see the result of smoke tests just after build was created.