We have 10K+ unit tests in C# solution which are passed when running in local and in TFS.
Now, we are setting up Jenkins for our solution stack and we are facing issue of around 250 Unitests failing consistently.
The same unit tests are passed when i tried running them in jenkins setup server by using Visual Studio and Commnad prompt(MSTest).
What do you think is the issue? Any leads to look at this issue will be helpful.
Edit 1:
I did the research and not able to find anything as the problem itself is strange one. If you are not clear please raise questions instead of Down voting.
Edit 2:
I am able to find out the issue. It is with the unittests dll config file. When i executed MSTest in server by removing the config file, i am seeing same set of tests failing which are failing with Jenkins setup.
I guess we need to modify the steps configure in Jenkins portal to load the unittests dll config file.
My guess would be you have tests that are not actually Unit tests, but instead are integration tests or worse, and they fail non-deterministically. Other than that, you're asking people to do the impossible here without source code posted.
Either post source, or hire a consultant who knows about Jenkins IMO.
There might be issues with conditional compilation symbols (e.g. DEBUG vs. RELEASE code): in VS you normally run the tests on a DEBUG build, on the CI server on a release build.
Also look at some global state not being cleaned up correctly. Some threads which may still be running after a test has seemingly finished may corrupt later tests even when those later tests are located in a different test dll. That can sometimes be detected if test failure depends on the order of tests run.
Another issue often faced is dependency on test data in files: the files may be missing in the virtualized environment where the test is actually run. Use the Deployment attribute.
I had a similar problem with NCrunch. Maybe you can check in VS your "Build"-tag the Platform target. It should be the same like Jenkins is configured. For example "Platform target" is x64, it should be used in Jenkins as well.
Related
I installed VSCode for Unity game development a few weeks ago and have been using it as an alternative to my usual setup of VS + Resharper. I enjoy VSCode quite a bit, especially given how fast and lightweight it is.
One problem I have, however, is writing tests. It's standard practice at our studio to create unit tests for certain features we implement, and we so we have an assembly for running tests in a Unity project.
There's a testing .asmdef that references the NUnit framework DLL and the assembbly that has the code I want to test. In VS, if I navigate through the code, it all looks fine. If I switch to VSCode, however, it tells me there's compilation errors all over the place, because VSCode can't see any of the NUnit classes or the classes from the assembly I want to test (in Unity, I get 0 compilation errors even when writing new code).
Is there any VSCode setup I can do to get rid of the compilation errors and write tests normally, or do I have to switch to VS to write tests?
UPDATE: A work colleague helped me out and linked me this extension which clears up the compilation errors.
It looks like it lets me run tests from the IDE, too, but I haven't set it up yet as I just run them from Unity's Test Runner window. This isn't a priority for me, so I'll mark this as the answer.
I'm trying to run unit tests from the command line.
For this, I need the test dlls, as per this:
https://msdn.microsoft.com/en-us/library/ms182490.aspx
In the Prerequisites section on that page, it says I need to "Run a Unit Test and Fix Your Code"
Question 1: Do I need to run tests from VS to be able to run them from command line? Surely not?
Now to the more important bit. When I build my solution, regardless of whether it's via VS or command line using MSBuild, I do not get any dlls generated for my tests.
I know this is for C++ but thought might still be relevant https://social.msdn.microsoft.com/Forums/vstudio/en-US/a89c2173-90e6-47b2-af8e-48865969cbca/msbuild15-does-not-create-a-dll-file-after-building-the-c-native-unit-test-project?forum=msbuild. (not that it helped of course).
Question 2: Why don't my unit test dlls get generated when building?
UWP is a bit different from other projects when it comes to Unit Testing. Universal Windows Apps run in a sandbox to make sure they don't do things they don't have permissions for. For this reason, the unit test project is not a simple DLL only, but in fact a full-fledged UWP app (and the generated executable has appx extension), that is launched and performs the tests.
That said, you can still launch the unit test project from the console using a special command as you can see in this SO answer.
vstest.console.exe /Platform:x64 AppPackages\UnitTestProject1_1.0.0.0_x64_Debug_Test\UnitTestProject1_1.0.0.0_x64_Debug.appx
The purpose of this is to be able to set the project's .exe file to automatically run on startup and have the tests run. The tests are integrated into BrowserStack where I can view the results. Thanks for any help!
Yes, with the latest changes to the framework, it's possible to do this. Some features may not be available to you. For example, you probably will not be able to recover warning results, which don't throw an exception, or use multiple asserts for the same reason.
Most recently, before the latest release, this was broken btw, so be sure to get the latest.
I need to organise the launch Unit-Tests in MS Test Manager. Can It be done? I create controller but I can't specify path for my solution or tests dll.
I'm not sure that launch Unit-Tests in Microsoft test manager are possible, but my boss requires it.
What you should probably do is setup your build server to run the tests. Then use test manager to trigger a build, which will in turn run the tests.
There's not much value in running the tests again on previously built code as it won't have changed. The tests should run on the code as it builds and then again when the code changes and is consequently rebuilt.
Using Visual Studio and TFS & preferably Specflow or standard unit test.
I want devs to run ALL unit test as a policy before check in. If a unit test breaks, then vS should stop them from checking in, just like when running across a merge conflict.
I know there're post build scripts that will do this, but really if unit test breaks, I rather that it doesn't get into source control at all. Plus the turn around is rather slow to wait for the full build. And then there's the bickering on who breaks whose stuff.
So no, I want unit test to pass locally before a check in. How would I do that? Yes they can just hit the button, but I like to get them a bit more "incentive" than that.
It sounds like what you're after is a TFS Gated Check-in. This can ensure that the code builds, merges and that tests run successfully prior to committing the check-in. You can read more about it here:
An introduction to gated
check-in
It's worth noting that it's a much slower process than CI builds, so depending on how many check-ins your developers are doing you may be better off looking at a CI build with 'Create Work Item on Failure' enabled and a Project Alert set up to notify the developer that they broke the build.
The TeamCity Visual Studio plugin supports pre-tested commits. I can't speak for TFS, however.