Visual Studio 2019 Version 16.8.4
Resharper Version 2020.3.2
Nunit 3.13.1
if I wasn't wrong, i should be using the test runner from Resharper.
I put my test code below. It has no logic but creating a large test case set.
The test runner just freeze for a really long time (as you can see at screendump) and only start running test after 20-30 mins wait.
In actual development, for the sake of 95%+ unit test coverage, we do have some test case that has more than 80,000 cases that cannot be split or remove.
Is there any other alternative way to make the test run work ?
[TestFixture]
public class Test_LargeTestCaseSet
{
private static IEnumerable<TestCaseData> LargeTestCases()
{
for(var i = 0; i < 100000; i++)
{
yield return new TestCaseData(i);
}
}
[TestCaseSource(nameof(LargeTestCases))]
public void LargeTest(int i)
{
Assert.AreEqual(i, i);
}
}
Yes, there is! Do not declare your test as 100k separate test-cases. Obviously, it would not work well with UI that has a visual representation for each individual test-case.
Instead, just write the loop inside of the test method itself. Use Assert.Multiple if you want all of "test-cases" to run on each execution of the test. If knowing the first encountered failure is enough, simple Assert.XXX would suffice.
Related
I'm attempting to use TestCaseSource to re-use the test with different data. Here I'm trying to give my tests their own name with the 'setName' property which works fine. My issue is that the original test also shows up in the testexplorer. It doesn't seem possible to run. How can I get rid of it in the test explorer?
Simple reproduction:
[TestFixture]
public class Tests
{
[TestCaseSource(nameof(MyTestData))]
[Category(name: "MyCategory")]
public void OriginalTest(string first, string second)
{
Assert.IsTrue(true);
}
private static IEnumerable<TestCaseData> MyTestData
{
get
{
yield return new TestCaseData("firstString", "secondString").SetName("FirstTest");
yield return new TestCaseData("firstString", "secondString").SetName("SecondTest");
}
}
}
My test explorer looks like this
This seemed to be a problem with the adapter.
I was having this same issue, using SetArgDisplayNames instead, which, while not providing the expected visual result, was the best fit for this kind of usage until the problem was fixed.
Updating NUnit3TestAdapter to v3.16.0 the problem no longer occurred:
I have an ordered mstest (.orderedtest) and every time I try to run it mstest executes it not from output directory (bin\debug) but from TestResults...\out directory. This is the real problem because tests use some files from solution directory and I don`t want to copy them every time before test run. Thanks in advance.
P.s.1 Any test outside of ordered test runs fine
P.s.2 <DeploymentEnabled>False</DeploymentEnabled> didn`t help
You can priorities test methods like below.
This test case will execute first
[TestMethod]
[Priority(1)]
public void TestMethod1()
{
}
This test case will going to execute just after TestMethod1
[TestMethod]
[Priority(2)]
public void TestMethod2()
{
}
If you will not provide test case priorities then it will going to execute after all priorities test case execution just like below TestMethod3 example.
[TestMethod]
public void TestMethod3()
{
}
Same i am doing in Windows Store (metro/WinRT) apps.
I'm struggle with this one quite a long time now. Some background: I created my automated test framework using Selenium. With one project being pure NUnit tests and second one that does all the work for the tests. Now, in test project I keep the directories to all environments I run my tests against. So far many of my tests were read-only and didn't bother much about if tests did not run on environment that they supposed to. This has changed as I started to run some 'write' tests.
I need to prevent this 'Write' tests to run on any other environment then localhost. So far I tried to use method attributes and getting test method names on run time and do work then but this is not quite efficient. Can you guys point me a good solution? Thanks!
I would tag the tests to exclude with a particular category name then define a SetUp function which will stop the tests from running if they are tagged with that name and if you are on a particular environment such as Production. Place the SetUp function in a BaseClass and have all your test fixtures inherit it. The SetUp function will run before every test and prevent it from running if it needs to.
Something like this:
public class BaseSetup
{
protected const string CategoryToExclude = "Write";
[SetUp]
public void Init()
{
string env = ConfigurationManager.GetEnvironment();
if ( env == Constants.Environments.PROD && (TestContext.CurrentContext.Test.Properties["Categories"].Contains(CategoryToExclude)))
{
Assert.Inconclusive(String.Format("Cannot run this test on environment: {0}", env));
}
}
}
[TestFixture]
public class UnitTests : BaseSetup
{
[Test]
[Category(CategoryToExclude)]
public void TestMethod1()
{
Console.WriteLine("TestMethod1");
}
[Test]
public void TestMethod2()
{
Console.WriteLine("TestMethod2");
}
}
Hope this helps!
NUnit have category attribute.
You can group tests by categories and run only wanted categories.
I'm running ncrunch, in a new MVC 4 solution in VS2012 using nunit and ninject.
When I first open the solution all 50 or so test run and pass successfully.
After I make any code change (even just a added empty space) ncrunch reports that most of my unit test fail. The same thing happens if I press the 'run all tests' in the ncrunch window.
But if you hit the 'Run all tests visible here' button all 50 test pass again and ncrunch reports everything is fine.
Also when you run each test individually they pass every time.
When they do fail they seem to be failing in my ninject setup code
Error: TestFixtureSetUp failed in ControllerTestSetup
public class ControllerTestSetup
{
[SetUp]
public void InitIntegrationTest()
{
var context = IntegrationTestContext.Instance;
context.Init();
context.NinjectKernel.Load<MediGapWebTestModule>();
}
[TearDown]
public void DisposeIntegrationTest()
{
IntegrationTestContext.Instance.Dispose();
}
}
public class IntegrationTestContext : IDisposable
{
private static IntegrationTestContext _instance = null;
private static readonly object _monitor = new object();
private IntegrationTestContext() { }
public static IntegrationTestContext Instance
{
get
{
if (_instance == null)
{
lock (_monitor)
{
if (_instance == null)
{
_instance = new IntegrationTestContext();
}
}
}
return _instance;
}
}
}
All the test also run in the resharper test runner without problems every time.
Does anyone know what could be causing this?
I guessing its something to do with the singleton lock code inside the Instance property but I'm not sure.
==============================================================================
Progress:
I was able to track this down to a error in the ninject setup method above by wrapping it in a try catch statement, and writing the error to the output window.
The exception was caused by trying to load a module more than once, even tho I definitely haven't and I don't use any type of automatic module loading.
This happen on the lines
LocalSessionFactoryModule.SetMappingAssemblies(() => new[] { typeof(ProviderMap).Assembly });
_kernel.Load<LocalSessionFactoryModule>();
_sessionFactory = _kernel.Get<ISessionFactory>();
where LocalSessionFactoryModule is the ninject module class derived for NinjectModule class.
Why is this only happening with ncrunch and what can I do to solve this issue? Is there a way to check if a module has already been loaded?
NCrunch will never execute tests concurrency within the same process, so unless you have multi-threaded behaviour inside your test logic, then it should be safe to say that this isn't an issue caused by the locking or threading over the singleton.
As you've already tried disabling parallel execution and this hasn't made a difference, I'm assuming that the problem wouldn't be caused by concurrent use of resources outside the test runner process (i.e. files on disk).
This means that the problem is almost certainly related to the sequence in which the tests are being executed. Almost all manual test runners (including Resharper) will run tests in a defined sequence from start to finish. This is good for consistency, but it can mask problems that may surface when the tests are run in an inconsistent/random order. NCrunch will execute tests in order of priority and can also reuse test processes between test runs, which can make the runtime behaviour of your tests different if they haven't been designed with this in mind.
A useful way to surface (and thus debug) sequence related issues is to try running your tests in a manually defined order by using NCrunch. If you right-click a test inside the NCrunch Tests Window, under the 'Advanced' menu you'll find an option to run a test using an existing task runner process. Try this action against several of your tests to see if you can reproduce the sequence that surfaces the problem. When it happens, you should easily be able to get a debugger onto the test and find out why it is failing.
Most sequence related problems are caused by uncleared static members, so make sure each of your tests is written in the assumption that existing state may be left behind by another test that has been run within the process. Another option is to ensure all state is fully cleared by tests on tear down (although in my opinion, this is often a less pragmatic approach).
Using VS 2010, NUnit, Typemock, Entity Framework...
I am having a bit of an odd problem using NUnit/Typemock. I am relative new to the testing world so it could be a beginners mistake. This is the problem I am having.
Test Project is compiled.
Launch NUnit from within Visual Studio
Run tests from within NUnit client app. First run, first test always fails.
Rerun and test passes.
It does not matter what test is first. I can select a particular test. If it is the first one to run, it fails on first execution. It passes on rerun.
This is an example class having the problem. But the class doesn't matter. Whatever test is run first has this problem. The exception code was just something that was there to test Typemock being loaded. It fails on the WhenCalled.
[TestClass, Isolated]
public class FirstTest
{
[TestMethod]
public void TestMe()
{
Exception e = new TypeMock.ArrangeActAssert.NestedCallException();
Isolate.WhenCalled(() => UnitOfWorkManager.GetCurrentSession(null)).WillReturn(null);
Assert.IsTrue(true);
}
}
The following is the error message.
HCSO.ESL.Test.Fakes.FirstTest.TestMe:
TypeMock.ArrangeActAssert.NestedCallException :
* WhenCalled does not support using a property call as an argument.
- To fix this pass false instead of AssemblyReader.IsDotNetFile
Example - this would work:
MyObj argument = Something.Other().GetStuff();
Isolate.WhenCalled(() => ObjUnderTest.MethodUnderTest(argument))...;
Example - this would not work:
Isolate.WhenCalled(() => ObjUnderTest.MethodUnderTest(Something.Other().GetStuff()))...;
(End error message)
Anyone have an idea why the first test always fails but runs fine on rerun? Something with how the assemblies are being loaded?
(Edit)Additional Details:
Versions:
Typemock Isolator: 6.0.10.0
Visual Studio: 10.0.30319.1
In addition, I added simplified test code. This way you can see the code being tested. And yes, this test fails first time, passes on every run after that.
[TestClass, Isolated]
public class FirstTest
{
public static int DummyCall(int i)
{
return 0;
}
[TestMethod]
public void TestMe()
{
Exception e = new TypeMock.ArrangeActAssert.NestedCallException();
//Isolate.WhenCalled(() => UnitOfWorkManager.GetCurrentSession(null)).WillReturn(null);
Isolate.WhenCalled(() => FirstTest.DummyCall(-1)).WillReturn(1);
Assert.IsTrue(true);
}
}
I work at Typemock,
It seems very strange, as this is definitely not a nested call from the looks of it.
Could you please try and email us a small solution demonstrating the problem to support#typemock.com?
What is UnitOfWorkManager? Is this a class belonging to EF, or is it your code?
Also, what version of Isolator are you using?
I resolved the issue. As I expected, it was partially a newbie mistake. Inside of NUnit there is a setting to determine how the assembly is isolated. The default option is to run the tests in the same process as NUnit. I tried changing the isolation in a seperate process per assembly and the problem goes away.
To reproduce error.
* Make sure NUnit option for "Run tests directly in NUnit process" is selected.
* Close NUnit (just to make sure setting is used)
* Launch NUnit from within VS.
* Select a test containing Isolate.WhenCalled()
* Run that test first.
Thanks for the help.
[EDIT: Update]
Updating this in the event someone else has this issue.
I found that in the NUnit client if I set the following options everything works great.
Under Settings:
Test Loader -> Assembly Isolation -> Default Process Model -> Run test directly in the NUnit process.
Test Loader -> Assembly Isolation -> Default Domain Usage -> Use a seperate AppDomain per Assembly