Develop Reference

Unit test passes when running the test from within Visual Studio IDE but fails when using vstest.console.exe and Microsoft Fakes

I have a solution, which contains numerous projects. Each of these projects has an associated test project which uses MSTest V2. Some of these test projects take advantage of Microsoft Fakes.
When I run the tests in the Visual Studio IDE, everything works fine. However, my build pipeline fails. I attempted to emulate what my build pipeline is doing and it also fails on my machine.
The error message I get is the following:
System.TypeLoadException: Could not load type
'System.Fakes.ShimDateTime' from assembly 'mscorlib.4.0.0.0.Fakes,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=0ae41878053f6703'.
After playing around with the vstest.console.exe application, it seems that if I run each test assembly separately all of the tests pass, but if I attempt to pass in multiple assemblies which utilize Microsoft Fakes they fail.
vstest.console.exe "Path/To/First.dll" Passes
vstest.console.exe "Path/To/Second.dll" Passes
vstest.console.exe "Path/To/First.dll" "Path/To/Second.dll" Fails
vstest.console.exe "Path/To/First.dll" "Path/To/Second.dll" /InIsolation Fails
I'm not sure what is going on here or how to resolve this issue.

I eventually figured out what the problem was:
Test Project A had not defined System.DateTime in a Microsoft Fakes config file, so an assembly was generated named mscorlib.4.0.0.0.Fakes.dll which was missing the System.Fakes.ShimDateTime.
Test Project B had defined System.DateTime in a Microsoft Fakes config file, so an assembly was generated named mscorlib.4.0.0.0.Fakes.dll which contained the type System.Fakes.ShimDateTime.
This lead the following structure:
ProjectA.Tests/bin/Debug/
ProjectA.Tests.dll -- Contains the unit tests for Project A
mscorlib.4.0.0.0.Fakes.dll -- Does not contain System.Fakes.ShimDateTime
ProjectB.Tests/bin/Debug/
ProjectB.Tests.dll -- Contains the unit tests for Project B
mscorlib.4.0.0.0.Fakes.dll -- Does contain System.Fakes.ShimDateTime
I'm not sure what the Visual Studio IDE is doing differently, but the vstest.console.exe seems to load all of the assemblies, ignoring duplicates based on fully qualified names. That is, it loads the following files:
ProjectA.Tests/bin/Debug/ProjectA.Tests.dll
ProjectA.Tests/bin/Debug/mcsorlib.4.0.0.0.Fakes.dll
ProjectB.Tests/bin/Debug/ProjectB.Tests.dll
It ignored the file ProjectB.Tests/bin/Debug/mcsorlib.4.0.0.0.Fakes.dll, which had the type System.Fakes.ShimDateTime defined within it.
The only solution to this is to run a separate vstest.console.exe per test project (instead of all at once), or to ensure that the generated Microsoft Fakes assemblies (*.Fakes.dll) have the same types defined across the board, so it doesn't matter which version of mscorlib.4.0.0.0.Fakes.dll is loaded. And, since the Azure Pipelines VsTest task uses a single run of vstest.console.exe this is important to know.

TFS Fakes Build Unit test failure

We have a VS2013 .net 5.0 Solution (VS2013 Premium) with all Unit tests passing fine locally but failing several tests when running in VS Test Loader by TFS Build with this or similar Exception: System.TypeLoadException: Could not load type 'System.Diagnostics.Fakes.ShimEventLog' from assembly 'System.4.0.0.0.Fakes, Version=4.0.0.0, Culture=neutral, PublicKeyToken=0ae41878053f6703'.
This is an example of a failing test:
[TestMethod]
public void WriteToEventLogTest_HappyPath()
{
EventLogEntryType eTypeInfo = EventLogEntryType.Information;
bool sourceExistCalled = false;
bool writeEntrycalled = false;
using (ShimsContext.Create())
{
ShimEventLog.SourceExistsString = s =>
{
sourceExistCalled = true;
return true;
};
ShimEventLog.AllInstances.WriteEntryStringEventLogEntryType = (#this, str, et) =>
{
writeEntrycalled = true;
};
Logging.WriteToEventLog(IpAddress, eTypeInfo);
Assert.IsTrue(sourceExistCalled, "SourceExist() not called");
Assert.IsTrue(writeEntrycalled, "WriteEntry() not called");
}
}`
We using TFS 2013 update 5 running on Windows Server 2012 R2. Is there anything that can likely cause this problem? Should we update TFS to the latest which is Update 5 at the moment?

Problem was resolved by sharing fakes configuration files between the Test projects on a Solution level

In our situation, we were running several unit test DLLs through VSTest in Jenkins.
Example call:
"c:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"
"./TestDLL1.UnitTests/bin/Debug/TestDLL1.UnitTests.dll"
"./TestDLL2.UnitTests/bin/Debug/TestDLL2.UnitTests.dll"
"./TestDLL3.UnitTests/bin/Debug/TestDLL3.UnitTests.dll"
/Enablecodecoverage /UseVsixExtensions:false /Logger:trx
Some of the test projects had Fakes for the same DLL, with most setup to Fake everything. One test project was only faking one class, as the XML shows below, in order to prevent the warning about "Some fakes could not be generated."
<Fakes xmlns="http://schemas.microsoft.com/fakes/2011/" Diagnostic="true">
<Assembly Name="Utilities"/>
<StubGeneration>
<Clear />
</StubGeneration>
<ShimGeneration>
<Clear />
<Add FullName="UIUtils.ExceptionDisplay"/>
</ShimGeneration>
</Fakes>
For some reason, VSTest was using the above version for the other test projects. Modifying the one test project to generate all the fakes fixed the issue with System.TypeLoadException. We may consolidate our Fakes in the future, to be able to more easily limit which classes are faked, but for now this was a much quicker solution to the issue.
If only there was an easier way to suppress the "Some fakes could not be generated" warning...

just has a similar issue when using Shims in multiple separate Unit test projects. Apparently the generated shims can overwrite each over or something like that.
Here's the instructions I followed to fix it: Scroll to the middle of this page: https://msdn.microsoft.com/en-us/library/hh708916.aspx
It's under the heading of optimise build times, but use it to fix your problem too.
Because such assemblies rarely change on your machine, you can reuse the generated Fakes assemblies in other projects. From your unit test projects, you can simply take a reference to the
compiled Fakes assemblies that are placed under the FakesAssemblies
in the project folder.
Create a new Class Library with the .NET runtime version matching
your test projects. Let’s call it Fakes.Prebuild. Remove the
class1.cs file from the project, not needed.
Add reference to all the System and third-party assemblies you need
Fakes for. and generate the fakes, edit the .fakes file if you like,
and build to generate the fake assemblies.
From your Unit Test projects Just make sure that you have a reference
to the Fakes runtime DLL: C:\Program Files\Microsoft Visual Studio
12.0\Common7\IDE\PublicAssemblies\Microsoft.QualityTools.Testing.Fakes.dll
Then for each assembly that you have created Fakes for, add a
reference to the corresponding DLL file in the
Fakes.Prebuild\FakesAssemblies folder of your project. (This folder
get created when you compile)
To confirm You will need to add the reference by Browsing to the
generated fake assembly..

Getting started with vstest.console.exe System.IO.FileNotFoundException

I have a simple executable project and a Unit Test project all in the same solution in Visual Studio 2013. All my unit tests work fine in Visual Studio. I wanted to play* with the console app vstest.console.exe but am having a hard time getting going. I suspect I'm missing something very simple. I tried going to the directory where the actual .exe is and ran:
C:\BuildTest\Frame\Frame\obj\Debug>vstest.console.exe
C:\BuildTest\Frame\FrameUnitTests\obj\Debug\FrameUnitTests.dll
/Platform:x64 /Tests:ConstructorTest
and I got the error:
Starting test discovery, please wait...
Failed ConstructorTest
Error Message:
Test method FrameUnitTests.MainWindowPresenterTests.ConstructorTest threw
exception:
System.IO.FileNotFoundException: Could not load file or assembly
'Frame, Vers ion=3.0.0.0, Culture=neutral, PublicKeyToken=null' or one
of its dependencies. T he system cannot find the file specified.WRN:
Assembly binding logging is turned OFF.
I also tried running it from the actual directory where the unit test dll is:
C:\BuildTest\Frame\FrameUnitTests\obj\Debug>
and got the same results.
The error indicates that I need to somehow tell vstest.console where to find the main assembly that is being tested. How do I do this? Can someone point me to an example?
I'm interested in playing with vstest.console.exe for the following reason: I would like to get a printout of all my unit tests (about 80 of them) for use in documentation. I can't find a good way to do this. It occurred to me that vstest.console might be helpful for this. If there is another way to get a list of all my tests, I'd like to know. Admittedly, I could have simply typed every test title into MSWord in the time I've worked on this, but I'm a glutton for punishment and it might be nice to know how to use vstest.
I tried John Koerner's suggestion (see his answer) to get a list of tests and got the following results which might be a hint as to what I have going wrong:
C:\BuildTest\Frame\FrameUnitTests\obj\Debug>vstest.console.exe /ListTests:
FrameUnitTests.dll
Microsoft (R) Test Execution Command Line Tool Version 12.0.21005.1
Copyright (c) Microsoft Corporation. All rights reserved.
Test run will use DLL(s) built for framework Framework45 and platform X86. Follo
wing DLL(s) will not be part of run:
ACMFrameUnitTests.dll is built for Framework Framework45 and Platform X64.
Go to http://go.microsoft.com/fwlink/?LinkID=236877&clcid=0x409 for more detail
s on managing these settings.
Error: None of the provided test containers match the Platform Architecture and
.Net Framework settings for the test run. Platform: X86 .Net Framework: Frame
work45. Go to http://go.microsoft.com/?link for more details on managing these
settings.

If you only need a list of tests, you can simply run the following command:
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\ide\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" /ListTests:UnitTests.dll

Microsoft.Win32.ShimRegistryKey not working on build machine

I Have lots of unit tests where I shim the Microsoft.Win32.RegistryKey using Microsoft Fakes. When I run the build on my local machine the unit test passed. But if I run it on build machine the unit test fails with the following error for these tests.
System.TypeLoadException: Could not load type
'Microsoft.Win32.Fakes.ShimRegistryKey' from assembly
'mscorlib.4.0.0.0.Fakes, Version=4.0.0.0, Culture=neutral,
PublicKeyToken=0ae41878053f6703'.
The curious thing is that both my local machine and build machine are 64-bit and the unit test project are targeting AnyCPU. Is there some configuration could be missing on the build machine?

I just has a similar issue when using Shims in multiple separate Unit test projects. Apparently the generated shims can overwrite each over or something like that.
Here's the instructions I followed to fix it:
Scroll to the middle of this page: https://msdn.microsoft.com/en-us/library/hh708916.aspx
It's under the heading of optimise build times, but use it to fix your problem too.
Because such assemblies rarely change on your machine, you can reuse the generated Fakes assemblies in other projects.
From your unit test projects, you can simply take a reference to the compiled Fakes assemblies that are placed under the FakesAssemblies in the project folder.
Create a new Class Library with the .NET runtime version matching
your test projects. Let’s call it Fakes.Prebuild. Remove the
class1.cs file from the project, not needed.
Add reference to all the System and third-party assemblies you need
Fakes for. and generate the fakes, edit the .fakes file if you like,
and build to generate the fake assemblies.
From your Unit Test projects Just make sure that you have a reference
to the Fakes runtime DLL: C:\Program Files\Microsoft Visual Studio
12.0\Common7\IDE\PublicAssemblies\Microsoft.QualityTools.Testing.Fakes.dll
Then for each assembly that you have created Fakes for, add a
reference to the corresponding DLL file in the
Fakes.Prebuild\FakesAssemblies folder of your project. (This folder
get created when you compile)
To confirm You will need to add the reference by Browsing to the
generated fake assembly..

Can't build project with Fakes assembly

I've added a Fakes assembly to my Visual Studio 2012 unit test project, corresponding to my application project (the System Under Test), as described in Peter Provost's article. However, the project will no longer build, on account of an unresolved type reference in the generated Fakes code:
The type or namespace name 'FieldDefinition' does not exist in the namespace 'bc::ProductStudio' (are you missing an assembly reference?)
[C:\Users\arvek\Projects\Project\Project.Tests\obj\Debug\Fakes\bc\f.csproj]
C:\Users\arvek\Projects\Project\Project.Tests\f.cs
What's going wrong here? From what I know, this is supposed to just work, so it would seem to me there's a bug in the Fakes facilities.

This bug is present in VS2013 as well. Link to MSDN bug.
Work Around: Delete file .messages from FakeAssemblies folder.

The error is most likely due to bug in Fakes triggered by the faked assembly. I've submitted the issue to Microsoft.

One option that may help you diagnose the issue is turning on diagnostics for you fakes. On your .fakes file.
<Fakes xmlns="http://schemas.microsoft.com/fakes/2011" Diagnostic="true" Verbosity="Noisy">
Also, make sure that your "MSBuild project build output verbosity" is set to Diagnostic. You can find this in Tools -> Options -> Projects and Solutions -> Build and Run.
Rebuild you test project and now your output window should be full of info including any failures for Fakes

Deleting Fakes folder will resolve this issue.

You're not going to believe this, but I was able to get the fakes working again by simply adding a new line to the file, removing that new line, and then saving what is effectively an unmodified file. After the next build everything was fine.
Things that make you go... hmm...