I'm having some trouble with computed data from a static class in a C# Xunit test being computed twice.
The actual production code this would be used for is much more complicated, but the code that follows is enough to exhibit the issue I am seeing.
In the code below I have a randomly generated, lazily loaded int seeded off of the current time.
All I am testing here is that this property is equal to itself. I insert the property's value into the test via a MemberData function.
Since the property ought to only be initialized once, I'd expect that this test should always pass. I would expect that the static field would be initialized when the RandomIntMemberData function is run and never again.
However, this test consistently fails. The value inserted into the test, and the value tested against are always different.
Further if I debug, I only see the initialization code being hit once. That is, the value being tested. I never see the initialization of the value being tested against.
Am I misunderstanding something, or is Xunit doing some weird behind the scenes magic to setup it's input data, then initializing the value again when the test is actually run?
Minimal Code to Reproduce Bug
public static class TestRandoIntStaticClass
{
private static readonly Lazy<int> LazyRandomInt = new Lazy<int>(() =>
{
// lazily initialize a random interger seeded off of the current time
// according to readings, this should happen only once
return new Random((int) DateTime.Now.Ticks).Next();
});
// according to readings, this should be a thread safe operation
public static int RandomInt => LazyRandomInt.Value;
}
The Test
public class TestClass
{
public static IEnumerable<object[]> RandomIntMemberData()
{
var randomInt = new List<object[]>
{
new object[] {TestRandoIntStaticClass.RandomInt},
};
return randomInt as IEnumerable<object[]>;
}
[Theory]
[MemberData(nameof(RandomIntMemberData))]
public void RandoTest(int rando)
{
// these two ought to be equal if TestRandoIntStaticClass.RandomInt is only initialized once
Assert.True(rando == TestRandoIntStaticClass.RandomInt,
$"{nameof(rando)} = {rando} but {nameof(TestRandoIntStaticClass.RandomInt)} = {TestRandoIntStaticClass.RandomInt}");
}
}
At the time of tests discovery, Visual Studio Xunit console runner creates AppDomain with test data for all attributes like MemberData, ClassData, DataAttribute so all data are just saved in memory after build (that is also why XUnit require classes to be serializable).
We can verify this by adding a simple logger to your methods:
namespace XUnitTestProject1
{
public class TestClass
{
public static IEnumerable<object[]> RandomIntMemberData()
{
var randomInt = new List<object[]>
{
new object[]
{TestRandoIntStaticClass.RandomInt},
};
return randomInt;
}
[Theory]
[MemberData(nameof(RandomIntMemberData))]
public void RandoTest(int rando)
{
// these two ought to be equal if TestRandoIntStaticClass.RandomInt is only initialized once
Assert.True(rando == TestRandoIntStaticClass.RandomInt, $"{nameof(rando)} = {rando} but {nameof(TestRandoIntStaticClass.RandomInt)} = {TestRandoIntStaticClass.RandomInt}");
}
}
public static class TestRandoIntStaticClass
{
private static readonly Lazy<int> LazyRandomInt = new Lazy<int>(() =>
{ // lazily initialize a random interger seeded off of the current time
// according to readings, this should happen only once
var randomValue = new Random((int) DateTime.Now.Ticks).Next();
File.AppendAllText(#"D:\var\log.txt", $"Call TestRandoIntStaticClass {randomValue}; ThreadId {Thread.CurrentThread.ManagedThreadId} " + Environment.NewLine);
return randomValue;
});
public static int RandomInt => LazyRandomInt.Value; // according to readings, this should be a thread safe operation
}
}
As a result we see in logs:
> Call TestRandoIntStaticClass 1846311153; ThreadId 11
> Call TestRandoIntStaticClass 1007825738; ThreadId 14
And in test execution result
rando = 1846311153 but RandomInt = 1007825738
Expected: True
Actual: False
at
However, if you use dotnet test will be successful because 'data generation' and test run will be launched on one process
Related
I've noticed that NUnit's TestContext.CurrentContext.Result.Outcome.Status is always Inconclusive at the end of a test run. The CurrentContext is also unaware that any assertions have taken place.
Is it possible to get the status of a test before the [TearDown]?
I was hoping to use the value during the Dispose() of my test management class to capture metrics and other data for post-test diagnosis.
Code example from a new .NET Framework 4.6.1 project that only has the NuGet packages NUnit and FluentAssertions:
namespace TestProject
{
using FluentAssertions;
using NUnit.Framework;
[TestFixture]
public class Class1
{
[Test]
public void test1()
{
var a = 1;
var b = 2;
var c = 1;
var context = TestContext.CurrentContext;
a.Should().Be(c);
Assert.AreEqual(a, c);
}
}
}
A test result starts out as Inconclusive. If the test is skipped or ignored, then the result changes, but of course it is never executed.
If it is executed, the Outcome is Inconclusive until the test is over. Clearly, while you are still executing the test, it is not yet finished. When teardown begins, the outcome of the test is known, so it will vary according to whether the test method itself was successful. Of course, an exception in teardown may change the result to an Error state.
Bottom line, the Outcome field is not useful while the test method itself is still running. In any case, if you are executing code in the test method, the test has not yet failed. Otherwise, you would not have continued execution!
You say you can't use TearDown but the link you provide doesn't deal with the issue of accessing the test result. Can you explain further? What exactly do you want to do after checking the test result?
You could try using AssertionScope along with the AssertionScope.Succeeded flag. Although the intelisense on that property specifies:
Gets a value indicating whether or not the last assertion executed through this scope succeeded.
Example usage of AssertionScope
[Test]
public void Test()
{
var a = 1;
var b = 2;
var c = 1;
var context = new AssertionScope();
try
{
throw new Exception("Test");
}
catch (Exception e)
{
context.FailWith(e.ToString());
}
var strings = context.Discard();
Console.WriteLine(strings.StringJoin(","));
context.Succeeded.Should().BeFalse();
var someOtherContext = new AssertionScope();
try
{
c.Should().Be(a);
}
catch (Exception e)
{
someOtherContext.FailWith(e.ToString());
}
var discard = someOtherContext.Discard();
Console.WriteLine(discard.StringJoin(","));
someOtherContext.Succeeded.Should().BeTrue();
}
The following test fails intermittently. It caches an item in MemoryCache with an absolute expiration time, and an update callback that should be called before the item is removed. However sometimes the callback is invoked before the test finishes, and sometimes not at all.
With a large enough buffer time it will always be invoked at least once. But that does not serve my purposes, since I require that the cache always attempts to update the data before it expired.
Now in my real world scenario I will not have 10 second expiration time and granularity, but it still bothers me that this test fails intermittently.
Anyone have thoughts on why this is happening?
Note: Also intermittently fails with 60 second expiry and 5 second buffer.
using System;
using System.Runtime.Caching;
using System.Threading.Tasks;
using Microsoft.VisualStudio.TestTools.UnitTesting;
[TestClass]
public class MemoryCacheTest
{
private const double ExpiryInSeconds = 10;
private const double ExpiryBufferInSeconds = 5;
private readonly object updateItemCounterLock = new object();
private int updateItemCounter = 0;
[TestMethod]
public async Task MemoryCacheUpdateTest()
{
// Set item in cache with absolute expiration defined above
MemoryCache cache = MemoryCache.Default;
CacheItem cacheItem = new CacheItem("key", "value");
CacheItemPolicy cacheItemPolicy = new CacheItemPolicy
{
AbsoluteExpiration = DateTimeOffset.Now + TimeSpan.FromSeconds(ExpiryInSeconds),
UpdateCallback = new CacheEntryUpdateCallback(this.UpdateItem)
};
cache.Set(cacheItem, cacheItemPolicy);
// Delay for absolute expiration time + buffer
await Task.Delay(TimeSpan.FromSeconds(ExpiryInSeconds) + TimeSpan.FromSeconds(ExpiryBufferInSeconds));
// Test that the update callback was invoked once
Assert.AreEqual(1, updateItemCounter);
}
// Incrememnts the updateItemCounter
private void UpdateItem(CacheEntryUpdateArguments args)
{
lock (updateItemCounterLock)
{
updateItemCounter++;
}
}
}
I suppose calling new CacheEntryUpdateCallback is redundant. You can call:
UpdateCallback = new CacheEntryUpdateCallback(this.UpdateItem) instead
Since there was no solution to this question, I abstracted the MemoryCache methods that I needed into an interface and tested against that. At that point the test became invalid because I would have just been testing my own implementation of the interface.
In debugging step through, Visual Studio 2013 shows BitConverter.IsLittleEndian is:
false: When I hover mouse on BitConverter and see the value of BitConverter.IsLittleEndian and
true: When I put it in a variable like var x = BitConverter.IsLittleEndian;
I assume BitConverter.IsLittleEndian should be already evaluated because I have called GetBytes on BitConverter so it's static constructor should be called at this point, right? What am I missing?
My code is this (I wanted to generate sequential Guid; rest is bytes of a long counter - at this version):
static Guid Id(long ticks, byte[] rest)
{
var ticksBytes = BitConverter.GetBytes(ticks).PolishEndian();
// var x = BitConverter.IsLittleEndian; // <- TESTED HERE
int a = BitConverter.ToInt32(new byte[] { ticksBytes[4], ticksBytes[5], ticksBytes[6], ticksBytes[7] }.PolishEndian(), 0);
short b = BitConverter.ToInt16(new byte[] { ticksBytes[2], ticksBytes[3] }.PolishEndian(), 0);
short c = BitConverter.ToInt16(new byte[] { ticksBytes[0], ticksBytes[1] }.PolishEndian(), 0);
return new Guid(a, b, c, rest.PolishEndian(true).ToArray());
}
static byte[] PolishEndian(this byte[] ba, bool reverse = false)
{
var flag = reverse ? BitConverter.IsLittleEndian : !BitConverter.IsLittleEndian;
if (flag) return ba.Reverse().ToArray();
return ba;
}
Note that in this case IsLittleEndian is actually a field and not a property. That has an effect on how the EE is able to process the value.
I tried this out locally and this is the behavior I saw
First i stepped until the cursor hit the var ticksBytes line. At that point I observed that IsLittleEndian == false. This is actually expected at this point. The EE does not always need to force a static constructor to run in order to read fields. Hence it is just reading the value as is and because no other code for BitConverter has run the value is false
Immediately after stepping over that line I observe that IsLittleEndian == true. The CLR ran the static constructor in order to execute the GetBytes method and hence that set the field. The EE was then reading the set field.
Note that you can recreate this example with your own code. For example
static class Test {
static readonly bool example;
static Test() {
example = true;
}
internal static void Go() {
// example == true
}
}
class Program {
static void Main() {
// Test.example == false;
Test.Go();
}
}
Earlier I mentioned that the EE didn't always need to execute a static constructor in order to read fields. One case where it often needs to is when reading static fields off of a generic type. The storage for a static field of a generic type isn't created essentially until the CLR instantiates an instance of the type. Hence in order to read a field off of a generic type which hasn't yet been used the EE will create an instance under the cover in order to force the CLR to read it. For example
static class Test<T>
{
static readonly bool example = false;
static Test()
{
example = true;
}
}
If you add this to your program and then evaluate the following in the watch window
Test<int>.example
you will find that the value is true clearly indicating the cctor ran
I'm trying to use this great project but since i need to scan many images the process takes a lot of time so i was thinking about multi-threading it.
However, since the class that makes the actual processing of the images uses Static methods and is manipulating Objects by ref i'm not really sure how to do it right. the method that I call from my main Thread is:
public static void ScanPage(ref System.Collections.ArrayList CodesRead, Bitmap bmp, int numscans, ScanDirection direction, BarcodeType types)
{
//added only the signature, actual class has over 1000 rows
//inside this function there are calls to other
//static functions that makes some image processing
}
My question is if it's safe to use use this function like this:
List<string> filePaths = new List<string>();
Parallel.For(0, filePaths.Count, a =>
{
ArrayList al = new ArrayList();
BarcodeImaging.ScanPage(ref al, ...);
});
I've spent hours debugging it and most of the time the results i got were correct but i did encounter several errors which i now can't seem to reproduce.
EDIT
I pasted the code of the class to here: http://pastebin.com/UeE6qBHx
I'm pretty sure it is thread safe.
There are two fields, which are configuration fields and are not modified inside the class.
So basically this class has no state and all calculation has no side effects
(Unless I don't see something very obscure).
Ref modifier is not needed here, because the reference is not modified.
There's no way of telling unless you know if it stores values in local variables or in a field (in the static class, not the method).
All local variables will be fine and instanced per call, but the fields will not.
A very bad example:
public static class TestClass
{
public static double Data;
public static string StringData = "";
// Can, and will quite often, return wrong values.
// for example returning the result of f(8) instead of f(5)
// if Data is changed before StringData is calculated.
public static string ChangeStaticVariables(int x)
{
Data = Math.Sqrt(x) + Math.Sqrt(x);
StringData = Data.ToString("0.000");
return StringData;
}
// Won't return the wrong values, as the variables
// can't be changed by other threads.
public static string NonStaticVariables(int x)
{
var tData = Math.Sqrt(x) + Math.Sqrt(x);
return Data.ToString("0.000");
}
}
Can you help explain how multiple threads access static methods? Are multiple threads able to access the static method concurrently?
To me it would seem logical that if a method is static that would make it a single resouce that is shared by all the threads. Therefore only one thread would be able to use it at a time. I have created a console app to test this. But from the results of my test it would appear that my assumption is incorrect.
In my test a number of Worker objects are constructed. Each Worker has a number of passwords and keys. Each Worker has an instance method that hashes it's passwords with it's keys. There is also a static method which has exactly the same implementation, the only difference being that it is static. After all the Worker objects have been created the start time is written to the console. Then a DoInstanceWork event is raised and all of the Worker objects queue their useInstanceMethod to the threadpool. When all the methods or all the Worker objects have completed the time it took for them all to complete is calculated from the start time and is written to the console. Then the start time is set to the current time and the DoStaticWork event is raised. This time all the Worker objects queue their useStaticMethod to the threadpool. And when all these method calls have completed the time it took until they had all completed is again calculated and written to the console.
I was expecting the time taken when the objects use their instance method to be 1/8 of the time taken when they use the static method. 1/8 because my machine has 4 cores and 8 virtual threads. But it wasn't. In fact the time taken when using the static method was actually fractionally faster.
How is this so? What is happening under the hood? Does each thread get it's own copy of the static method?
Here is the Console app-
using System;
using System.Collections.Generic;
using System.Security.Cryptography;
using System.Threading;
namespace bottleneckTest
{
public delegate void workDelegate();
class Program
{
static int num = 1024;
public static DateTime start;
static int complete = 0;
public static event workDelegate DoInstanceWork;
public static event workDelegate DoStaticWork;
static bool flag = false;
static void Main(string[] args)
{
List<Worker> workers = new List<Worker>();
for( int i = 0; i < num; i++){
workers.Add(new Worker(i, num));
}
start = DateTime.UtcNow;
Console.WriteLine(start.ToString());
DoInstanceWork();
Console.ReadLine();
}
public static void Timer()
{
complete++;
if (complete == num)
{
TimeSpan duration = DateTime.UtcNow - Program.start;
Console.WriteLine("Duration: {0}", duration.ToString());
complete = 0;
if (!flag)
{
flag = true;
Program.start = DateTime.UtcNow;
DoStaticWork();
}
}
}
}
public class Worker
{
int _id;
int _num;
KeyedHashAlgorithm hashAlgorithm;
int keyLength;
Random random;
List<byte[]> _passwords;
List<byte[]> _keys;
List<byte[]> hashes;
public Worker(int id, int num)
{
this._id = id;
this._num = num;
hashAlgorithm = KeyedHashAlgorithm.Create("HMACSHA256");
keyLength = hashAlgorithm.Key.Length;
random = new Random();
_passwords = new List<byte[]>();
_keys = new List<byte[]>();
hashes = new List<byte[]>();
for (int i = 0; i < num; i++)
{
byte[] key = new byte[keyLength];
new RNGCryptoServiceProvider().GetBytes(key);
_keys.Add(key);
int passwordLength = random.Next(8, 20);
byte[] password = new byte[passwordLength * 2];
random.NextBytes(password);
_passwords.Add(password);
}
Program.DoInstanceWork += new workDelegate(doInstanceWork);
Program.DoStaticWork += new workDelegate(doStaticWork);
}
public void doInstanceWork()
{
ThreadPool.QueueUserWorkItem(useInstanceMethod, new WorkerArgs() { num = _num, keys = _keys, passwords = _passwords });
}
public void doStaticWork()
{
ThreadPool.QueueUserWorkItem(useStaticMethod, new WorkerArgs() { num = _num, keys = _keys, passwords = _passwords });
}
public void useInstanceMethod(object args)
{
WorkerArgs workerArgs = (WorkerArgs)args;
for (int i = 0; i < workerArgs.num; i++)
{
KeyedHashAlgorithm hashAlgorithm = KeyedHashAlgorithm.Create("HMACSHA256");
hashAlgorithm.Key = workerArgs.keys[i];
byte[] hash = hashAlgorithm.ComputeHash(workerArgs.passwords[i]);
}
Program.Timer();
}
public static void useStaticMethod(object args)
{
WorkerArgs workerArgs = (WorkerArgs)args;
for (int i = 0; i < workerArgs.num; i++)
{
KeyedHashAlgorithm hashAlgorithm = KeyedHashAlgorithm.Create("HMACSHA256");
hashAlgorithm.Key = workerArgs.keys[i];
byte[] hash = hashAlgorithm.ComputeHash(workerArgs.passwords[i]);
}
Program.Timer();
}
public class WorkerArgs
{
public int num;
public List<byte[]> passwords;
public List<byte[]> keys;
}
}
}
Methods are code - there's no problem with thread accessing that code concurrently since the code isn't modified by running it; it's a read-only resource (jitter aside). What needs to be handled carefully in multi-threaded situations is access to data concurrently (and more specifically, when modifying that data is a possibility). Whether a method is static or an instance method has nothing to do with whether or not it needs to ne serialized in some way to make it threadsafe.
In all cases, whether static or instance, any thread can access any method at any time unless you do explicit work to prevent it.
For example, you can create a lock to ensure only a single thread can access a given method, but C# will not do that for you.
Think of it like watching TV. A TV does nothing to prevent multiple people from watching it at the same time, and as long as everybody watching it wants to see the same show, there's no problem. You certainly wouldn't want a TV to only allow one person to watch it at once just because multiple people might want to watch different shows, right? So if people want to watch different shows, they need some sort of mechanism external to the TV itself (perhaps having a single remote control that the current viewer holds onto for the duration of his show) to make sure that one guy doesn't change the channel to his show while another guy is watching.
C# methods are "reentrant" (As in most languages; the last time I heard of genuinely non-reentrant code was DOS routines) Each thread has its own call stack, and when a method is called, the call stack of that thread is updated to have space for the return address, calling parameters, return value, local values, etc.
Suppose Thread1 and Thread2 calls the method M concurrently and M has a local int variable n. The call stack of Thread1 is seperate from the call stack of Thread2, so n will have two different instantiations in two different stacks. Concurrency would be a problem only if n is stored not in a stack but say in the same register (i.e. in a shared resource) CLR (or is it Windows?) is careful not to let that cause a problem and cleans, stores and restores the registers when switching threads. (What do you do in presence of multiple CPU's, how do you allocate registers, how do you implement locking. These are indeed difficult problems that makes one respect compiler, OS writers when one comes to think of it)
Being reentrant does not prove no bad things happen when two threads call the same method at the same time: it only proves no bad things happen if the method does not access and update other shared resources.
When you access an instance method, you are accessing it through an object reference.
When you access a static method, you are accessing it directly.
So static methods are a tiny bit faster.
When you instanciate a class you dont create a copy of the code. You have a pointer to the definition of the class, and the code is acceded through it. So, instance methods are accessed the sane way than static methods