Unit testing multi-threaded code using Mocks

The worse kind of "unit test" is an inconsistent one. A test that fails (and pass) without any cause can reduce the entire test suite "credibility". Once developers get used to the same old test that fails then pass and then fail again they start to ignore it and pretty soon when a build fails no one even cares. Yet when faced with multi-threaded or asynchronous code most developers would choose to write such a test.

It is widely believed that since such code is inherently non-deterministic it could only be tested (partially) by tests which are as bad...
But the truth is that there are ways to write robust unit tests that test multi-threaded code in a simple, deterministic way - by using fake objects.

a quick intro to mocking

In most cases writing unit tests require us to remove external dependencies. Those dependencies are used throughout our code and prevent us from testing a specific behavior. Creating a full fledged database or connecting to an external server before each test would be costly and introduces additional complexity to our tests. Instead we use fake objects that look just like the real object but with specific behavior relevant only to the current test.
Consider the following method:

public bool CheckPassword(string userName, string password)
{
    User user = _dataAccess.GetUser(userName);
    if (user != null)
    {
       if (user.VerifyPassword(password))
       {
           return true;
       }
    }

    return false;
}

In order to test this method would need to replace _dataAccess with a fake object that would return a specific user. To do so we can create a new type that has a similar signature as our original DataAccess but has no implementation. Since we do not want nor need to maintain yet another class for the sake of our tests we'll use a Mocking framework to create that fake instance for us.

[Test]
public void CheckPassword_ValidUserAndPassword_ReturnTrue()
{
    var userForTest = new User("President Skroob", "12345");
    var fakeDataAccess = Isolate.Fake.Instance<DataAccess>();
    Isolate.WhenCalled(() => fake.DataAccess.GetUser("")).WillReturn(userForTest);
    ...
}

The first line creates a new User for our test. Then we create a fake DataAccess using Typemock Isolator and define a new behavior - when GetUser will be called, return the user instance previously created.

Note: Although I'll be using Isolator in order to keep some of the examples simple. The principles outlined in this post should work with any Mocking framework

But what do fake object has with making parallel execution simple? Just keep on reading...

Replacing concurrent objects with fakes

For our first example consider the following class:

public class ClassWithTimer
{
    private readonly Timer _timer;

    public ClassWithTimer(Timer timer)
    {
        _timer = timer;
        _timer.Elapsed += OnTimerElapsed;
    }

    private void OnTimerElapsed(object sender, ElapsedEventArgs e)
    {
        SomethingImportantHappened = true;
    }

    public bool SomethingImportantHappened { get; private set; }
}

The problem with this class is the Timer spawn new thread (or threads) and it's hard to know if the Timer has already raised its Elapsed event or if we need to wait a while more before checking.

Faced with this kind of code some developers would write the following code:

[Test]
public void ThisIsABadTest()
{
    var timer = new Timer(1);

    var cut = new ClassWithTimer(timer);
    timer.Start()

    Thread.Sleep(1000);

    Assert.IsTrue(cut.SomethingImportantHappened);
}

In this test we assume that since the timer's interval is smaller than the sleep interval - this is not always true. I have seen cases where such timer's event was only raised after the timer's second.
Another problem is that such test would run for a full second where a unit test should be faster then that. Imagine what would happen if you have 1000 such tests - they would run for more then 15 min.
Using fake object in this case is simple. Since the timer logic is not part of the logic we're testing we can fake it and cause the Elapsed event in the exact moment its needed.

We need to create a fake timer and cause its Elapsed event to signal in the exact moment we want:

[Test]
public void ThisIsAGoodTest()
{
    var fakeTimer = Isolate.Fake.Instance<Timer>();

    var cut = new ClassWithTimer(fakeTimer);

    var fakeArgs = Isolate.Fake.Instance<ElapsedEventArgs>();
    Isolate.Invoke.Event(() => fakeTimer.Elapsed += null, this, fakeArgs);

    Assert.IsTrue(cut.SomethingImportantHappened);
}

The example above creates a fake Timer on the first line and pass it to the class we're testing. Then we use a weird syntax (that's the +=) to cause the event to be raised just before we check the result.

Faking problematic objects is what a mocking framework was created for. There is however another use for a Mocking framework - showing us that a specific event or step has passed.

Using fakes to signal execution stage

Sometimes we have no choice but to run multi-threaded code in the context of our unit test. In those cases we want to make sure that the executing code has reached a certain stage before checking for the test result.
The following class has an operation performed in another task and we need to make sure it happened before we check the result:

public class ClassWithAsyncOperation
{
    private IOtherClass _otherClass;

    public ClassWithAsyncOperation(IOtherClass otherClass)
    {
        _otherClass = otherClass;
    }

    public void DiffcultCalcAsync(int a, int b)
    {
        Task.Run(() =>
        {
            Result = a + b;

            _otherClass.DoSomething(Result);
        });
    }

    public int Result { get; private set; }
}

In other words - we do not check for Result before the task has finished execution or at least passed the code that calculates it (i.e. Result = a + b).
We can use the _otherClass to make sure we've passed that state. All we need to do is replace it's behavior with code that would signal us when _otherClass.DoSomething is called:

[Test]
public void TestUsingSignal()
{
    var waitHandle = new ManualResetEventSlim(false);

    var fakeOtherClass = A.Fake<IOtherClass>();
    A.CallTo(() => fakeOtherClass.DoSomething(A<int>._)).Invokes(waitHandle.Set);

    var cut = new ClassWithAsyncOperation(fakeOtherClass);

    cut.DiffcultCalcAsync(2, 3);

    var wasCalled = waitHandle.Wait(10000);

    Assert.IsTrue(wasCalled, "OtherClass.DoSomething was never called");
    Assert.AreEqual(5, cut.Result);
}

What we did in the test is create a fake class and then set a behavior on it to invoke custom code. The code that would be invoked will set a WaitHandle.
The next order of business is to start the test and then wait on that same WaitHandle until it is signaled (which means we can test the result now) or a very long time has passed - which means that there's a bug in the code.

And so we've used a fake object to signal the test that a specific event has occurred and that it can continue to the next step.

Conclusion

In this post I've shown two strategies of testing concurrent code using fake objects.
The first example showed how we can replace the concurrent part with a fake object and run the code under test in a single thread.
The second example showed how we can use a fake dependency to "tell" the test that a specific action has run.
There are several other such tricks (or patterns) and if you want to hear more about them, all you need to do is come to my NDC Oslo session Unit testing patterns for concurrent code.

See you there...