I've gotten the chance to visit and consult various companies and I see this all the time: Over specification in tests, especially when doing mocks and stubs.

One main reason for that is that people use their stub as a mock object (stubs are fake objects that we will not assert on so that we can test something else). here is a small example from something posted on the alt.net mailing list:

[Test]

public void Can_Return_All_Plans()

{

IPlanDao mockPlanDao = MockRepository.GenerateMock<IPlanDao>();

mockPlanDao.Expect(x => x.AllPlans(_fakeAdmin)).Return(new List<Plan>{_fakePlan});

PlanController planController = new PlanControllerForTesting(_mockCommonDao, _mockCommonService, mockPlanDao);

ViewResult result = planController.All(false);

mockPlanDao.VerifyAllExpectations();

result.AssertViewResultNameAndViewDataType(typeof(PlanContainer), "Plans");

}

in this test the following line is used to "stub" out a return value into the system under test.

mockPlanDao.Expect(x => x.AllPlans(_fakeAdmin)).Return(new List<Plan>{_fakePlan});

no problem here. The problem lies in the last two lines of the test:

mockPlanDao.VerifyAllExpectations();

result.AssertViewResultNameAndViewDataType(typeof(PlanContainer), "Plans");

the second line (the assert) is probably the thing you really want to test. but the call to "verifyAllExpectations" is the one that makes the test very fragile. It is "asserting" that someone has actually called "GetAllPlans()" to get the results, when it clearly does not matter how one got the results. all that matters is that:

"Given a set of results returned into the application under test, the view in the controller is correct" (the translation of the last line in the test).

If by any chance the application later does several more calls to the stubbed out dao, or uses a different method to get results, the test could break, even though the application would still ultimately work with the same end result.

It's like me ordering a pizza delivery and then asking the delivery guy "did you get here by car or motorcycle?" . I realy don't care, as long as the end result: PIZZA, is correct.

So the general rules for me are:

• If you see both "Verify" and "Assert" in the same test, it is usualy a smell of over specified tests
• if you see "expect" and "Return" on a fake object, make sure that you name it "stubXX" or "mockXX" so that you can distinguish whether you want to call verify on it later or not(most cases should be "not")
• Try to test on the end result or end state rather than verify interactions.  The only time you absolutely have not choice but to test an interaction using verify is when calling void methods on external objects. that is clearly a one-way communication and is (or part of)the end result of what you are trying to accomplish.

A lot of people think that adding that extra "verify" just means it is a good thing since they are doing more assert. Well, they sure are testing more things, but they are internal things to the app's behavior and that is usually leading to brittle tests. Try to test on the end result or end state rather than verify interactions.

Here's a sneak preview of the upcoming Typemock Racer product, which I'm currently working on. We should be out in private CTP in a couple of weeks I hope.

Typemock Racer tries to solve one huge problem for developers working in multi threaded environments: Detecting code that can result in deadlocks or race conditions. It does this by providing a framework API which you can use in your tests (under NUnit for example), that allows you to execrise your code under test until it finds a deadlock or times out.

Here's an example of how you'd use it. Consider the following class which has two methods, that if run by two seperate threads, can result in a deadlock condition:

At some point in time the threads could execute such that thread 1 is waiting on resource b, which is held by thread two , which is waiting on resource a, held by thread 1 - a pure deadlock.

Given this class we can write a test that looks like this:

The ThreadTest class has an "AddThreadAction" method that receives a delegate. you can use it to invoke your own code, it will be invoked in a different thread. IN this example we are creating two threads, each one running a different method on MyClassWithLocks.

We can just right click and run the test with TestDriven.NET to recieve the following results:

The output states that there is a deadlock that was found, with the exact steps that happened (with line locations) to reproduce this.

It currently only works with lock (monitor.Enter\Exit) but will support all locking constructs in .NET.

What if the methods you point to create their own threads? Like this:

Here is the output that you'd get if you run this test:

Update: Here's how you can extend Depender with your own dependency rules

Depender is a tool that I created to illustrate several concepts both for my book and both for my training classes. I think it should prove valuable (as it gets more and more features) to anyone who is trying to write tests for existing code.

Depender allows you to load an assembly and inspect the types within it according to very simple rules. the main question on my mind and I run it is: "How easy would it be to test class X?". Depender will list all the classes in the assembly and for each one will try to detect possible testability problems (non decoupling) and places where you might be able to "inject" your own implementation (calls to virtual methods, fields that are interfaces or types who are not sealed..etc..)

so you can:

• see places where you might want to refactor your code or use a tool like Typemock Isolator
• see places where you can already use "seams" to inject behavior into your code without needing to look up the source code manually.

here's what the main screen looks like when loading an assembly:

the type "RealLogger" is marked in red because it contains calls to static methods. double clicking on the type's node will load a separate window just for this type:

I will be adding more rules and also releasing the source so that other people can help develop this. for now I just want to get it out there and see what people think.

possible futures:

• Make it command line driven so it can run in your CI builds
• Make it give "scores" to assemblies or types on testability and decoupling
• Make it add "suggestions" on how to fix decoupling issues
• Make it create the base diagram for my test feasibility matrix

Download Depender from here.

This is an interesting problem I've been running into at work. Sometimes we would try to write a test against the system or a component and we would literally have too many choices of how to write it or where to "intercept" or "inject" our stubs or mock objects.  In a way, the software is almost "too testable" in that there is usually at least 3 ways to accomplish a similar level of isolation.

What it comes down to is the fact that for a real system, there are different "depth" layers where indirection can be injected. For example, Component X needs object Y that relies on object Z. If we were testing component X we could "replace" either Y or Z.  Some would argue that replacing one level too deep is making this an integration test, but as long as its all running in memory I don't see a real problem with that. I gave up on trying to do "pure" unit tests for 100% of the time a long time ago.

Many times, it is much more complex than this. For example, replacing object Y instead of Z (at the closes layer to the object under test) can also lead to problems if:

• The contracts between X and Y are very chatty, than stubbing out Y could lead to a very hard to read or very fragile test
• You are trying to test that interaction occurs between Y and X than that might be an over specified test. If replacing Z would lead to a more state-base test that I'd choose that over replacing X. (shortly: I'd rather use stubs over mocks so that my tests become less fragile. some layers require mocking and some are very simple to just stub out)

Here's a thought or two:

if you one one layer farther away than the one you are using, and it does not require mocks to test, that try to use that layer and forget the inner layer?

If you treat mocks as a smell for over specified tests that will force you to find a way to test the same thing using a stub. I find that sometimes you have no other choice but use mocks (when your app under test has a one way out communication with a component) but in all other cases, I am starting to look at it like a design smell for my tests (over specified tests)

getting back to the original problem (too many ways to test the system) I am trying to think if this is a good or bad thing. The only bad here is that someone who is testing the system for the first time has no real guidance on where to replace things (except look at other tests) which can make writing the first few tests really hard to figure out.

The fact that we are using a sort of auto mocking container in our tests is both a blessing and a bit of a curse. Currently i find that it saves more time that it takes away. the tests and their dependencies are relatively more readable since we are using a more explicit version of a test container that requires you to explicitly state what will be replaced (it just doesn't tell you who requires that component).

I can already hear the pundits say "well that means you have a design smell". The only thing that smells here to me is that because we followed strict DI and interface based injection rules, we ended up with a system that is not just made of lego parts (a good thing), the lego parts or too darn small!

My interview at teched us on advanced unit testing and legacy code can be seen here:

http://msdn.microsoft.com/en-us/events/teched/cc676818.aspx 

or downloaded here

We occasionally stop and laugh a bit because "speaker idol" contest was being held at the booth next to us, which was really really loud. 

This great news was just published at the alt.net mailing list. putting it here for future reference.

"Fixture Gallery 2.0 (release 2008-06-09) is now available for download
from SourceForge.
Fixture Gallery is a cookbook for FIT/FitNesse tests. It provides
developers with a quick overview of the most important fixture types and
concepts for agile acceptance testing using the FIT framework. For each
fixture type, this document explains the table format and fixture class
structure and provides advice when to use and when not to use it. Each
example is accompanied by the source code for Java, .NET and Python FIT
implementations, in a form that can be easily copied and used as a
template for similar fixtures.
Version 2.0 brings:
- Python examples for all fixtures and concepts explained in the document.
- Examples for ConstraintFixture, CombinationFixture, CalculateFixture
and notes about SetFixture and SubSetFixture.
Many thanks to Jens Engel and Mike Stockdale for their contributions to
this version.
For more information and download links see
http://gojko.net/fitnesse/fixturegallery
--
gojko adzic
http://gojko.net

"

Here's an analogy I came up with to explain the difference between unit tests and integration tests.

Imagine going to a restaurant with a bunch of your friends. there are some couples there as well as singles. At the end of the meal, it is time to pay. Unit testing is like giving a separate bill to each couple or single. Each person knows only what they needs to pay as well as tip, but do not care what anyone else had, or what the total meal amount is. as long as they pay what they need, they can go home peacefully.

Integration testing is like giving one big check to the group, with everyone having to calculate on their own how much they need to pay. sometimes everyone things they are good, but in the end the total amount may be too high or too low. There is a lot of interaction going on to decide who pays for what and who has already paid.

As part of My Book I have a chapter on the challenges of integrating unit testing practices into an organization. One of the sections deals with "Tough questions and answers". In short, what are the tough questions you will have to deal with when introducing unit testing.

here are the questions I've come across but I'd love to see if there are some that I am missing. Also, I am looking for great answers and references as responses to these questions. Your comments are appreciated:

• 1 How much time will this add to the current process?
• 2 Will my QA job be at risk because of this?
• 3 How do we know this is actually working?
• 4 How much time will this add to the current process?
• 5 Is there proof that unit testing helps?

- any other tough questions?

- Got good answers?

Comments:

I'm using a beta test of Disqus.com's comments INSIDE this post so please try to respond there:

Eli Lopian, Typemock CEO and awesome Coder, just created a nice little API wrapper around Typemock Isolator that would allow a very simple and readable "Swapping" effect between classes. It would allow you to write code like this:

The first test uses the "Swap" class to replace static calls and redirect them to your own class with static methods. The second one is more advanced and will mock all instance calls made on the next instance that will be created of that type. so the last line on the second test will actually call a method against the FakeClass type.

pretty cool and very readable. download the code and binaries from his post.

This NullObject Factory from Paul is very nice, simple and clean in its design. I really like it and I can see many uses for it, in parallel with working with your standard mocking framework or IoC Container Implementation.

here is a little usage example (last line):

var controller = new AccountController(         // Class I am testing
    mockCustomerRepository,                     // Class I am mocking
    NullObject.For<ILogger>());                 // Who cares about logging!

Today was both a good day for e and a bad day for me. Good day because I managed to get a full suite of tests that were somehow failing to work again after a refactoring that got out of hand. Bad - because out of all the tests, one of them failed.

Here's the part I'm ashamed of:

I was working with a pair, and when we both looked at the failing test, I went out and said "Oh, I think this test is no longer valid. It may just need to be removed". We both knew it was still a valid test, that for some reason was failing. But I was on such a role with making everything work, that test was feeling like a thorn in my thumb that I wanted to get rid of.

It was a hard few minutes to convince ourselves to debug through the test and realize what the problem may be

So, denial pattern #1:

Forces: You just want to get on with your code, with only one or two tests that are "ruining the party"

Behavior:

Instead of fixing them debugging them or accepting that you might not have fixed everything, you try to tell yourself that the tests may no longer be valid without actually checking your assumption, which you really wish would be true.

How to avoid: work in pairs. It's harder to ignore a problem (or at least more awkward) with 4 eyes staring at it.

In regards to my post about injecting mocks and stubs using a container in your tests, Dave asks in the comments: Why would you want to do this in the first place?

Here are the things I can think about right now (pros and cons). I'd love to hear your thoughts.

Pros for using a container in your tests:

• Can lead to easier test maintainability. You can add new dependencies to object under test without worrying about existing tests that use that object, since those dependencies will usually be automatically stubbed out as part of the object creation process
• Allows easier focus on testing interaction. When you have a class that has 3+ dependencies, you'd usually want to have no more than one mock against which you'll test the interaction of the class, while the rest would be stubs.  Since everything is getting stubbed by default, all you need to worry about when writing the test is about the mock's expected interaction and perhaps setting one of the stubs to run using a specific behavior.  In any case, writing the boilerplate stubbing code for all the dependencies is spared from you, something which I have struggled with before

Cons of using a container:

• It could make the test a little less readable in terms of understanding what gets injected and what the actual dependencies are. Assuming the reader already knows how to use containers, there is still a very clear separation between setting up the container's mocks and stubs and looks at where they are being used. By design , a container's work is hidden under an abstraction layer, which also makes the test reader not see how the mocks and stubs fit together inside the object under test.

Not sure how I feel about this particular brand of testing syntax. still, it's good that someone thought this problem through - testing xml serialization attributes.

[via the morning brew]

Someone asked for examples of how you'd inject stubs and mocks when you'd like your object under test to use a container such as StructureMap.

here are a couple of simple tests to show this using Typemock Isolator.

the first uses the string based mocks found in the free community version of Typemock Isolator. the second uses the natural style syntax in the enterprise edition:

• assume you have ILogger, IEmailer interfaces and a MessageManager class under test that takes both of these in the ctor.
• The logic under test is: if calling the emailer throws an exception, the logger should be called with the message that was sent

[Test, VerifyMocks]
public void MessageManager_FailsEmailing_LogsMessage_ReflectiveStructureMap()
{
    StructureMapConfiguration.BuildInstancesOf<IEmailer>().AddConcreteType<Emailer>();
    StructureMapConfiguration.BuildInstancesOf<ILogger>().AddConcreteType<Logger>();

    MockObject<IEmailer> emailStub = MockManager.MockObject<IEmailer>();
    emailStub.AlwaysThrow("Send", new Exception("fake exception"));

    MockObject<ILogger> logMock = MockManager.MockObject<ILogger>();
    logMock.ExpectCall("Write").Args("could not email message abc");

    ObjectFactory.InjectStub(emailStub.MockedInstance);
    ObjectFactory.Inject(logMock.MockedInstance);

    MessageManager mm = ObjectFactory.GetInstance<MessageManager>();
    mm.SendMessage("abc");
}

----------------------------------------

[Test, VerifyMocks]
public void MessageManager_FailsEmailing_LogsMessage_NaturalStructureMap()
{
    StructureMapConfiguration.BuildInstancesOf<IEmailer>().AddConcreteType<Emailer>();
    StructureMapConfiguration.BuildInstancesOf<ILogger>().AddConcreteType<Logger>();

    IEmailer emailStub = RecorderManager.CreateMockedObject<IEmailer>();
    ILogger logMock = RecorderManager.CreateMockedObject<ILogger>();

    using (RecordExpectations re = RecorderManager.StartRecording())
    {
        emailStub.Send("", "", "", "");
        re.Throw(new Exception("fake exception")).RepeatAlways();

        logMock.Write("could not email message abc");
    }

    ObjectFactory.InjectStub(emailStub);
    ObjectFactory.Inject(logMock);

    MessageManager mm = ObjectFactory.GetInstance<MessageManager>();
    mm.SendMessage("abc");
}

I'm working on chapter 8 of my upcoming book and I need to come up with a good example of over specification in unit tests, that leads to an unmaintainable test in the long run (fragile test). I have my ideas on how to show it, but I'd love to know what you consider an over specified test, that can be shown with a concise (or even a bit contrived) code example.

- How would you define test over specification?

- how would you show it?

- are there multiple kinds of over specification with different ways to tackle them?