New Mockito stubbing api

Top Mockito priorities (equal importance):

I. Maximize test’s clarity II. Maximize test’s failure output (easy test failure debugging) III. Guide the engineer toward cleaner, higher quality tests that are easy to maintain

Use cases, in the order of importance.

1. Unused stubbing is dead code, an irrelevant detail that clutters test’s clarity.
2. Unintentionally unused stubbing can make test failures harder to debug.
3. When the stubbed call needs to be verified, it is declared twice in test (violation of DRY principle).

Proposed solution

The solution consists of multiple changes. Perhaps it’s better to show the code first, then describe the changes.

//JUnit rule automatically fails when stubs are unused:
MockitoJUnit.rule().strict();

//Offer a way to make specific stubbings optional:
given(mock.foo(), optional()).willReturn("bar");

//Offer a way to validate stubs for users who don't use JUnit rules:
Mockito.validateStubs();

Summary:

1. All stubbed method calls must be used (realized), otherwise the test fails.
2. Test failure is triggered by JUnit rule or Runner

• When JUnit rule or runner is not used, new Mockito.verifyStubs() method can be used. The method is not intended for engineers to use in tests. It’s intended for framework integrators (TestNG runners) or a common base class.

1. “optional” stubbings. Consider default stubbings that user may put in test constructor or setup method. Those default stubbings are used by most test methods but not all. This is a legit scenario that needs to be supported.
2. verifyNoMoreInteractions() no longer includes used stubs for validation when strict rule / runner is in use.

Mockito 3:

1. Strict rule by default
2. verifyNoMoreInteractions() no longer includes used stubs for validation

[ryan-gustafson]

It’s unfortunate this isn’t slated for 2.0 anymore, these are the kinds of features I’m most interested in seeing, as they help keep the tests accurate and complete.

In most cases, I’d like my mocks to:

• Not have magical default behavior, they should only do what I stub them to do, and nothing else. Failure to use a stubbing should fail fast, not lazily. It would be great if the failure could indicate the “closest” stubbings to the actual call made, to help diagnose the misalignment.
• If I stubbed it, I usually intend it to be called, and therefore want it to be verified without repeating myself.

The lack of these kinds of features increases the time-cost of using Mockito because:

• After repeated refactorings by different people of varying abilities, the test stubbing can have almost nothing at all to do with the actual code. Time is wasted trying to understand what is really being tested, and reconciling the test setup against the actual code.
• Verifying interactions is highly repetitious to the point of being onerous. It’s really the only way to ensure the mock is being used in the expected fashion, so it should be far less effort.
• Default answers of null/zero/empty are fine if you care very little about the mock interactions, they are completely wrong when you do care. If I only want the mock to have behaviors defined strictly by what was stubbed, I should get an exception at method call time, not a bogus return result that in the worst case could result in an indirect error very far from the call on the mock but before you make it to the verify() call. Debugging these kinds of problems can be a huge time sink, only to discover you missed a stubbing or it was slightly off.

[ryan-gustafson]

@szczepiq I’ve experimented with changing the default answer in Mockito 1.This interacts poorly with when/then style stubbing though. Workarounds are to (a) use do/when style, which loses compile time type safety on the returns, (b) temporarily disable the default answer behavior during setup, but this is a pain to manage and make sure you re-enable when done, or © wrap Mockito stubbing APIs to apply (b) automatically during the ongoing stubbing call.

This basically boils down to changing the answer behavior during stubbing to be different than a nonstubbing mock interaction.

I see Erik added some great comments whole I was writing this.

A separate point… I don’t think standard Mockito behavior needs to change, as there are legitimate uses for wanting mocks with null defaults. It would also be a huge shift for the user base.

That’s said, I personally find the most arguments for using loose mocks in tests so they are not over specified to be logically unsound. If you don’t know what was done with the mock, and you haven’t mocked a behavior, then usually follows that the test is of the feel-good variety and not very good. Ultimately you can’t really say what the code did other than it didn’t blow up, produced a particular result in an unknown context, or had a few interactions on a mock. It’s usually not possible to reason that there were no unexpected interactions that invalidate the test, or that the handwaving used when the test was first written still holds in the future. For these reasons I don’t use Mockito like many apparently do, I want my mocks absolutely specified and fully verified whenever possible. I blame my mathematics background for this desire for rigorousness.

When I don’t or can’t then I used Mockito like most people, but I make it clear in the test the intended bounds of this lack of rigorousness. Such tests are usually rather shallow and only exist to provide loose constraints on the behavior of the code. I usually think through such cases multiple times to ensure this is truly the proper strategy to use, as the test is inherently ill-defined and potentially invalid now or in the future, even it if it keeps on passing the entire time.