Strict mocks support

Introduction

Supporting “strict” mocks will help with test debuggability in certain scenarios.

This design note explains the rationale and suggested implementation plan to support “strict” mocks. Please contribute! If you want to help our community and implement this feature, please read the design spec carefully and let us know if something is unclear. This feature has been asked for many years now and we think that now we are ready to nail it. We already started to make Mockito stricter (#769).

We did our best to create this design spec in a way anyone can get started on implementing this feature in no time. Read it, code it and give back a pull request 😃 If something is unclear, you have feedback about the design or a better idea how to solve the use case - leave us a comment.

Rationale

Mockito mocks are loose by default - methods can be freely invoked on a mock, non-void methods will return default empty value. “Strict” mock works differently, you are not allowed to invoke a method on a strict mock unless this method was previously “configured”. This is how traditional mocking frameworks like JMock or EasyMock work by default. Strict mocks offer better debuggability in certain scenarios, however overusing strict mocks leads to overspecified, brittle tests. Strict VS loose mock explanation is further described in an article on Mockito blog.

Mockito mocks will remain loose by default but we want to offer better support for strict mocks.

Useful links:

• previous design and discussion in #649
• initial discussion in #303

Implementation

New behavior

Strict mocking can be enabled per test method execution and introduces following behavior:

• unstubbed, non-void method fails immediately with “I am unstubbed!” exception
• such method is implicitly marked as verified for the verifyNoMoreInteractions() calls
• unused stubs are detected at the end of test and trigger failure
• requires to use doReturn API for stubbing mocks (limitation of the syntax, already known gotcha when using test “spy” or strict stubs)

Options and questions

• Q: Do we separate behavior for voids / non-voids? A: Mockito API makes clear distinction between stubbing (typically non-void methods) and verification (typically void methods). It is a design choice to drive clarity and understanding of ASK vs TELL style interactions (this design spec does not explain this topic further). Unless we hear more feedback, we want the strict mock to only fail early on non-void methods.
• Q: Do we offer Strictness level configuration per mock object? A: Is it needed? So far it was not reported by users that it is needed so we will ignore this use case.

New API

• new class: MockitoJUnitRunner.StrictMocks, similar to MockitoJUnitRunner.StrictStubs
• new enum type: Strictness.STRICT_MOCKS, similar to Strictness.STRICT_STUBS

New API side by side with existing API

MockitoJUnitRunner.Silent MockitoJUnitRunner.Strict MockitoJUnitRunner.StrictStubs MockitoJUnitRunner.StrictMocks

MockitoRule.silent() MockitoRule.strictness(Strictness)

Strictness.LENIENT Strictness.WARN Strictness.STRICT_STUBS Strictness.STRICT_MOCKS

Getting started

Suggested starting point for the implementation:

• unit test, see test that covers similar functionality - StrictStubbingTest.java. New test needs to demonstrate that exception is thrown when we invoke non-void, unstubbed method on a mock.
• to implement the feature, I think we need to tweak a little bit DefaultStubbingLookupListener.

Test coverage

Suggested test coverage:

• demonstrate the feature works with:
  • MockitoSession API
  • MockitoJUnitRunner
  • MockitoRule
• feature coverage
  • void and non-void methods
  • configure strict mocks for single test method only, all other test methods can have different Strictness
  • detects unused stubs
  • implicitly verified for verifyNoMoreInteractions()
  • documented the need for using doReturn API
    • in Javadoc
    • strict mock verification exception (~“I am unstubbed!”) contains hint
  • if method was stubbed but with different args, the verification exception message should indicate it

[mockitoguy]

FYI: I’m making progress on the implementation. Stay tuned 😃

[ivan-mladenovic]

I completely agree with @imod. That is only reason I am avoiding to use Mockito. Adding strict mode will make debugging easier, for example if test have to be updated after some implementation changes.