Strictness in Mockito

Introduction

This issue explores the addition of “strict” behavior and APIs to Mockito. Traditionally, Mockito is a lenient mocking framework. For background and motivation, check out Szczepan’s short article on LinkedIn or in-depth article on Mockito blog.

Why strictness in Mockito:

• productivity - improved debugging and writing tests. Currently MockitoHint warns about stubbing argument mismatches. Strict stubbing would promote the warning into an exception, making the test fail fast and improve debuggability / productivity.
• clean code - avoid unnecessary stubbings, test fails when unncessary stubs are present.

Concrete API proposals:

• “strict stubbing” proposal: #770
• “strict mocking” proposal: #1097
• opt-out from strict stubbing at mock / method level: #792

Example exception message indicating subbing problem:

org.mockito.exceptions.misusing.PotentialStubbingProblem:
Strict stubbing argument mismatch. Please check:
- this invocation of 'simpleMethod' method:
    mock.simpleMethod(“Foo");
      -> at … StrictStubbingEndToEndTest.java:101
- has following stubbing(s) with different arguments:
    1. mock.simpleMethod(“foo");
      -> at … StrictStubbingEndToEndTest.java:100
Typically, stubbing argument mismatch indicates user mistake when writing tests.
Mockito fails early so that you can debug potential problem easily.

Details

Strictness in Mockito can have 2 forms:

• Strict stubbing that requires that all declared stubs are actually used
• Strict mocks, that require all non-void methods to be stubbed before they are called

Strictness behavior can be implemented as:

• warning on the console
• exception thrown

Strictness API can be implemented in:

• JUnit runner / rule
• TestNG runner
• explicit method to call by the user

Future direction:

• strict stubbing on by default, opt-out available
• strict mocking off by default, opt-in available

Implementation as of Mockito 2.2.x:

• JUnitRunner - fails at the end of test class when unused stubbings are present. Opt out available via JUnitRunner.Silent
• JUnitRule - warn at the end of test method about a) unused stubs b) stubs invoked with different arguments. Opt-out available via rule().silent().

Mockito 2.x proposal:

1. New “strict stubbing” API for JUnit rule - #770
2. New Answer.StrictMock that fails immediately when non-void method invoked on without prior stubbing - #649
3. new verifyStubs() method to use when rule/runner not in used

Mockito 3 proposal:

• strict stubbing is the default
• change behavior of verifyNoMoreInvocations()
• runner and rule behave the same

[Druckles]

While writing an integration test I came across a specific example where this Strictness may actually be useful. In this example, I want to explicitly mock a call to a service and ensure it is definitely called. I have, for example, the following method:

public void unzipRemotely(String location, String filename) {
    File original = service.download(location, filename);
    ZipFile zip = new ZipFile(original);
    File tempDirectory = createTempDirectory();
    zip.unzip(tempDirectory);
    service.uploadDirectory(location, removeExtension(filename), tempDirectory, true);
}

The two things I need to test here are:

• Were the files unzipped?
• Was uploadDirectory called?

I don’t extrinsically care whether download was called; it’s implied by the other two facts.

To test this, I need something like the following:

@Test
@DisplayName("")
void success() {
    when(service.download("somewhere", "file.zip")).thenReturn(zippedFile);
    when(service.uploadDirectory(eq("somewhere"), eq("file"), any(File.class), eq(true)))
        .thenAnswer(invocationOnMock -> {
            File sourceDirectory = invocationOnMock.getArgument(2);
            assertThatAllFilesAreThere(sourceDirectory);
        });

    unzipper.unzipRemotely("somewhere", "file.zip");
}

The problem here is that there is no guarantee that uploadDirectory was ever called. There are three existing solutions to this:

• Assert that some flag is set
• Rely on Strict mode
• verify

Assert some flag is set

AtomicBoolean wasCalled = new AtomicBoolean(false);
when(service. ...)
    .thenAnswer(invocationOnMock -> {
        // ...
        wasCalled.set(true);
    });
assertThat(wasCalled, is(true));

Obvious problem with this: it requires more code. It’s also not as succinct for describing what is happening.

Rely on Strict mode

This is the current solution. You called when. You should expect it to be called.

Possible issues:

• Anyone can come along and add LENIENT over the tests to expand them. It’s no longer verified.
• It’s not explicit from the name. “When” is pretty close to “if”. In fact the Germans don’t even distinguish between the two. In English, “when” means both, “when this thing definitely occurs in the future” and also more generally “whenever this happens”. If you say, “when trees fall they make a noise”, it doesn’t mean a tree in that forest over there will fall.

Verify

The most natural of the options, I want to explicitly verify that something happened. But for that, it need the following line:

verify(service).uploadDirectory(eq("somewhere"), eq("file"), any(File.class), eq(true));

Problem: duplicated code. Pretty obvious that that’s not ideal.

An alternative solution

What about instead of changing the API (which is what Strict-mode does; it means you have to change all your tests in some way), the API is expanded with an extra command:

expect(service.uploadDirectory(eq("somewhere"), eq("file"), any(File.class), eq(true)))
    .andAnswer(...);

• Works just like when but must be called at least once
• Like verify it accepts a VerificationMode like times
• Explicitly describes what it does
• when no longer verifies that the stubbed method was called

This would solve all the above issues. Non-mocked method calls could still throw an exception for missing stubbings (which does indicate a problem).

[imod]

I think from a code maintenance point of view, “behavior: warning on the console” is not really useful - in such a case a CI system would not inform anyone about unused stubs. An exception would be a lot better.