New lenient() strictness setting available at mock and at stubbing level

Problems

Overview of strictness: #769

1. It is not possible to have any common stubbing when using Strictness.STRICT_STUBS. Strict stubs is very useful and it is most likely the future default for Mockito 3. Common stubbing typically is a part of “before” method or a test utility class that generates mocks.

public MockitoRule rule = MockitoJUnit.rule().strictness(Strictness.STRICT_STUBS);

@Before public void before() {
  //a) common stubbing, needed for most test methods
  when(mock.foo()).thenReturn("bar");  
}

1. API caveats:

@Test public void demo() {
  //b) not possible to stub the same method using 'when', with different args:
  when(mock.foo(1)).thenReturn(1);
  when(mock.foo(2)).thenReturn(2); // <- will throw PotentialStubbingProblem

  //c) code under test cannot use stubbed call with different argument (sometimes we need it)
  when(mock.foo(1)).thenReturn(1);
  mock.foo(2); // <- will throw PotentialStubbingProblem whether we need it or not
}

Suggested solution

New public API overview:

@Test public void demo() {
  //1. New method on MockSettings interface:
  Foo mock = Mockito.mock(Foo.class, withSettings().lenient());

  //2. New method on Mockito class:
  Mockito.lenient().when(mock.foo(1)).thenReturn(1);
  Mockito.lenient().doReturn(1).when(mock).foo(1);
}

Details:

• Why 2 new public methods? Sometimes common stubbing has a form of one or few stubbings in “before” method - it’s best to configure leniency per stubbing. Sometimes a mock object has many common stubbings (described in detail in the ticket thread below) - it’s best to configure leniency per mock.

Examples

1. Common stubbing

public MockitoRule rule = MockitoJUnit.rule().strictness(Strictness.STRICT_STUBS);
@Mock(lenient = true) Foo mock;

@Before public void before() {
  //a) common stubbing, needed for most test methods
  when(mock.foo()).thenReturn("bar");  
}

1. API caveats:

@Test public void demo() {
  //b) not possible to stub the same method using 'when', with different args:
  lenient().when(mock.foo(1)).thenReturn(1);
  when(mock.foo(2)).thenReturn(2); // <- works!

  //c) code under test cannot use stubbed call with different argument (sometimes we need it)
  lenient().when(mock.foo(1)).thenReturn(1);
  mock.foo(2); // <- works!
}

Original report

The new UnnecessaryStubbingException logic is great. However, it is sometimes useful to disable this detection for specific methods. Could it be possible to add an annotation that says “unnecessary stubs should not be detected in this method”?

This would also make it easier to migrate from Mockito 1.* to Mockito 2. This is the case in the project I am currently working at, where we have created some utility methods that creates a mock and configures it to fit most of our use cases. As we use this method at many places, where different configuration is needed, it will cause a lot of unnecessary stubbing. Thus, we would like to keep this method out of unneccessary stubbing-check, while doing this check on the rest of the code base.

[digulla]

@mockitoguy Sorry, that doesn’t help at all.

With ~200K on StackOverflow, that was the most simple solution I could come up with. You’re welcome to show me a better solution.

The testing strategy is sound. It doesn’t break easily or unexpectedly and is easy to understand for new members of the team. I could copy only the necessary lines to new tests but that would mean I would be the only one who can write new tests. Also: Violates DRY.

Refactoring is not possible. ZK is a UI framework with a huge code base and many projects use it. Asking to change the API is like asking to “fix” the Java Collections API: Understandable, maybe even reasonable but unrealistic.

Hand mocks would mean I would have to write about a lot of useless code which violates your own rule to keep tests simple.

So I’m between a rock and a hard place: You’re right for open source projects which no one uses or green field commercial projects. For existing commercial projects that I can’t move, it’s not helpful. They have ugly and unmodifiable APIs, so I have to move the only place where I have influence: That’s the tests.

So for now, I have to disable a good feature of Mockito and can no longer make sure that my tests stay clean. That really sucks.

[mockitoguy]

I’m finalizing the implementation in #1272, it will be released within days.