Strict Stubbing and Overloaded Methods

Hello,

I’ve been trying to reach out about an issue that I’m facing with overloaded methods in certain class in #769. The problem appeared when I was applying obfuscation on source code and running tests on them since in obfuscation aggressive overloading is used. However the problem also occurs without obfuscation. If two overloaded methods within the same class are being stubbed with strict stubbing rule applied, PotentialStubbingProblem occurs.

Here are pieces of code that can generate this problem. I’m using org.mockito:mockito-android:2.16.0.

The test:

public class MockitoOverloadInstrumentationTest {

    @Mock
    private MyInterface myInterface;

    @Rule
    public MockitoRule mockitoRule = MockitoJUnit.rule().strictness(Strictness.STRICT_STUBS);

    @Before
    public void initMocks() {
        MockitoAnnotations.initMocks(this);
    }

    @Test
    public void testOverload() {
        // Set Up
        Mockito.when(myInterface                                // <-- Exception happening here
                .myMethod(Mockito.anyString(), Mockito.anyString()))
                .thenReturn("Mocked Hello World!");
        Example example = new Example(myInterface);

        // Test
        example.start();
        Mockito.verify(myInterface, Mockito.timeout(10000L))
                .myMethod(Mockito.anyString(), Mockito.anyString());
    }
}

Interface to be mocked:

public interface MyInterface {

    boolean myMethod();

    String myMethod(String i, String j);
}

and finally the example class:

public class Example {

    private static final String TAG = Example.class.getSimpleName();

    private final MyInterface myInterface;

    public Example(MyInterface myInterface) {
        this.myInterface = myInterface;
    }

    public void start() {
        if (!myInterface.myMethod()) {     // <-- Exception pointing to here as well.
            Log.i(TAG, "start: " + myInterface.myMethod("hello", "world"));
        }
    }

}

The exception that I am getting is the following:

org.mockito.exceptions.misusing.PotentialStubbingProblem: 
Strict stubbing argument mismatch. Please check:
 - this invocation of 'myMethod' method:
    myInterface.myMethod();
    -> at test.ahasbini.com.test.Example.start(Example.java:20)
 - has following stubbing(s) with different arguments:
    1. myInterface.myMethod("", "");
      -> at test.ahasbini.com.test.MockitoOverloadInstrumentationTest.testOverload(MockitoOverloadInstrumentationTest.java:33)
Typically, stubbing argument mismatch indicates user mistake when writing tests.
Mockito fails early so that you can debug potential problem easily.
However, there are legit scenarios when this exception generates false negative signal:
  - stubbing the same method multiple times using 'given().will()' or 'when().then()' API
    Please use 'will().given()' or 'doReturn().when()' API for stubbing.
  - stubbed method is intentionally invoked with different arguments by code under test
    Please use default or 'silent' JUnit Rule (equivalent of Strictness.LENIENT).
For more information see javadoc for PotentialStubbingProblem class.
    at test.ahasbini.com.test.Example.start(Example.java:20)
    at test.ahasbini.com.test.MockitoOverloadInstrumentationTest.testOverload(MockitoOverloadInstrumentationTest.java:37)
    at java.lang.reflect.Method.invoke(Native Method)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at android.support.test.internal.runner.junit4.statement.RunBefores.evaluate(RunBefores.java:80)
    at org.mockito.internal.junit.JUnitRule$1.evaluateSafely(JUnitRule.java:52)
    at org.mockito.internal.junit.JUnitRule$1.evaluate(JUnitRule.java:43)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at test.ahasbini.com.test.PermissionRule$1.evaluate(PermissionRule.java:156)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
    at android.support.test.internal.runner.TestExecutor.execute(TestExecutor.java:58)
    at android.support.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:375)
    at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1932)

I’ve traced the issue to the following lines in https://github.com/mockito/mockito/blob/71f265568f24e2d690b0212ec5c177b7dd1b4c2b/src/main/java/org/mockito/internal/junit/DefaultStubbingLookupListener.java#L53-L62 Where the if condition only checks if the names are the same and then reports the problem. Although the names are same, the methods have different parameters signature. If we want to mock these methods for our tests to run properly, the check should consider parameter signature as well.

[nayeemd]

As mentioned on the issue 1496:

It looks like this has to do with calling when the second time. I couldn’t find what was going on, but the second when doesn’t attempt to add the mock logic, it just calls the method. Replacing with doReturn().when() works.

doReturn(expected1).when(overloadedMethods).getString(any());
doReturn(expected2).when(overloadedMethods).getString(any(), any());

https://github.com/mockito/mockito/issues/1496#issuecomment-423310950

[TimvdLippe]

My personal opinion is that we should not, as that is quite some “magic” behavior. Personally I would prefer to keep our “strict” as pure “strict” without several special corner-cases. I do acknowledge that this requires a little bit more work by our users, but I think that makes the code more understandable than if we would not enforce this rule. However, let’s see what @mockitoguy thinks about this.