verifyNoMoreInteractions behaviour change in Junit5 integration

We found that the behaviour of verifyNoMoreInteractions changed in the Junit5 integration. In Junit4, verifyNoMoreInteractions would take interactions with stubbed methods into account, which means that it will fail a test if there are interactions that are not verified. Since Junit5, it only counts interactions from methods that are not stubbed, which means it passes a test when an interaction is not verified.

As we use identical Mockito versions, we would expect the behaviour to be unchanged.

Example Junit4 project (fails the test) pom.xml:

<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
		 xmlns="http://maven.apache.org/POM/4.0.0"
		 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>test</groupId>
	<artifactId>test</artifactId>
	<version>1.0-SNAPSHOT</version>

	<dependencies>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>4.12</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.mockito</groupId>
			<artifactId>mockito-core</artifactId>
			<version>2.24.0</version>
			<scope>test</scope>
		</dependency>
	</dependencies>
</project>

Test case:

import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.verifyNoMoreInteractions;
import static org.mockito.Mockito.when;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mock;
import org.mockito.junit.MockitoJUnitRunner;

import java.util.List;

@RunWith(MockitoJUnitRunner.class)
public class Example {
  @Mock
  private List<String> list;

  @Test
  public void testNoMoreInteractions() {
    when(list.size()).thenReturn(1);
    assertEquals(1, list.size());
    verifyNoMoreInteractions(list);
  }
}

Example Junit5 project (passes the test) pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
		 xmlns="http://maven.apache.org/POM/4.0.0"
		 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>test</groupId>
	<artifactId>test2</artifactId>
	<version>1.0-SNAPSHOT</version>

	<dependencies>
		<dependency>
			<groupId>org.junit.jupiter</groupId>
			<artifactId>junit-jupiter-api</artifactId>
			<version>5.3.2</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.mockito</groupId>
			<artifactId>mockito-core</artifactId>
			<version>2.24.0</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.mockito</groupId>
			<artifactId>mockito-junit-jupiter</artifactId>
			<version>2.24.0</version>
			<scope>test</scope>
		</dependency>
	</dependencies>
</project>

Test case:

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.mockito.Mockito.verifyNoMoreInteractions;
import static org.mockito.Mockito.when;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;

import java.util.List;

@ExtendWith(MockitoExtension.class)
public class Example {
  @Mock
  private List<String> list;

  @Test
  public void testNoMoreInteractions() {
    when(list.size()).thenReturn(1);
    assertEquals(1, list.size());
    verifyNoMoreInteractions(list);
  }
}

[bartsopers]

Thanks to @krc1504’s comment, I found that this is actually not really a bug as it’s documented behavior. The MockitoExtension uses a different default Strictness then the MockitoJUnitRunner (STRICT_STUBS vs WARN). From the javadoc:

If you use Mockito.verifyNoMoreInteractions(Object…) you no longer need to explicitly verify stubbed invocations. They are automatically verified for you.

So another “workaround” would be to annotate test classes with:

@MockitoSettings(strictness = Strictness.WARN)

However, I personally think that automatically verifying stubs has little to do with strictness on a conceptual level. If one would argue that they are related, I think it doesn’t make sense that the most strict level would automatically verify stubs, it would make much more sense if that would happen with (one of) the less strict levels.

[Maarten-Damen]

@bric3 Is there any progress on this issue? We encounter testcases that succeed with JUnit 5, but fail with JUnit 4, which is a problem for us.