Duplicated smoke tests

Credit

Credit goes to @eyalkoren for the discovery of this issue

Overview

It appears that certain tests are duplicated across sets of smoke tests. For example, take the greetingShouldReturnDefaultMessage test. This test is located in the integrations/test directory, and can be seen here.

If one examines a test run by opening a run and going to the Blue Ocean page, they can see that the same test appears to have run as a part of both the Smoke Tests 01 group and the Smoke Tests 02 group.

Details

Tests are segmented through the use of simple bash scripts. The scripts which provide this segmentation are below:

Smoke Tests 01 Smoke Tests 02(https://github.com/elastic/apm-agent-java/blob/master/scripts/jenkins/smoketests-02.sh)

Each script contains a preparatory step to generate the modules to be tested. Below is the output of each:

smoketest01

find apm-agent-plugins -maxdepth 1 -mindepth 1 -type d|grep -v "target"

apm-agent-plugins/apm-mongoclient-plugin
apm-agent-plugins/apm-redis-plugin
apm-agent-plugins/apm-spring-webmvc-plugin
apm-agent-plugins/apm-quartz-job-plugin
apm-agent-plugins/apm-jaxrs-plugin
apm-agent-plugins/apm-jsf-plugin
apm-agent-plugins/apm-httpclient-core
apm-agent-plugins/apm-kafka-plugin
apm-agent-plugins/apm-java-concurrent-plugin
apm-agent-plugins/apm-jdbc-plugin
apm-agent-plugins/apm-jmx-plugin
apm-agent-plugins/apm-servlet-plugin
apm-agent-plugins/apm-es-restclient-plugin
apm-agent-plugins/apm-opentracing-plugin
apm-agent-plugins/apm-scheduled-annotation-plugin
apm-agent-plugins/apm-mule4-plugin
apm-agent-plugins/apm-jaxws-plugin
apm-agent-plugins/apm-api-plugin
apm-agent-plugins/apm-jms-plugin
apm-agent-plugins/apm-slf4j-plugin
apm-agent-plugins/apm-process-plugin
apm-agent-plugins/apm-urlconnection-plugin
apm-agent-plugins/apm-okhttp-plugin
apm-agent-plugins/apm-apache-httpclient-plugin
apm-agent-plugins/apm-asynchttpclient-plugin
apm-agent-plugins/apm-error-logging-plugin
apm-agent-plugins/apm-hibernate-search-plugin
apm-agent-plugins/apm-spring-resttemplate-plugin

smoketest02

find integration-tests -maxdepth 1 -mindepth 1 -type d|grep -v "target"|

integration-tests/soap-test
integration-tests/cdi-app
integration-tests/spring-boot-2
integration-tests/application-server-integration-tests
integration-tests/spring-boot-1-5
integration-tests/jsf-app
integration-tests/simple-webapp

Reasoning

It appears as though the preparatory step correctly separates the two sets of tests but that Maven may be running the same test(s) in each instance. Maven is executed with the following parameters:

./mvnw -q -Dmaven.javadoc.skip=true -am -amd -pl ${MOD} -P integration-test-only verify (where $MOD is the set of modules from each prepatory step.)

Could, perhaps, the -am or the -amd flag inadvertently be triggering a larger set of tests? Or, is something in one each tests linking to the other set through a dependency of some kind?

[cachedout]

Let us know what we should do.

I think that broadly speaking, the steps are going to be as follows. (I’ve adopted these from poking around a bit in the hbase repo, which seems to have done this pretty well.)

1. Find a place to store test annotations. Here are some examples from hbase.

2. Create these annotations, corresponding to the original groups or new groups, if you like.

3. Apply annotations to test classes using @Category. ex with pseudo-code:

import org.junit.experimental.categories.Category;
import my.path.to.annotations.SmokeTests01

@Category(SmokeTests01.class)
public class MongoClientSyncInstrumentationIT extends AbstractMongoClientInstrumentationTest {

<snip>

1. Add groups to the <configuration> block of the Surefire plugin. Here is an arbitrary example of the needed elements:

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-surefire-plugin</artifactId>
	<configuration>
                		<excludedGroups>my.path.to.annotations.SmokeTest01</excludedGroups>
	</configuration>
</plugin>

1. At this point, you should be able to run tests with the -Dgroups=my.path.to.annotations.SmokeTest01 flag with Maven.

We may need to adjust course depending on what we find, but hopefully this should give you a clearer idea of how I think this should probably be done.

[cachedout]

Oops. I didn’t mean to close this one entirely.