Whitelists are not loaded correctly in driver based tests

The problem

After an upgrade to Corda 4.3 from 4.0, my driver based tests started to fail because the nodes were unable to find classes to be serialized on the whitelist. However, they are present on a whitelist published as a META-INF.service in the CorDapp.

An investigation into the problem revealed that, when running a node via the driver based test framework, all CorDapps are loaded with an empty list of whitelists. The problem is in the function findWhitelists()in the file JarScanningCordappLoader.kt. It extends the classpath with the jar of the CorDapp that has been deployed on the node and then proceeds to create a list of instances of all classes implementing SerializationWhitelist that can be found on the extended classpath, filtering out the instances whose class does not come from the deployed CorDapp jar.

The code seems to hinge on the assumption that all SerializationWhitelist services on the classpath will actually be loaded, but this is not true. The documentation of ServiceLoader includes the following paragraph:

If a particular concrete provider class is named in more than one configuration file, or is named in the same configuration file more than once, then the duplicates are ignored. The configuration file naming a particular provider need not be in the same jar file or other distribution unit as the provider itself. The provider must be accessible from the same class loader that was initially queried to locate the configuration file; note that this is not necessarily the class loader from which the file was actually loaded.

Since the driver framework starts nodes with a classpath that includes the build paths of the project, and because parent classloaders are visited first, ServiceLoader will instantiate the whitelists from the wrong location, and the instances in the deployed jar will never be created because they have the same package and class names.

This happens regardless of the value of startNodesInProcess in the driver parameters.

To reproduce in Corda 4.3

The bug is reproduced here: https://github.com/ulrikrasmussen/reproduce-whitelist-bug

Run the driver based test and observe that it fails. However, the whitelist class is actually instantiated, and I have inserted logging code in the constructor which tells you the class location of the instantiated whitelist and the stack trace. This is printed to stdout on the nodes:

[INFO] 12:39:35,356 [main] cordapp.JarScanningCordappLoader. - Looking for CorDapps in [/home/ulrik/deon/dev/corda-whitelist-bug/cordapp-template-kotlin/workflows/build/node-driver/20200131-113853.736/BankA/cordapps]
[INFO] 12:39:35,376 [main] cordapp.JarScanningCordappLoader. - Loading CorDapps from RestrictedURL(url=file:/home/ulrik/deon/dev/corda-whitelist-bug/cordapp-template-kotlin/workflows/build/node-driver/20200131-113853.736/BankA/cordapps/workflows-0.1.jar, rootPackageName=null), RestrictedURL(url=file:/home/ulrik/deon/dev/corda-whitelist-bug/cordapp-template-kotlin/workflows/build/no
de-driver/20200131-113853.736/BankA/cordapps/contracts-0.1.jar, rootPackageName=null)
[INFO] 12:39:35,411 [main] cordapp.JarScanningCordappLoader. - Scanning CorDapp in file:/home/ulrik/deon/dev/corda-whitelist-bug/cordapp-template-kotlin/workflows/build/node-driver/20200131-113853.736/BankA/cordapps/workflows-0.1.jar
[INFO] 12:39:35,556 [main] flows.MyWhitelist. - Whitelist instantiated.
        location=file:/home/ulrik/deon/dev/corda-whitelist-bug/cordapp-template-kotlin/workflows/build/classes/kotlin/main/
        stacktrace:
                java.lang.Thread.getStackTrace(Thread.java:1559)
                com.template.flows.MyWhitelist.<init>(MyWhitelist.kt:13)
                sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                java.lang.Class.newInstance(Class.java:442)
                java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
                java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
                java.util.ServiceLoader$1.next(ServiceLoader.java:480)
                kotlin.collections.CollectionsKt___CollectionsKt.toCollection(_Collections.kt:1062)
                kotlin.collections.CollectionsKt___CollectionsKt.toMutableList(_Collections.kt:1095)
                kotlin.collections.CollectionsKt___CollectionsKt.toList(_Collections.kt:1086)
                net.corda.node.internal.cordapp.JarScanningCordappLoader.findWhitelists(JarScanningCordappLoader.kt:272)
                net.corda.node.internal.cordapp.JarScanningCordappLoader.toCordapp(JarScanningCordappLoader.kt:147)
                net.corda.node.internal.cordapp.JarScanningCordappLoader.loadCordapps(JarScanningCordappLoader.kt:106)
                [... snip ...]

Note that the location of the instantiated class is from the build output, and not the CorDapp jar. Only this instance is created.

Further digging: Why did it work in Corda 4.0?

AMQPSerializationScheme used to also invoke the ServiceLoader to find whitelists, which would skip verification of the class location and load the whitelist from the build directory anyway. This has probably masked the issue, and is why the problem did not occur in Corda 4.0. The behavior was removed in #5112, although the description in this PR does not indicate that this change of behavior was intentional.

Is it fixed in Corda 4.4?

Only partially. Tested at version 1380779a9c.

The easiest way to test this was to apply the patch in the bottom of this issue to the simm-valuation-demo and observe the location of the instantiated whitelist in stdout when running the driver based test. I ran the driver based test with startNodesInProcess equal to true and false, respectively.:

• startNodesInProcess = false: The loaded whitelist is the correct one from the deployed jar, so this case seems to have been fixed.
• startNodesInProcess = true: The incorrect whitelist from the build output is loaded. Also, custom serializers cannot be found, so the test fails for other reasons before it can get to the whitelist issue. This is probably a related bug that also has to be fixed.

Debugging Patch:

diff --git a/samples/simm-valuation-demo/contracts-states/src/main/kotlin/net/corda/vega/plugin/SimmContractsPluginRegistry.kt b/samples/simm-valuation-demo/contracts-states/src/main/kotlin/net/corda/vega/plugin/SimmContractsPluginRegistry.kt
index 1e6abdfd0f..e4f7c2fd72 100644
--- a/samples/simm-valuation-demo/contracts-states/src/main/kotlin/net/corda/vega/plugin/SimmContractsPluginRegistry.kt
+++ b/samples/simm-valuation-demo/contracts-states/src/main/kotlin/net/corda/vega/plugin/SimmContractsPluginRegistry.kt
@@ -11,10 +11,13 @@ import com.opengamma.strata.market.param.CurrencyParameterSensitivities
 import com.opengamma.strata.market.param.CurrencyParameterSensitivity
 import com.opengamma.strata.market.param.TenorDateParameterMetadata
 import com.opengamma.strata.market.param.ParameterMetadata
+import net.corda.core.internal.location
 import net.corda.core.serialization.SerializationWhitelist
 import net.corda.vega.analytics.CordaMarketData
 import net.corda.vega.analytics.InitialMarginTriple
+import org.slf4j.LoggerFactory
 
+private val log = LoggerFactory.getLogger(SimmContractsPluginRegistry::class.java)
 /**
  * [SimmService] is the object that makes available the flows and services for the Simm agreement / evaluation flow.
  * It is loaded via discovery - see [SerializationWhitelist].
@@ -22,6 +25,11 @@ import net.corda.vega.analytics.InitialMarginTriple
  * It is loaded via discovery see [WebServerPluginRegistry].
  */
 class SimmContractsPluginRegistry : SerializationWhitelist {
+    init {
+        val cp = SimmContractsPluginRegistry::class.java.location
+        log.info("Whitelist instantiated.\n\tlocation=$cp\n\tstacktrace:\n\t\t${Thread.currentThread().stackTrace.joinToString("\n\t\t")}")
+    }
+
     override val whitelist = listOf(
             MultiCurrencyAmount::class.java,
             Ordering.natural<Comparable<Any>>().javaClass,

[rick-r3]

Filed as https://r3-cev.atlassian.net/browse/CORDA-3566.

Thanks for the excellent investigation / analysis.

[chrisr3]

Hi @ulrikrasmussen, I think I fixed this in CORDA-3643, which should be included in Corda 4.5 😃 .