Mocking via subclassing Kotlin coroutine Channel interface produces sometimes missing method `cancel()`

Hi,

have a problem described in a title.

I narrowed it down. It is happening in race condition.

Compare following GIST: https://gist.github.com/oleksiyp/bce110683d98b7bd66d95934b70af88e ByteBuddy for some reason skips method cancel(), but not always

Then I tried to make it reproducible in console. It was not very easy task. Long classpath somehow affects it. It is not even presence of many libraries, but long classpath.

I am not sure if below code will reproduce the problem for you. Most of the libs here are not needed to run. Needed: ByteBuddy, kotlinx-coroutine-core and Objenesis.

Following output means reproduction is successful:

Running 20x times
01100010001100100000
Successes - 6

Reproduction bash script:

#!/bin/bash

JAVA_HOME=/usr/lib/jvm/java-8-oracle

function d {
    URL="$1"
    LOC="jars$2"
    mkdir -p `dirname $LOC`
#    echo `basename $LOC`
    wget -nc $URL -O $LOC 2> /dev/null
}

d http://central.maven.org/maven2/com/squareup/okio/okio/1.15.0/okio-1.15.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/com.squareup.okhttp3/okhttp/3.12.0/b36f4a04584c0fb0d9af2d3401cdff8dacb1ea54/okhttp-3.12.0.jar
d http://central.maven.org/maven2/com/squareup/okhttp3/okhttp/3.12.0/okhttp-3.12.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/com.squareup.okio/okio/1.15.0/bc28b5a964c8f5721eb58ee3f3c47a9bcbf4f4d8/okio-1.15.0.jar
d http://central.maven.org/maven2/io/netty/netty-buffer/4.1.30.Final/netty-buffer-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.30.Final/597adb653306470fb3ec1af3c0f3f30a37b1310a/netty-buffer-4.1.30.Final.jar
d http://central.maven.org/maven2/io/netty/netty-codec-dns/4.1.30.Final/netty-codec-dns-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.30.Final/515c8f609aaca28a94f984d89a9667dd3359c1b1/netty-codec-4.1.30.Final.jar
d http://central.maven.org/maven2/io/netty/netty-codec/4.1.30.Final/netty-codec-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.30.Final/7d28ce324f6cd5ae4ddd7f3e5027e2a7f126740b/netty-codec-dns-4.1.30.Final.jar
d http://central.maven.org/maven2/io/netty/netty-codec-http2/4.1.30.Final/netty-codec-http2-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.30.Final/2da92f518409904954d3e8dcc42eb6a562a70302/netty-codec-http2-4.1.30.Final.jar
d http://central.maven.org/maven2/io/netty/netty-codec-socks/4.1.30.Final/netty-codec-socks-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.30.Final/1384c630e8a0eeef33ad12a28791dce6e1d8767c/netty-codec-http-4.1.30.Final.jar
d http://central.maven.org/maven2/io/netty/netty-codec-http/4.1.30.Final/netty-codec-http-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.30.Final/ea272e3bb281d3a91d27278f47e61b4de285cc27/netty-codec-socks-4.1.30.Final.jar
d http://central.maven.org/maven2/io/netty/netty-common/4.1.30.Final/netty-common-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.30.Final/5dca0c34d8f38af51a2398614e81888f51cf811a/netty-common-4.1.30.Final.jar
d http://central.maven.org/maven2/io/netty/netty-handler-proxy/4.1.30.Final/netty-handler-proxy-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.30.Final/ecc076332ed103411347f4806a44ee32d9d9cb5f/netty-handler-4.1.30.Final.jar
d http://central.maven.org/maven2/io/netty/netty-handler/4.1.30.Final/netty-handler-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.30.Final/1baa1568fa936caddca0fae96fdf127fd5cbad16/netty-handler-proxy-4.1.30.Final.jar
d http://central.maven.org/maven2/io/netty/netty-resolver/4.1.30.Final/netty-resolver-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.30.Final/5106fd687066ffd712e5295d32af4e2ac6482613/netty-resolver-4.1.30.Final.jar
d http://central.maven.org/maven2/io/netty/netty-transport/4.1.30.Final/netty-transport-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.30.Final/3f4bcf2e9fff1361ac9ad0bd27a10a1b31399294/netty-resolver-dns-4.1.30.Final.jar
d http://central.maven.org/maven2/io/netty/netty-resolver-dns/4.1.30.Final/netty-resolver-dns-4.1.30.Final.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.30.Final/3d27bb432a3b125167ac161b26415ad29ec17f02/netty-transport-4.1.30.Final.jar
d http://central.maven.org/maven2/io/projectreactor/reactor-core/3.1.0.RELEASE/reactor-core-3.1.0.RELEASE.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.1.0.RELEASE/f584a224eb7059a9a4284920b46ce7113bc5a2f3/reactor-core-3.1.0.RELEASE.jar
d http://central.maven.org/maven2/io/vertx/vertx-bridge-common/3.6.0/vertx-bridge-common-3.6.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.vertx/vertx-auth-common/3.6.0/ef98a948e84f7446bcc76263fa0a0d182aa5d607/vertx-auth-common-3.6.0.jar
d http://central.maven.org/maven2/io/vertx/vertx-auth-common/3.6.0/vertx-auth-common-3.6.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.vertx/vertx-bridge-common/3.6.0/7eed656c64c5c6c33104521b0eb99adbb00fd0e2/vertx-bridge-common-3.6.0.jar
d http://central.maven.org/maven2/io/vertx/vertx-core/3.6.0/vertx-core-3.6.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.vertx/vertx-core/3.6.0/e3f35beea601f90c9316cdc4aa9fe8da99086b1a/vertx-core-3.6.0.jar
d http://central.maven.org/maven2/io/vertx/vertx-web-common/3.6.0/vertx-web-common-3.6.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.vertx/vertx-web/3.6.0/332fa80b409f1229809319fd5cc8c05ae42df738/vertx-web-3.6.0.jar
d http://central.maven.org/maven2/io/vertx/vertx-web/3.6.0/vertx-web-3.6.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/io.vertx/vertx-web-common/3.6.0/62085dd08c976409951efc7e14abaa8b9bc84b70/vertx-web-common-3.6.0.jar
d http://central.maven.org/maven2/net/bytebuddy/byte-buddy/1.9.3/byte-buddy-1.9.3.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy/1.9.3/f32e510b239620852fc9a2387fac41fd053d6a4d/byte-buddy-1.9.3.jar
d http://central.maven.org/maven2/org/jetbrains/annotations/16.0.3/annotations-16.0.3.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy-agent/1.9.3/f5b78c16cf4060664d80b6ca32d80dca4bd3d264/byte-buddy-agent-1.9.3.jar
d http://central.maven.org/maven2/net/bytebuddy/byte-buddy-agent/1.9.3/byte-buddy-agent-1.9.3.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/org.jetbrains/annotations/16.0.3/62c7299ced2a089cc541726c6d763da9417604a0/annotations-16.0.3.jar
d http://central.maven.org/maven2/org/jetbrains/kotlin/kotlin-reflect/1.3.0/kotlin-reflect-1.3.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.3.0/6fd129fd9ba8581f2cb9c58bfd431dda4ee0457e/kotlin-reflect-1.3.0.jar
d http://central.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-common/1.3.0/kotlin-stdlib-common-1.3.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.3.0/a134b0cfe9bb44f98b0b3e889cda07923eea9428/kotlin-stdlib-1.3.0.jar
d http://central.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib/1.3.0/kotlin-stdlib-1.3.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-common/1.3.0/84a2e0288dc17cd64d692eb1e5e0de8cd5ff0846/kotlin-stdlib-common-1.3.0.jar
d http://central.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-jdk7/1.3.0/kotlin-stdlib-jdk7-1.3.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk7/1.3.0/683e04a4e7f17437d7e1390480f312e122e42e9e/kotlin-stdlib-jdk7-1.3.0.jar
d http://central.maven.org/maven2/org/jetbrains/kotlinx/kotlinx-coroutines-core/1.0.0/kotlinx-coroutines-core-1.0.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.3.0/7157cc6bc26328e2e97723d65d4ed7d7eddf603b/kotlin-stdlib-jdk8-1.3.0.jar
d http://central.maven.org/maven2/org/jetbrains/kotlin/kotlin-stdlib-jdk8/1.3.0/kotlin-stdlib-jdk8-1.3.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-core/1.0.0/a33c4bf7581f71f01d4ed660b09ca199abbb44da/kotlinx-coroutines-core-1.0.0.jar
d http://central.maven.org/maven2/org/jetbrains/kotlinx/kotlinx-coroutines-core-common/1.0.0/kotlinx-coroutines-core-common-1.0.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-core-common/1.0.0/5f7551ec1edc068deaa0397100a1f14f32270274/kotlinx-coroutines-core-common-1.0.0.jar
d http://central.maven.org/maven2/org/jetbrains/kotlinx/kotlinx-coroutines-reactor/1.0.0/kotlinx-coroutines-reactor-1.0.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-reactive/1.0.0/c611be261e60a8f75965595179a31110e3da0eff/kotlinx-coroutines-reactive-1.0.0.jar
d http://central.maven.org/maven2/org/jetbrains/kotlinx/kotlinx-coroutines-reactive/1.0.0/kotlinx-coroutines-reactive-1.0.0.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-reactor/1.0.0/a184a135fc88e26a13260b3a1fd0228ebaa09d41/kotlinx-coroutines-reactor-1.0.0.jar
d http://central.maven.org/maven2/org/reactivestreams/reactive-streams/1.0.1/reactive-streams-1.0.1.jar /home/oleksiyp/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.1/1b1c911686eb40179219466e6a59b634b9d7a748/reactive-streams-1.0.1.jar
d http://central.maven.org/maven2/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar /home/oleksiyp/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar
d http://central.maven.org/maven2/ch/qos/logback/logback-core/1.2.3/logback-core-1.2.3.jar /home/oleksiyp/.m2/repository/ch/qos/logback/logback-core/1.2.3/logback-core-1.2.3.jar
d http://central.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.9.0/jackson-annotations-2.9.0.jar /home/oleksiyp/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.9.0/jackson-annotations-2.9.0.jar
d http://central.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.9.7/jackson-core-2.9.7.jar /home/oleksiyp/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.9.7/jackson-core-2.9.7.jar
d http://central.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.9.7/jackson-databind-2.9.7.jar /home/oleksiyp/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.9.7/jackson-databind-2.9.7.jar
d http://central.maven.org/maven2/io/mockk/mockk/1.9/mockk-1.9.jar /home/oleksiyp/.m2/repository/io/mockk/mockk/1.9.1-preload-SNAPSHOT/mockk-1.9.1-preload-SNAPSHOT.jar
d http://central.maven.org/maven2/io/mockk/mockk-agent-api/1.9/mockk-agent-api-1.9.jar /home/oleksiyp/.m2/repository/io/mockk/mockk-agent-api/1.9.1-preload-SNAPSHOT/mockk-agent-api-1.9.1-preload-SNAPSHOT.jar
d http://central.maven.org/maven2/io/mockk/mockk-agent-common/1.9/mockk-agent-common-1.9.jar /home/oleksiyp/.m2/repository/io/mockk/mockk-agent-common/1.9.1-preload-SNAPSHOT/mockk-agent-common-1.9.1-preload-SNAPSHOT.jar
d http://central.maven.org/maven2/io/mockk/mockk-agent-jvm/1.9/mockk-agent-jvm-1.9.jar /home/oleksiyp/.m2/repository/io/mockk/mockk-agent-jvm/1.9.1-preload-SNAPSHOT/mockk-agent-jvm-1.9.1-preload-SNAPSHOT.jar
d http://central.maven.org/maven2/io/mockk/mockk-dsl/1.9/mockk-dsl-1.9.jar /home/oleksiyp/.m2/repository/io/mockk/mockk-common/1.9.1-preload-SNAPSHOT/mockk-common-1.9.1-preload-SNAPSHOT.jar
d http://central.maven.org/maven2/io/mockk/mockk-common/1.9/mockk-common-1.9.jar /home/oleksiyp/.m2/repository/io/mockk/mockk-dsl/1.9.1-preload-SNAPSHOT/mockk-dsl-1.9.1-preload-SNAPSHOT.jar
d http://central.maven.org/maven2/io/mockk/mockk-dsl-jvm/1.9/mockk-dsl-jvm-1.9.jar /home/oleksiyp/.m2/repository/io/mockk/mockk-dsl-jvm/1.9.1-preload-SNAPSHOT/mockk-dsl-jvm-1.9.1-preload-SNAPSHOT.jar
d http://central.maven.org/maven2/junit/junit/4.12/junit-4.12.jar /home/oleksiyp/.m2/repository/junit/junit/4.12/junit-4.12.jar
d http://central.maven.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar /home/oleksiyp/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar
d http://central.maven.org/maven2/org/objenesis/objenesis/2.6/objenesis-2.6.jar /home/oleksiyp/.m2/repository/org/objenesis/objenesis/2.6/objenesis-2.6.jar
d http://central.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar /home/oleksiyp/.m2/repository/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.jar

cat >ProblemWithSubclassingKotlinCode.java <<JAVA

import kotlinx.coroutines.channels.Channel;
import net.bytebuddy.ByteBuddy;
import net.bytebuddy.dynamic.loading.ClassLoadingStrategy;
import net.bytebuddy.implementation.MethodDelegation;
import net.bytebuddy.implementation.attribute.MethodAttributeAppender;
import net.bytebuddy.implementation.bind.annotation.*;
import net.bytebuddy.matcher.ElementMatchers;
import org.objenesis.ObjenesisStd;
import java.io.PrintWriter;

import java.lang.reflect.Method;
import java.util.concurrent.Callable;

public class ProblemWithSubclassingKotlinCode {
    public static class Interceptor {
        @RuntimeType
        @BindingPriority(BindingPriority.DEFAULT * 2)
        public static void intercept1(@This Object self,
                                      @Origin Method method,
                                      @AllArguments Object[] args,
                                      @SuperCall final Callable<Object> originalMethod) throws Throwable {
        }

        @RuntimeType
        public static void intercept2(@This Object self,
                                      @Origin Method method,
                                      @AllArguments Object[] args) throws Throwable {
        }
    }

    public static void main(String[] args) throws Throwable {
        try {
            Thread.sleep(10);
            Class<? extends Channel> cls = new ByteBuddy()
                    .subclass(Channel.class)
                    .method(ElementMatchers.any())
                    .intercept(
                            MethodDelegation.withDefaultConfiguration()
                                    .to(Interceptor.class)
                    )
                    .attribute(MethodAttributeAppender.ForInstrumentedMethod.INCLUDING_RECEIVER)
                    .make()
                    .load(Channel.class.getClassLoader(),
                            ClassLoadingStrategy.Default.INJECTION.with(Channel.class.getProtectionDomain()))
                    .getLoaded();

            new ObjenesisStd().newInstance(cls).cancel();
            System.out.println(1);
        } catch (Throwable error) {
            try (PrintWriter pw = new PrintWriter("error.txt")) {
                error.printStackTrace(pw);
            }
            System.out.println(0);
        }
    }
}
JAVA

CP=".:$(find jars -name *.jar | tr "\n" ':')"
CP="$JAVA_HOME/jre/lib/charsets.jar:$JAVA_HOME/jre/lib/deploy.jar:$JAVA_HOME/jre/lib/ext/cldrdata.jar:$JAVA_HOME/jre/lib/ext/dnsns.jar:$JAVA_HOME/jre/lib/ext/jaccess.jar:$JAVA_HOME/jre/lib/ext/jfxrt.jar:$JAVA_HOME/jre/lib/ext/localedata.jar:$JAVA_HOME/jre/lib/ext/nashorn.jar:$JAVA_HOME/jre/lib/ext/sunec.jar:$JAVA_HOME/jre/lib/ext/sunjce_provider.jar:$JAVA_HOME/jre/lib/ext/sunpkcs11.jar:$JAVA_HOME/jre/lib/ext/zipfs.jar:$JAVA_HOME/jre/lib/javaws.jar:$JAVA_HOME/jre/lib/jce.jar:$JAVA_HOME/jre/lib/jfr.jar:$JAVA_HOME/jre/lib/jfxswt.jar:$JAVA_HOME/jre/lib/jsse.jar:$JAVA_HOME/jre/lib/management-agent.jar:$JAVA_HOME/jre/lib/plugin.jar:$JAVA_HOME/jre/lib/resources.jar:$JAVA_HOME/jre/lib/rt.jar:$CP"

$JAVA_HOME/bin/javac -cp $CP ProblemWithSubclassingKotlinCode.java || exit

n=20
echo Running "${n}x" times
SUM=0
for i in `seq $n`; do
    RES=$($JAVA_HOME/bin/java -Dfile.encoding=UTF-8 -classpath $CP ProblemWithSubclassingKotlinCode)
    SUM=$(expr $SUM + $RES)
    echo -n $RES
done
echo
echo Successes - $SUM

[oleksiyp]

Thanks for investigation and your effort. Opened ticket https://youtrack.jetbrains.com/issue/KT-29658

[raphw]

I found the issue, it is because of Kotlin’s declaration of kotlinx.coroutines.channels.ReceiveChannel. It declares two methods:

public abstract void cancel();
public abstract boolean cancel();

While the JVM supports methods differing only by return type, the Java language does not and neither does the reflection API which Byte Buddy relies on. The result of the instrumentation will depend on the order those two method representations are added to a tree map which makes the outcome random where the other method is discarded as an incompatible covariant override attempt. The Java language always chooses to invoke the void method in this case but Byte Buddy chooses randomly which one to instrumented by said tree map order.

You might be able to work around this by registering MethodGraph.Compiler.Default.forJvmHierarchy() which forces Byte Buddy to consider the return type when distinguishing methods but this way you will also miss covariant overrides such as when overriding a method returning Object to return String instead.

All in all I consider this a bug in Kotlin, it should not allow to declare methods with the same name as it is not defined which method would be invoked from the Java language when both methods are declared.