[BUG] java.lang.NoSuchMethodError: io.netty.channel.SingleThreadEventLoop. calling azure blob storage

Describe the bug When calling the azure blob storage in an azure function runtime locally. The endpoint will turn unresponsive and the log will print an error. Exception or Stack Trace Add the exception log and stack trace if available

Host lock lease acquired by instance ID ‘000000000000000000000000AACC515C’. [5/8/2020 6:52:41 AM] Executing HTTP request: { [5/8/2020 6:52:41 AM] “requestId”: “1aab61e3-e2cf-4928-b0d3-2a07e6294509”, [5/8/2020 6:52:41 AM] “method”: “GET”, [5/8/2020 6:52:41 AM] “uri”: “/api/HttpExample” [5/8/2020 6:52:41 AM] } [5/8/2020 6:52:41 AM] Executing ‘Functions.HttpExample’ (Reason=‘This function was programmatically called via the host APIs.’, Id=f0d0e7aa-db9e-476b-ab74-367a1dad948b) [5/8/2020 6:52:41 AM] Java HTTP trigger processed a request. [5/8/2020 6:52:41 AM] var storageURI = https://stortestjarromijava.blob.core.windows.net [5/8/2020 6:52:41 AM] blobServiceClient created [5/8/2020 6:52:42 AM] May 08, 2020 8:52:42 AM reactor.util.Loggers$Slf4JLogger error

[5/8/2020 6:52:42 AM] SEVERE: Scheduler worker in group main failed with an uncaught exception

[5/8/2020 6:52:42 AM] java.lang.NoSuchMethodError: io.netty.channel.SingleThreadEventLoop.<init>(Lio/netty/channel/EventLoopGroup;Ljava/util/concurrent/Executor;ZLjava/util/Queue;Ljava/util/Queue;Lio/netty/util/concurrent/RejectedExecutionHandler;)V [5/8/2020 6:52:42 AM] at io.netty.channel.epoll.EpollEventLoop.<init>(EpollEventLoop.java:87) [5/8/2020 6:52:42 AM] at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:143) [5/8/2020 6:52:42 AM] at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35) [5/8/2020 6:52:42 AM] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) [5/8/2020 6:52:42 AM] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) [5/8/2020 6:52:42 AM] at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) [5/8/2020 6:52:42 AM] at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) [5/8/2020 6:52:42 AM] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:104) [5/8/2020 6:52:42 AM] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:91) [5/8/2020 6:52:42 AM] at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:68) [5/8/2020 6:52:42 AM] at reactor.netty.resources.DefaultLoopEpoll.newEventLoopGroup(DefaultLoopEpoll.java:64) [5/8/2020 6:52:42 AM] at reactor.netty.resources.DefaultLoopResources.cacheNativeServerLoops(DefaultLoopResources.java:234) [5/8/2020 6:52:42 AM] at reactor.netty.resources.DefaultLoopResources.cacheNativeClientLoops(DefaultLoopResources.java:248) [5/8/2020 6:52:42 AM] at reactor.netty.resources.DefaultLoopResources.onClient(DefaultLoopResources.java:184) [5/8/2020 6:52:42 AM] at reactor.netty.tcp.TcpResources.onClient(TcpResources.java:179) [5/8/2020 6:52:42 AM] at reactor.netty.http.client.HttpClientConnect$HttpTcpClient.connect(HttpClientConnect.java:134) [5/8/2020 6:52:42 AM] at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) [5/8/2020 6:52:42 AM] at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) [5/8/2020 6:52:42 AM] at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) [5/8/2020 6:52:42 AM] at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) [5/8/2020 6:52:42 AM] at reactor.netty.tcp.TcpClientOperator.connect(TcpClientOperator.java:43) [5/8/2020 6:52:42 AM] at reactor.netty.tcp.TcpClient.connect(TcpClient.java:187) [5/8/2020 6:52:42 AM] at reactor.netty.http.client.HttpClientFinalizer.connect(HttpClientFinalizer.java:68) [5/8/2020 6:52:42 AM] at reactor.netty.http.client.HttpClientFinalizer.responseConnection(HttpClientFinalizer.java:85) [5/8/2020 6:52:42 AM] at com.azure.core.http.netty.NettyAsyncHttpClient.send(NettyAsyncHttpClient.java:70) [5/8/2020 6:52:42 AM] at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:44) [5/8/2020 6:52:42 AM] at com.azure.storage.common.policy.ScrubEtagPolicy.process(ScrubEtagPolicy.java:35) [5/8/2020 6:52:42 AM] at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) [5/8/2020 6:52:42 AM] at com.azure.core.http.policy.HttpLoggingPolicy.process(HttpLoggingPolicy.java:84) [5/8/2020 6:52:42 AM] at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) [5/8/2020 6:52:42 AM] at com.azure.storage.common.policy.ResponseValidationPolicyBuilder$ResponseValidationPolicy.process(ResponseValidationPolicyBuilder.java:77) [5/8/2020 6:52:42 AM] at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) [5/8/2020 6:52:42 AM] at com.azure.storage.common.policy.StorageSharedKeyCredentialPolicy.process(StorageSharedKeyCredentialPolicy.java:41) [5/8/2020 6:52:42 AM] at com.azure.core.http.HttpPipelineNextPolicy.process(HttpPipelineNextPolicy.java:46) [5/8/2020 6:52:42 AM] at com.azure.core.http.policy.AddDatePolicy.lambda$process$0(AddDatePolicy.java:29) [5/8/2020 6:52:42 AM] at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44) [5/8/2020 6:52:42 AM] at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) [5/8/2020 6:52:42 AM] at reactor.core.publisher.MonoDelaySubscription.accept(MonoDelaySubscription.java:52) [5/8/2020 6:52:42 AM] at reactor.core.publisher.MonoDelaySubscription.accept(MonoDelaySubscription.java:33) [5/8/2020 6:52:42 AM] at reactor.core.publisher.FluxDelaySubscription$DelaySubscriptionOtherSubscriber.onNext(FluxDelaySubscription.java:123) [5/8/2020 6:52:42 AM] at reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:117) [5/8/2020 6:52:42 AM] at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) [5/8/2020 6:52:42 AM] at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) [5/8/2020 6:52:42 AM] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [5/8/2020 6:52:42 AM] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [5/8/2020 6:52:42 AM] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [5/8/2020 6:52:42 AM] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [5/8/2020 6:52:42 AM] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [5/8/2020 6:52:42 AM] at java.lang.Thread.run(Thread.java:748)

` To Reproduce Build a function (see code below)

Run mvn clean package azure-functions:run Call the generated api. The api call will timeout and a stacktrace will be printed.

Code Snippet

`@FunctionName(“HttpExample”) public HttpResponseMessage run(@HttpTrigger(name = “req”, methods = { HttpMethod.GET, HttpMethod.POST }, authLevel = AuthorizationLevel.ANONYMOUS) HttpRequestMessage<Optional<String>> request, final ExecutionContext context) throws Exception { context.getLogger().info(“Java HTTP trigger processed a request.”);

    String storageURI = "https://stortestjarromijava.blob.core.windows.net"; // my storage account
    context.getLogger().info("var storageURI = " + storageURI);
    

    StorageSharedKeyCredential defaultCredential = new StorageSharedKeyCredential("someAccountName", "someKey");

    
    BlobServiceClient blobServiceClient = new BlobServiceClientBuilder()
    .endpoint(storageURI)
    .credential(defaultCredential)
    .buildClient(); 
    context.getLogger().info("blobServiceClient created ");

    DateTimeFormatter dtf = DateTimeFormatter.ofPattern("HHmmA");
    LocalDateTime now = LocalDateTime.now();
    String tstamp = dtf.format(now);
    try{
        blobServiceClient.createBlobContainer(tstamp); //create container
        context.getLogger().info("container created named " + tstamp);
    }
    catch (Exception e){
        e.printStackTrace();
        context.getLogger().info("FAIL!");
    }
    return request.createResponseBuilder(HttpStatus.OK).body("Success!").build();
}`

Build this using the pom.xml `<?xml version="1.0" encoding="UTF-8" ?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion>

<groupId>com.javafuncmi</groupId>
<artifactId>javafuncmi-test</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>

<name>Azure Java Functions</name>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <azure.functions.maven.plugin.version>1.4.1</azure.functions.maven.plugin.version>
    <azure.functions.java.library.version>1.3.1</azure.functions.java.library.version>
    <functionAppName>javafuncmi-test-20200501183037488</functionAppName>
    <stagingDirectory>${project.build.directory}/azure-functions/${functionAppName}</stagingDirectory>
</properties>

<repositories>
    <repository>
        <id>maven.snapshots</id>
        <name>Maven Central Snapshot Repository</name>
        <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
        <releases>
            <enabled>false</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </repository>
</repositories>

<pluginRepositories>
    <pluginRepository>
        <id>maven.snapshots</id>
        <name>Maven Central Snapshot Repository</name>
        <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
        <releases>
            <enabled>false</enabled>
        </releases>
        <snapshots>
            <enabled>true</enabled>
        </snapshots>
    </pluginRepository>
</pluginRepositories>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.4.2</version>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-core</artifactId>
            <version>2.23.4</version>
        </dependency>
        <dependency>
            <groupId>com.microsoft.azure.functions</groupId>
            <artifactId>azure-functions-java-library</artifactId>
            <version>${azure.functions.java.library.version}</version>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>com.google.code.gson</groupId>
        <artifactId>gson</artifactId>
        <version>2.8.6</version>
    </dependency>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-core-http-netty</artifactId>
        <version>1.2.0</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-jdk14</artifactId>
        <version>1.7.25</version>
    </dependency>        
    <dependency>
        <groupId>com.microsoft.azure.functions</groupId>
        <artifactId>azure-functions-java-library</artifactId>
    </dependency>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-storage-blob</artifactId>
        <version>12.6.0</version>
    </dependency>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-identity</artifactId>
        <version>1.0.5</version>
    </dependency>
    <!-- Test -->
    <dependency>
        <groupId>org.junit.jupiter</groupId>
        <artifactId>junit-jupiter</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.mockito</groupId>
        <artifactId>mockito-core</artifactId>
        <scope>test</scope>
    </dependency>
</dependencies>

<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>com.microsoft.azure</groupId>
                <artifactId>azure-functions-maven-plugin</artifactId>
                <version>${azure.functions.maven.plugin.version}</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-resources-plugin</artifactId>
                <version>3.1.0</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.1.1</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-clean-plugin</artifactId>
                <version>3.1.0</version>
            </plugin>
        </plugins>
    </pluginManagement>

    <plugins>
        <plugin>
            <groupId>com.microsoft.azure</groupId>
            <artifactId>azure-functions-maven-plugin</artifactId>
            <configuration>
                <!-- function app name -->
                <appName>${functionAppName}</appName>
                <!-- function app resource group -->
                <resourceGroup>java-functions-group</resourceGroup>
                <!-- function app service plan name -->
                <appServicePlanName>java-functions-app-service-plan</appServicePlanName>
                <!-- function app region-->
                <!-- refers https://github.com/microsoft/azure-maven-plugins/tree/develop/azure-functions-maven-plugin#supported-regions for all valid values -->
                <region>westus</region>
                <!-- function pricingTier, default to be consumption if not specified -->
                <!-- refers https://github.com/microsoft/azure-maven-plugins/tree/develop/azure-functions-maven-plugin#supported-pricing-tiers for all valid values -->
                <!-- <pricingTier></pricingTier> -->
                <runtime>
                    <!-- runtime os, could be windows, linux or docker-->
                    <os>windows</os>
                    <!-- for docker function, please set the following parameters -->
                    <!-- <image>[hub-user/]repo-name[:tag]</image> -->
                    <!-- <serverId></serverId> -->
                    <!-- <registryUrl></registryUrl>  -->
                </runtime>
                <appSettings>
                    <property>
                        <name>FUNCTIONS_EXTENSION_VERSION</name>
                        <value>~3</value>
                    </property>
                </appSettings>
            </configuration>
            <executions>
                <execution>
                    <id>package-functions</id>
                    <goals>
                        <goal>package</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-resources-plugin</artifactId>
            <executions>
                <execution>
                    <id>copy-resources</id>
                    <phase>package</phase>
                    <goals>
                        <goal>copy-resources</goal>
                    </goals>
                    <configuration>
                        <overwrite>true</overwrite>
                        <outputDirectory>${stagingDirectory}</outputDirectory>
                        <resources>
                            <resource>
                                <directory>${project.basedir}</directory>
                                <includes>
                                    <include>host.json</include>
                                    <include>local.settings.json</include>
                                </includes>
                            </resource>
                        </resources>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
                <execution>
                    <id>copy-dependencies</id>
                    <phase>prepare-package</phase>
                    <goals>
                        <goal>copy-dependencies</goal>
                    </goals>
                    <configuration>
                        <outputDirectory>${stagingDirectory}/lib</outputDirectory>
                        <overWriteReleases>false</overWriteReleases>
                        <overWriteSnapshots>false</overWriteSnapshots>
                        <overWriteIfNewer>true</overWriteIfNewer>
                        <includeScope>runtime</includeScope>
                        <excludeArtifactIds>azure-functions-java-library</excludeArtifactIds>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <!--Remove obj folder generated by .NET SDK in maven clean-->
        <plugin>
            <artifactId>maven-clean-plugin</artifactId>
            <version>3.1.0</version>
            <configuration>
                <filesets>
                    <fileset>
                        <directory>obj</directory>
                    </fileset>
                </filesets>
            </configuration>
        </plugin>
    </plugins>
</build>

</project>` Add the code snippet that causes the issue.

Expected behavior I expect the calls to the azure blob storage to just function.

Screenshots If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

• OS: Linux
• IDE :InteliJ/Terminal
• Version of the Library used: 12.6.0

Additional context

• The method that is not found should be available according to maven dependencies.
• The exact same code was working for a colleague on Windows.
• Bug was placed here since it is unclear what causes the exact issue. Blob storage library works in a unit test without azure functions runtime. Seems to work on windows.
• Will make a reproducer repo once I’m back from my long weekend if needed.

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[anuchandy]

@jelmew - just to unblock your usecase temporarily, you can exclude the use of Epoll transport in your project like below:

<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-core-http-netty</artifactId>
<version>1.2.0</version>
<exclusions>
    <exclusion>
        <groupId>io.netty</groupId>
        <artifactId>netty-transport-native-epoll</artifactId>
    </exclusion>
    <exclusion>
        <groupId>io.netty</groupId>
        <artifactId>netty-transport-native-unix-common</artifactId>
    </exclusion>
</exclusions>
</dependency>

this issue is slightly different from the normal dependency conflict, it is also possible that EPoll Socket is disabled in Azure Function. I’ve created an issue in Function repo, where this can be reproduced using low-level netty (without Azure Storage SDK or Reactor-netty in the classpath). Here is the link to the issue https://github.com/Azure/azure-functions-java-worker/issues/373 Please subscribe to this issue and meantime use the above suggestion to unblock your development.

[joshfree]

@anuchandy could you take the first look real quick?