gRPC Server Streaming: Client Only Receives #onNext Response After Server Calls #onCompleted

Describe the bug

The server’s responseObserver#onNext calls seem to be collected and only sent to the client after the server calls responseObserver#onComplete.

Expected behavior

The server’s responseObserver#onNext calls are sent to the client immediately when the server calls them and not only after responseObserver#onComplete was called.

Actual behavior

The server’s responseObserver#onNext calls seem to be collected and only sent to the client after the server calls responseObserver#onComplete.

How to Reproduce?

Create a Maven project and follow these steps (note that I put client and server in the same Maven project)

Generate code for this protobuf:

syntax = "proto3";

option java_multiple_files = true;
option java_package = "examples";
option java_outer_classname = "HelloWorldProto";

package helloworld;

service Greeter {
    rpc SayHelloServerStr (HelloRequest) returns (stream HelloReply) {}
}

message HelloRequest {
    repeated int32 duration = 1;
}

message HelloReply {
    string message = 1;
}

pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.acme</groupId>
    <artifactId>grpc-plain-text-quickstart</artifactId>
    <version>1.0.0-SNAPSHOT</version>

    <properties>
        <quarkus.platform.artifact-id>quarkus-bom</quarkus.platform.artifact-id>
        <quarkus.platform.group-id>io.quarkus</quarkus.platform.group-id>
        <quarkus.platform.version>2.14.1.Final</quarkus.platform.version>
        <surefire-plugin.version>3.0.0-M7</surefire-plugin.version>
        <compiler-plugin.version>3.8.0</compiler-plugin.version>
        <assertj.version>3.22.0</assertj.version>

        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>${quarkus.platform.group-id}</groupId>
                <artifactId>${quarkus.platform.artifact-id}</artifactId>
                <version>${quarkus.platform.version}</version>
                <scope>import</scope>
                <type>pom</type>
            </dependency>
            <dependency>
                <groupId>org.assertj</groupId>
                <artifactId>assertj-core</artifactId>
                <version>${assertj.version}</version>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-resteasy-reactive</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-grpc</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-junit5</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>io.rest-assured</groupId>
            <artifactId>rest-assured</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.awaitility</groupId>
            <artifactId>awaitility</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.assertj</groupId>
            <artifactId>assertj-core</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
	<plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${compiler-plugin.version}</version>
                <configuration>
                    <source>11</source>
                    <target>11</target>
                </configuration>
            </plugin>

            <plugin>
                <groupId>${quarkus.platform.group-id}</groupId>
                <artifactId>quarkus-maven-plugin</artifactId>
                <version>${quarkus.platform.version}</version>
                <executions>
                    <execution>
                        <goals>
			    <goal>build</goal>
			    <goal>generate-code</goal>
		            <goal>generate-code-tests</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${surefire-plugin.version}</version>
                <configuration>
                    <systemPropertyVariables>
                        <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
                    </systemPropertyVariables>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <profiles>
        <profile>
            <id>native</id>
            <activation>
                <property>
                    <name>native</name>
                </property>
            </activation>
            <properties>
                <quarkus.package.type>native</quarkus.package.type>
            </properties>
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-failsafe-plugin</artifactId>
                        <version>${surefire-plugin.version}</version>
                        <executions>
                            <execution>
                                <goals>
                                    <goal>integration-test</goal>
                                    <goal>verify</goal>
                                </goals>
                                <configuration>
                                    <systemPropertyVariables>
                                        <native.image.path>${project.build.directory}/${project.build.finalName}-runner</native.image.path>
                                    </systemPropertyVariables>
                                </configuration>
                            </execution>
                        </executions>
                    </plugin>
                </plugins>
            </build>
        </profile>
    </profiles>
</project>

Java server service impl.:

import examples.GreeterGrpc.GreeterImplBase;
import examples.HelloReply;
import examples.HelloRequest;
import io.grpc.stub.StreamObserver;
import io.quarkus.grpc.GrpcService;

@GrpcService
public class HelloWorldService extends GreeterImplBase {
	
	@Override
	public void sayHelloServerStr(HelloRequest request, StreamObserver<HelloReply> responseObserver) {

		request.getDurationList().parallelStream().forEach(duration -> {
			System.out.println("server: starting " + duration);
			sleep(duration);
			responseObserver.onNext(HelloReply.newBuilder().setMessage("Slept for " + duration + " ms").build());
			System.out.println("server: finished " + duration);
		});

		System.out.println("server: reached #onCompleted");
		sleep(2000);
		responseObserver.onCompleted();
		System.out.println("server: finished #onCompleted");
	}

	public static void sleep(long duration) {
		try {
			Thread.sleep(duration);
		} catch (InterruptedException e) {
			throw new RuntimeException(e);
		}
	}

}

Java client code:

import static java.util.Arrays.asList;

import examples.GreeterGrpc;
import examples.HelloRequest;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;

public class MainServerStreaming {
	public static void main(String[] args) {
		ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 9000).usePlaintext().build();

		try {
			final var durations = asList(1000, 1500, 50);

			final var request = HelloRequest.newBuilder().addAllDuration(durations).build();

			final var stub = GreeterGrpc.newBlockingStub(channel);

			stub.sayHelloServerStr(request).forEachRemaining(response -> {
				System.out.println("client:" + response.getMessage());
			});
			System.out.println("client: done");

		} finally {
			channel.shutdown();
		}
	}
}

Steps

• Run the server (e.g. with mvn compile quarkus:dev)
• Run the client

You’ll notice that the client output

client:Slept for 50 ms
client:Slept for 1000 ms
client:Slept for 1500 ms
client: done

only starts after the server calls responseObserver.onCompleted();.

Output of uname -a or ver

MSYS_NT-10.0-19044 LT-000X9DJ1M2 3.3.3-341.x86_64 2022-01-17 11:45 UTC x86_64 Msys

Output of java -version

openjdk version “17.0.3” 2022-04-19 OpenJDK Runtime Environment Temurin-17.0.3+7 (build 17.0.3+7) OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (build 17.0.3+7, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.14.1.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.5

Additional information

I implemented a gRPC server with vanilla gRPC with the Maven dependencies described on https://github.com/grpc/grpc-java/blob/master/README.md. I used the very same service implementation and client code. I only had to start the server on my own like

public static void main(String[] args) throws Exception {
	System.out.println("Server started on 9000");

	Server server = ServerBuilder.forPort(9000).addService(new HelloWorldService()).build();
	server.start();

	Runtime.getRuntime().addShutdownHook(new Thread(() -> {
		server.shutdown();
		System.out.println("Successfully stopped the server");
	}));

	server.awaitTermination();
}

-> The vanilla gRPC server works as specified: The #onNext responses are sent to the client immediately. The client starts printing its output already before the server calls #onCompleted on the observer.

I.e. with vanilla gRPC I can achieve the desired behavior I cannot achieve with Quarkus gRPC.

[cescoffier]

Yes, it works as expected, see my comment where I explain the monopolization of the event loop.

[cescoffier]

Well, that was interesting and the test is wrong, not the Quarkus code 😄

Let me explain. The sayHelloServerStr runs on the event loop, the same event loop used underneath by netty to write the frames into the wire. When calling onNext, the message gets serialized and enqueued to a write queue. A flush action is scheduled on the same event loop. So, the flush happens later (to avoid writing too much).

However, the code shown above is monopolizing that thread, meaning that while the messages are written on the write queue, the flush doesn’t have the chance to run, as the event loop is never released.

How to fix the code: add @Blocking, and it will work.

Note that if you have kept monopolizing/blocking the event loop for more than 2s, you would have seen a warning in the log.