"StatusRuntimeException: INTERNAL: HTTP/2 error code: INTERNAL_ERROR" ,"StatusRuntimeException:UNKNOWN"

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.0.0, 1.2.0

What JVM are you using (java -version)?

1.8

What did you do?

I am running a gRPC-java server with multiple grpc services defined.

When running multiple stubs which multiple threads share, so multiple rpc calls simultaneously, I am seeing this exception below, the exception is only occasionally appear.

(1) io.grpc.statusRuntimeException: UNKNOWN at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:227) at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:208) at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141) … … (2) io.grpc.statusRuntimeException: INTERNAL: HTTP/2 error code: INTERNAL_ERROR Received Rst Stream at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:227) at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:208) at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141) … …

What is benchmark scenario?

1. my benchmark is a distributed program: one client to three servers.

2. client tps: 150k+ /s, each server receive message : 50k+ /s

question:

1. why the exception is only occasionally appear? Concurrency is too big?
2. mutiple thread can call the same stub object at the same time?

[longbozhan]

I have the same problems when I send a bytes to server, the bytes’s size is about 14Mb, and I have set MAX_SIZE = 50 *1024*1024; channelBuilder.maxMessageSize(MAX_SIZE) (NettyChannelBuilder) nettyServerBuilder.maxMessageSize(MAX_SIZE); @ejona86

[kaiyangzhang]

The problem has been solved, and the problem is caused by the server. The queue size of grpc server’s thread pool is too small, resulting in tasks submit failed. the server throw exception below:

rejected execution of io.grpc.internal.SerializingExecutor$TaskRunner@5f207f61 on AsThreadPoolExecutor[GrpcServer_ASTERIA_SERVER, queue capacity=200, com.fiberhome.asteriadb.common.concurrent.AsthreadPoolExecutor@5dcfbbfe[Running, pool size=100, active threads =100, queued tasks =200, completed tasks =3473355]] at com.fiberhome.asteriadb.common.concurrent.AsAbortPolicy.rejectedExecution at java.util.concurrent.ThreadPoolExecutor.reject at java.util.concurrent.ThreadPoolExecutor.execute at com.fiberhome.asteriadb.common.current.AsThreadPoolExecutor.execute at io.grpc.internal.SerializingExecutor.execute at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener at io.grpc.internal.AbstractServerStream$TransportState.closeListener at io.grpc.internal.AbstractServerStream$TransportState.complete at io.grpc.nettyServerStream$TransportState.complete … …