Different message size leads to different performance

1. The attach code is for repetition
2. My question is , when i assign different size for the byte array ,i meet big difference in performance，for Example

The core code as below

public class ServerTest {
    private static final byte[] BYTES = new byte[1024 * 240];

    public static void main(String[] args) throws IOException, InterruptedException {
        TxnStreamRpcServer rpcServer = new TxnStreamRpcServer(9999, new TxnMessageProvider() {

            @Override
            public void dump(TxnOutputStream outputStream) {

                while (true) {

                    if (!outputStream.isReady()) {
                        LockSupport.parkNanos(1000);
                        continue;
                    }

                    outputStream.onNext(DumpReply.newBuilder().setMessage(ByteString.copyFrom(BYTES)).build());
                }
            }
        });
        rpcServer.start();
        rpcServer.blockUntilShutdown();
    }
}

When the byte array size is set to 1024 * 70, the receiveBytesPerSecond as below，just 100M+

2021-01-25 22:22:49.735 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :45770234 2021-01-25 22:22:59.797 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :104472261 2021-01-25 22:23:09.827 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :100809209 2021-01-25 22:23:19.857 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :102716003 2021-01-25 22:23:29.900 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :108558249 2021-01-25 22:23:39.930 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :102594140 2021-01-25 22:23:49.937 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :103662232 2021-01-25 22:23:59.987 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :107583347 2021-01-25 22:24:10.081 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :112357501 2021-01-25 22:24:20.090 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :102228552 2021-01-25 22:24:30.123 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :101991995 2021-01-25 22:24:40.208 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :112407680 2021-01-25 22:24:50.227 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :106744644 2021-01-25 22:25:00.285 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :111095863 2021-01-25 22:25:10.286 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :106801991 2021-01-25 22:25:20.367 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :109647846 2021-01-25 22:25:30.378 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :106235688 2021-01-25 22:25:40.383 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :107017043 2021-01-25 22:25:50.431 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :107418474 2021-01-25 22:26:00.512 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :102687330 2021-01-25 22:26:10.535 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :106995538 2021-01-25 22:26:20.615 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :114844936 2021-01-25 22:26:30.619 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :104852186

When the byte array size is set to 1024 * 80, the receiveBytesPerSecond as below，about 800M and gradually increased

2021-01-25 22:32:47.347 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :309492487 2021-01-25 22:32:57.349 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :380962984 2021-01-25 22:33:07.380 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :487914766 2021-01-25 22:33:17.394 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :367601180 2021-01-25 22:33:27.455 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :252768309 2021-01-25 22:33:37.477 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :248696686 2021-01-25 22:33:47.477 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :454276772 2021-01-25 22:33:57.530 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :468760935 2021-01-25 22:34:07.623 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :310098724 2021-01-25 22:34:17.673 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :438031243 2021-01-25 22:34:27.762 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :397642711 2021-01-25 22:34:37.770 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :409964080 2021-01-25 22:34:47.783 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :564382628 2021-01-25 22:34:57.783 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :870868504 2021-01-25 22:35:07.783 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :885590247 2021-01-25 22:35:17.783 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :797243406 2021-01-25 22:35:27.860 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :737668273 2021-01-25 22:35:37.860 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :821230753 2021-01-25 22:35:47.860 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :901385194 2021-01-25 22:35:57.860 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :817904638 2021-01-25 22:36:07.913 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :874792664 2021-01-25 22:36:17.931 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :802806045 2021-01-25 22:36:27.981 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :807066093

When the byte array size is set to 1024 * 160, the receiveBytesPerSecond as below，just 100M +

2021-01-25 22:39:22.085 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :52577539 2021-01-25 22:39:32.087 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :123685835 2021-01-25 22:39:42.145 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :114985719 2021-01-25 22:39:52.241 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :108513881 2021-01-25 22:40:02.340 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :114658031 2021-01-25 22:40:12.361 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :116460315 2021-01-25 22:40:22.404 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :108792416 2021-01-25 22:40:32.492 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :119163741 2021-01-25 22:40:42.525 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :161845103 2021-01-25 22:40:52.525 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :155225805 2021-01-25 22:41:02.610 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :119507813 2021-01-25 22:41:12.689 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :119409507 2021-01-25 22:41:22.752 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :132189339 2021-01-25 22:41:32.824 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :119294816 2021-01-25 22:41:42.902 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :117869373 2021-01-25 22:41:52.919 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :122964922 2021-01-25 22:42:02.919 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :119573351 2021-01-25 22:42:13.018 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :120654721 2021-01-25 22:42:23.086 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :113920733 2021-01-25 22:42:33.148 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :117050153 2021-01-25 22:42:43.235 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :117230382 2021-01-25 22:42:53.254 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :121293713 2021-01-25 22:43:03.277 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :123849679 2021-01-25 22:43:13.364 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :118033217

When the byte array size is set to 1024 * 240, the receiveBytesPerSecond as below，can up to 2000M ！！！

2021-01-25 22:48:12.992 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :568845354 2021-01-25 22:48:22.995 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :2116789908 2021-01-25 22:48:32.995 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :1672620631 2021-01-25 22:48:42.995 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :1572299766 2021-01-25 22:48:52.995 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :2053530254 2021-01-25 22:49:02.995 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :813601722 2021-01-25 22:49:13.025 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :1521672382 2021-01-25 22:49:23.025 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :2074764264 2021-01-25 22:49:33.025 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :2018582614 2021-01-25 22:49:43.025 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :2026275027 2021-01-25 22:49:53.025 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :2039079331 2021-01-25 22:50:03.025 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :1640327241 2021-01-25 22:50:13.025 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :2145863789 2021-01-25 22:50:23.025 [log-file-generator-0] INFO com.lubiao.test.ClientTest - receiveBytesPerSecond is :2109318682

It is so unstable，but i want a stable performance no matter how large the size is，because i can‘t control the data received

Attache： grpc-test.zip

[ejona86]

I found the reason why OnReadyHandler can not be called，because my dump method in TxnStreamRpcServer.java is running in loop and not return，so the method named “run” in SerializingExecutor.java is blocked…

Not blocked, but enqueued, but yes. The various StreamObserver callbacks are not-thread-safe, so if one is still running the other can’t be. This is to prevent your code from having to be thread-safe. You have to return to receive future callbacks.

It is so unstable，but i want a stable performance no matter how large the size is，because i can‘t control the data received

The system performance will vary based on size. It seems you should just consider 100 MBps throughput as the minimum. I’ll note that I tend to estimate 1 GBps per plaintext connection as the rough upper limit (obviously varies depending on your hardware). That applies even for actual networks, as long as the network itself has such capacity. Multiple connections (via a client-side LB or using multiple channels) expands that further, as each connection is limited to a core.

I agree it is weird that 160 KB is slower than 80 KB. But it is very time-intensive to investigate benchmarks, as the benchmark itself greatly impacts results because most benchmarks don’t actually measure what people think they measure. It isn’t really worth the time to investigate this benchmark for the “it is strange” factor. If you would like to profile and look into it, you are welcome to, but there’s a lot of factors that would contribute.

The benchmark creates new channels every iteration; that by itself will lead to variability as is something we discourage (unless you are contacting different backends each time). New channels take some time to get up-to-speed as they dynamically adjust some of their settings. Minor perturbations can dramatically influence that. I’d just suggest you try setting nettyChannelBuilder.initialFlowControlWindow(1024*1024) (which is generally an overkill; the default is 64 KB) and see what result you get.

Running our own benchmarks, which are carefully designed such that we have a reasonable idea of what they tell us, I don’t see the same behavior as you. But it also wasn’t clear what your test setup was, like what sort of connection was between your client and server.

These results were with 1 fork, 10 warmups, 10 trials, 1s each. Client and server on same machine, a laptop with i7-8650U 1.90GHz (4 core; 8 hyper-threaded). I’ll note that I saw clear signs of temperature clamping as warmups ran faster than the actual trials, but this impacted each test equally and these aren’t meant to be all that scientific of results. An “op” in the score is a “byte”, so the scores are bytes per second. Heap size and other things can impact the results, but I don’t care about such details for this.

Message Size  Benchmark                                        (direct)  (transport)   Mode  Cnt          Score          Error  Units
   70 * 1024  TransportBenchmark.streamingCallsByteThroughput     false        NETTY  thrpt   10  1372584897.944 ± 47758374.764  ops/s
   80 * 1024  TransportBenchmark.streamingCallsByteThroughput     false        NETTY  thrpt   10  1329110881.554 ± 93482992.451  ops/s
  160 * 1024  TransportBenchmark.streamingCallsByteThroughput     false        NETTY  thrpt   10  1376187062.678 ± 17934374.571  ops/s
  240 * 1024  TransportBenchmark.streamingCallsByteThroughput     false        NETTY  thrpt   10  1208582782.490 ± 37572657.580  ops/s
  512 * 1024  TransportBenchmark.streamingCallsByteThroughput     false        NETTY  thrpt   10  1014387923.479 ± 21440324.653  ops/s
 1024 * 1024  TransportBenchmark.streamingCallsByteThroughput     false        NETTY  thrpt   10   969348435.651 ± 19148980.568  ops/s
 4096 * 1024  TransportBenchmark.streamingCallsByteThroughput     false        NETTY  thrpt   10   905026766.749 ± 41871586.492  ops/s

[lulu2panpan]

Yes

1. I moved my business code from grpc-executor-thread to my custom thread
2. Add a parameter to control the message size per onNext The Result is OK