Android OOM upon network disconnect

With grpc-java 1.15.0 on Android I am seeing out of memory issues when network connectivity goes away. isReady() indicates unlimited readiness on the outbound stream for a bidi call until Java runs out of memory.

Here is the relevant code, the complete example is at https://github.com/bubenheimer/grpcsinglestreamingcallflush

StreamDataGrpc.newStub(channel).streamData(
        new ClientResponseObserver<Item, Item>() {
            private int outCounter = 0;
            private int inCounter = 0;

            @Override
            public void beforeStart(
                    final ClientCallStreamObserver<Item> requestStream) {
                requestStream.setOnReadyHandler(() -> {
                    while (requestStream.isReady()) {
                        final Item item = Item.newBuilder()
                                .setValue(Integer.toString(outCounter++)).build();
                        requestStream.onNext(item);
                        if (outCounter == 50_000) {
                            requestStream.onCompleted();
                            break;
                        }
                    }
                });
            }
            @Override
            public void onNext(final Item value) {
                ++inCounter;
            }

            @Override
            public void onError(final Throwable t) {
                Log.w(TAG, t);
            }

            @Override
            public void onCompleted() {
                Log.i(TAG, "Call completed after " + inCounter + " Items");
            }
        });

where channel is more persistently defined as

channel = AndroidChannelBuilder
            // localhost server from Android emulator
            .forAddress("10.0.2.2", 8082)
            .usePlaintext()
            .context(getApplicationContext())
            .build();

To trigger the issue I do one complete run of the RPC (50,000 messages upstream & 50,000 messages downstream) with the mobile data network on, then I turn mobile data off and do a second run on the same channel. The second run quickly runs out of memory as grpc-java buffers messages it can’t send. The channel does not see a break in the connection when mobile data switches off.

I expected that isReady() would take care of this problem, but it did not. I have tried different variations, including one where writing to the stream is done from a separate, dedicated thread, with 1ms delays between each write, to not block anything and avoid races, but the outcome was the same.

[ejona86]

Oops. I guess I can respond. Thanks, @bubenheimer, that is helpful and agrees with what we were thinking the problem was. The problem can manifest multiple ways, so I’m not concerned about the difference between emulator and actual phone.

[bubenheimer]

@ejona86 and others: I wrote this test in the last days, so not a known regression.

Just to clarify, the code from the logs uses 100,000 messages instead of the 50,000 from what I posted, and it uses OkHttpChannelBuilder, otherwise it’s the same code. The log shows two calls, the first one with mobile data on (“Call finished” on line 22 marks the end), and the second one after I disabled mobile data.

The code runs on the Android emulator (I tried with Android API levels 28 and 23) on my Mac and talks to a simple grpc-java server on the Mac via that 10.0.2.2 IP that identifies the host system to the emulator. The grpc-java server uses Netty and all defaults. The emulator pretends that data goes through a mobile data connection, so it must be doing something special; I don’t know the details, or what buffering & proxying may happen in-between.

I also ran this test on a real Android phone (talking to a remote server I set up), and there the broken connection is instantly seen once I turn mobile data off, so it will refuse to send any messages, no OOM.