[Help wanted] io.grpc.okhttp.OkHttpClientTransportTest fails in Mac and GitHub Actions

In master, I cannot build :grpc-okhttp module because OkHttpClientTransportTest fails in my Mac (Mac OS 11.2.3) and GitHub Actions (link).

suztomo-macbookpro44% git log -1                                                                  
commit d4fa0ecc07495097453b0a2848765f076b9e714c (HEAD -> master, origin/master, origin/HEAD)
Author: Chengyuan Zhang <[e-mail]>
Date:   Mon Apr 12 14:55:31 2021 -0700
...(omit)...
suztomo-macbookpro44% cat gradle.properties 
skipCodegen=true
skipAndroid=true
suztomo-macbookpro44%  ./gradlew cleanTest :grpc-okhttp:test --tests=io.grpc.okhttp.OkHttpClientTransportTest
*** Skipping the build of codegen and compilation of proto files because skipCodegen=true
  * Skipping the build of Android projects because skipAndroid=true

> Task :grpc-okhttp:test

io.grpc.okhttp.OkHttpClientTransportTest > proxy_200 FAILED
    org.junit.runners.model.TestTimedOutException: test timed out after 10 seconds
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at io.grpc.okhttp.OkHttpClientTransportTest.proxy_200(OkHttpClientTransportTest.java:1876)

io.grpc.okhttp.OkHttpClientTransportTest > proxy_500 FAILED
    org.junit.runners.model.TestTimedOutException: test timed out after 10 seconds
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at io.grpc.okhttp.OkHttpClientTransportTest.proxy_500(OkHttpClientTransportTest.java:1933)

io.grpc.okhttp.OkHttpClientTransportTest > proxy_immediateServerClose FAILED
    org.junit.runners.model.TestTimedOutException: test timed out after 10 seconds
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at io.grpc.okhttp.OkHttpClientTransportTest.proxy_immediateServerClose(OkHttpClientTransportTest.java:1989)

68 tests completed, 3 failed

> Task :grpc-okhttp:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':grpc-okhttp:test'.
> There were failing tests. See the report at: file:///Users/suztomo/grpc-java/okhttp/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 34s
58 actionable tasks: 3 executed, 55 up-to-date

The build works fine in my Debian machine. (I also see this repository has setup Travis CI enabled.)

suztomo@suztomo:~/grpc-java$  ./gradlew cleanTest  :grpc-okhttp:test --tests=io.grpc.okhttp.OkHttpClientTransportTest
*** Skipping the build of codegen and compilation of proto files because skipCodegen=true
  * Skipping the build of Android projects because skipAndroid=true

BUILD SUCCESSFUL in 5s
58 actionable tasks: 2 executed, 56 up-to-date

Therefore I think I’ll need to setup something for the failing environment to make the test pass.

While I continue to dig further, I appreciate if anyone know the prerequisite for OkHttpClientTransportTest.

[suztomo]

@ejona86 new ServerSocket(0, 0, InetAddress.getLoopbackAddress()) solved the problem!

Now proxy_immediateServerClose succeeds:

suztomo-macbookpro44% git diff                                                                                
diff --git a/okhttp/src/test/java/io/grpc/okhttp/OkHttpClientTransportTest.java b/okhttp/src/test/java/io/grpc/okhttp/OkHttpClientTransportTest.java
index 0994bc900..b59aea9c1 100644
--- a/okhttp/src/test/java/io/grpc/okhttp/OkHttpClientTransportTest.java
+++ b/okhttp/src/test/java/io/grpc/okhttp/OkHttpClientTransportTest.java
@@ -1963,7 +1963,7 @@ public class OkHttpClientTransportTest {
 
   @Test
   public void proxy_immediateServerClose() throws Exception {
-    ServerSocket serverSocket = new ServerSocket(0);
+    ServerSocket serverSocket = new ServerSocket(0, 0, InetAddress.getLoopbackAddress());
     InetSocketAddress targetAddress = InetSocketAddress.createUnresolved("theservice", 80);
     clientTransport = new OkHttpClientTransport(
         targetAddress,
@@ -1985,6 +1985,12 @@ public class OkHttpClientTransportTest {
         transportTracer,
         false);
     clientTransport.start(transportListener);
+    org.mockito.Mockito.doAnswer(new org.mockito.stubbing.Answer<Void>() {
+      @Override public Void answer(org.mockito.invocation.InvocationOnMock invocation) {
+        System.out.println(Arrays.asList(invocation.getArguments()));
+        return null;
+      }
+    }).when(transportListener).transportShutdown(any(Status.class));
 
     Socket sock = serverSocket.accept();
     serverSocket.close();
suztomo-macbookpro44%  ./gradlew cleanTest  :grpc-okhttp:test --tests=io.grpc.okhttp.OkHttpClientTransportTest
*** Skipping the build of codegen and compilation of proto files because skipCodegen=true
  * Skipping the build of Android projects because skipAndroid=true

> Task :grpc-okhttp:test

io.grpc.okhttp.OkHttpClientTransportTest > proxy_200 FAILED
    org.junit.runners.model.TestTimedOutException: test timed out after 10 seconds
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at io.grpc.okhttp.OkHttpClientTransportTest.proxy_200(OkHttpClientTransportTest.java:1876)

io.grpc.okhttp.OkHttpClientTransportTest > proxy_500 FAILED
    org.junit.runners.model.TestTimedOutException: test timed out after 10 seconds
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at io.grpc.okhttp.OkHttpClientTransportTest.proxy_500(OkHttpClientTransportTest.java:1933)

68 tests completed, 2 failed

> Task :grpc-okhttp:test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':grpc-okhttp:test'.
> There were failing tests. See the report at: file:///Users/suztomo/grpc-java/okhttp/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 36s

This time standard output captured the invocation argument:

[Status{code=UNAVAILABLE, description=Failed trying to connect with proxy, cause=java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:210)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at okio.Okio$2.read(Okio.java:140)
	at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
	at io.grpc.okhttp.OkHttpClientTransport.readUtf8LineStrictUnbuffered(OkHttpClientTransport.java:731)
	at io.grpc.okhttp.OkHttpClientTransport.createHttpProxySocket(OkHttpClientTransport.java:680)
	at io.grpc.okhttp.OkHttpClientTransport.access$1100(OkHttpClientTransport.java:109)
	at io.grpc.okhttp.OkHttpClientTransport$4.run(OkHttpClientTransport.java:559)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
}]

@ejona86 @sanjaypujare Thank you for your help in troubleshooting!

My Memo

Why ServerSocket(0) does not work? ServerSocket(0) is ServerSocket(port: 0, backlog: 50, bindAddr: null).

(ServerSocket serverSocket = new ServerSocket(0, 50, loopbackAddress); also works good.)

https://docs.oracle.com/javase/8/docs/api/java/net/ServerSocket.html says:

If bindAddr is null, it will default accepting connections on any/all local addresses.

InetAddress.getLoopbackAddress() returns Inet4Address@2726 localhost/127.0.0.1.

How about ServerSocket(0, 50, null)?

This timed out… (Of course ServerSocket(0) is the alias to ServerSocket(0, 50, null))

ServerSocket serverSocket = new ServerSocket(0, 50, null);

How about 0.0.0.0?

This timed out…

    InetAddress ipAddrAny = InetAddress.getByAddress(new byte[]{0, 0, 0, 0});
    ServerSocket serverSocket = new ServerSocket(0, 50, ipAddrAny);

How about 127.0.0.1?

This worked!

    InetAddress localhost = InetAddress.getByAddress(new byte[]{127, 0, 0, 1});
    ServerSocket serverSocket = new ServerSocket(0, 50, localhost);

Any security system that blocks listening on 0.0.0.0 port 80?

In my Mac, nc command can listen on 0.0.0.0 port 80.

What’s the difference in the client code?

when I use InetAddress.getLoopbackAddress(), then the client’s SocketFactory.createSocket takes 127.0.0.1.

In the bad case, the client was trying to create a socket to 0.0.0.0 (screenshot). As per https://serverfault.com/a/300408 (which mentions https://tools.ietf.org/html/rfc1122#page-30 ), address 0.0.0.0 must not be sent.

How about specifying proxyAddress with InetAddress.getLoopbackAddress?

The code below also worked and clarifies the difference between the bad case and good case:

diff --git a/okhttp/src/test/java/io/grpc/okhttp/OkHttpClientTransportTest.java b/okhttp/src/test/java/io/grpc/okhttp/OkHttpClientTransportTest.java
index 0994bc900..359675374 100644
--- a/okhttp/src/test/java/io/grpc/okhttp/OkHttpClientTransportTest.java
+++ b/okhttp/src/test/java/io/grpc/okhttp/OkHttpClientTransportTest.java
@@ -1965,6 +1965,10 @@ public class OkHttpClientTransportTest {
   public void proxy_immediateServerClose() throws Exception {
     ServerSocket serverSocket = new ServerSocket(0);
     InetSocketAddress targetAddress = InetSocketAddress.createUnresolved("theservice", 80);
+    InetSocketAddress proxyAddress = new InetSocketAddress(
+        InetAddress.getLoopbackAddress(),
+        serverSocket.getLocalPort()
+    );
     clientTransport = new OkHttpClientTransport(
         targetAddress,
         "authority",
@@ -1979,7 +1983,7 @@ public class OkHttpClientTransportTest {
         INITIAL_WINDOW_SIZE,
         HttpConnectProxiedSocketAddress.newBuilder()
             .setTargetAddress(targetAddress)
-            .setProxyAddress(serverSocket.getLocalSocketAddress()).build(),
+            .setProxyAddress(proxyAddress).build(),
         tooManyPingsRunnable,
         DEFAULT_MAX_INBOUND_METADATA_SIZE,
         transportTracer,

[sanjaypujare]

My Mac is a normal MacBook Pro. It has wifi connection.

I’m not sure why destination 0.0.0.0 works for certain machines. (In my Debian machine, the destination address 0.0.0.0 works fine for the OkHttp test.)

I found some more info https://superuser.com/questions/1307050/how-does-the-os-resolve-a-request-to-connect-to-0-0-0-0 but again it doesn’t explain the behavior difference between your Mac and my Mac