Online Boutique does not work on IPv6 single stack kubernetes cluster

Describe the bug

The Online Boutique page shows Internal Server Error. Online Boutique is deployed to IPv6 single stack cluster.

To Reproduce

1. Create IPv6 single stack cluster with Calico.
2. Deploy Online Boutique and enable port-forwaring.
3. Access to the deployed service and the page shows Internal Server Error.

Logs

Logs for frontend

{"message":"Tracing disabled.","severity":"info","timestamp":"2022-11-01T10:11:21.144638627Z"}
{"message":"Profiling disabled.","severity":"info","timestamp":"2022-11-01T10:11:21.144921177Z"}
{"message":"starting server on :8080","severity":"info","timestamp":"2022-11-01T10:11:21.146948308Z"}
{"message":"Failed to fetch the name of the cluster in which the pod is runningGet \"http://169.254.169.254/computeMetadata/v1/instance/attributes/cluster-name\": dial tcp 169.254.169.254:80: connect: network is unreachable","severity":"error","timestamp":"2022-11-01T10:11:21.150043979Z"}
{"message":"Failed to fetch the Zone of the node where the pod is scheduledGet \"http://169.254.169.254/computeMetadata/v1/instance/zone\": dial tcp 169.254.169.254:80: connect: network is unreachable","severity":"error","timestamp":"2022-11-01T10:11:21.150230692Z"}
{"cluster":"","hostname":"frontend-7966dc59fb-n5zkh","message":"Loaded deployment details","severity":"debug","timestamp":"2022-11-01T10:11:21.150254973Z","zone":""}
{"http.req.id":"7b74fd6d-b0b7-4dd9-94a4-88de48f1122c","http.req.method":"GET","http.req.path":"/_healthz","message":"request started","session":"x-liveness-probe","severity":"debug","timestamp":"2022-11-01T10:11:39.410934568Z"}
{"http.req.id":"a23de3f6-3b0c-4236-8268-c8afd36b4088","http.req.method":"GET","http.req.path":"/_healthz","message":"request started","session":"x-readiness-probe","severity":"debug","timestamp":"2022-11-01T10:11:39.41088243Z"}
{"http.req.id":"7b74fd6d-b0b7-4dd9-94a4-88de48f1122c","http.req.method":"GET","http.req.path":"/_healthz","http.resp.bytes":2,"http.resp.status":200,"http.resp.took_ms":1,"message":"request complete","session":"x-liveness-probe","severity":"debug","timestamp":"2022-11-01T10:11:39.412115234Z"}
{"http.req.id":"a23de3f6-3b0c-4236-8268-c8afd36b4088","http.req.method":"GET","http.req.path":"/_healthz","http.resp.bytes":2,"http.resp.status":200,"http.resp.took_ms":1,"message":"request complete","session":"x-readiness-probe","severity":"debug","timestamp":"2022-11-01T10:11:39.412203078Z"}
{"http.req.id":"6c9b5433-fd25-4c00-9825-0d834122d8d0","http.req.method":"GET","http.req.path":"/_healthz","message":"request started","session":"x-liveness-probe","severity":"debug","timestamp":"2022-11-01T10:11:49.409260277Z"}
{"http.req.id":"e19656c0-1cd5-4585-bf22-e1f95aa80bdc","http.req.method":"GET","http.req.path":"/_healthz","message":"request started","session":"x-readiness-probe","severity":"debug","timestamp":"2022-11-01T10:11:49.409328551Z"}
{"http.req.id":"e19656c0-1cd5-4585-bf22-e1f95aa80bdc","http.req.method":"GET","http.req.path":"/_healthz","http.resp.bytes":2,"http.resp.status":200,"http.resp.took_ms":1,"message":"request complete","session":"x-readiness-probe","severity":"debug","timestamp":"2022-11-01T10:11:49.410027627Z"}
{"http.req.id":"6c9b5433-fd25-4c00-9825-0d834122d8d0","http.req.method":"GET","http.req.path":"/_healthz","http.resp.bytes":2,"http.resp.status":200,"http.resp.took_ms":1,"message":"request complete","session":"x-liveness-probe","severity":"debug","timestamp":"2022-11-01T10:11:49.410502244Z"}
{"http.req.id":"6022bf9e-79f9-4cfd-9c24-f74c6840a9e5","http.req.method":"GET","http.req.path":"/","message":"request started","session":"dbd99d36-9590-4583-a910-5bb01b7e7db1","severity":"debug","timestamp":"2022-11-01T10:11:58.038227138Z"}
{"currency":"USD","http.req.id":"6022bf9e-79f9-4cfd-9c24-f74c6840a9e5","http.req.method":"GET","http.req.path":"/","message":"home","session":"dbd99d36-9590-4583-a910-5bb01b7e7db1","severity":"info","timestamp":"2022-11-01T10:11:58.038483621Z"}
{"http.req.id":"d91d0fa8-23d8-4df2-8e02-447c683841ef","http.req.method":"GET","http.req.path":"/","message":"request started","session":"77e22e41-1f84-437e-ae7e-1ce0407fc9c9","severity":"debug","timestamp":"2022-11-01T10:11:59.026430074Z"}
{"currency":"USD","http.req.id":"d91d0fa8-23d8-4df2-8e02-447c683841ef","http.req.method":"GET","http.req.path":"/","message":"home","session":"77e22e41-1f84-437e-ae7e-1ce0407fc9c9","severity":"info","timestamp":"2022-11-01T10:11:59.026580378Z"}
{"error":"could not retrieve currencies: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp [(here is ipv6 address)]:7000: connect: connection refused\"","http.req.id":"6022bf9e-79f9-4cfd-9c24-f74c6840a9e5","http.req.method":"GET","http.req.path":"/","message":"request error","session":"dbd99d36-9590-4583-a910-5bb01b7e7db1","severity":"error","timestamp":"2022-11-01T10:11:59.045619196Z"}

Screenshots

Environment

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.2
Kustomize Version: v4.5.7
Server Version: v1.25.3

Additional context

The cause of this problem seems to be that some services’ gRPC servers are only listening on IPv4. https://github.com/GoogleCloudPlatform/microservices-demo/blob/573f4db73306b779426fa47a7df7ab682188abea/src/currencyservice/server.js#L173 https://github.com/GoogleCloudPlatform/microservices-demo/blob/573f4db73306b779426fa47a7df7ab682188abea/src/emailservice/email_client.py#L26 https://github.com/GoogleCloudPlatform/microservices-demo/blob/573f4db73306b779426fa47a7df7ab682188abea/src/paymentservice/server.js#L67

Exposure

This would affect anyone using IPv6 single stack cluster.

[mathieu-benoit]

Hey @naoki9911, if you have time and still have access to your single-stack IPv6 cluster, could you please give https://github.com/GoogleCloudPlatform/microservices-demo/pull/1340 a try and report back what you will find there? Thanks in advance!

[naoki9911]

Thanks for your replys!

I haven’t assigned a high priority to this issue because I’m guessing IPv6 single-stack isn’t super common yet (among Kubernetes users). Do you know if this a false assumption?

I completely agree with you. I think IPv6 single-stack cluster is very unusual, too. I tried IPv6 single-stack for my experiments and I’m not in a hurry at all about this issue.

This issue just intends to inform you that some services are not supporting IPv6 😃 If you don’t mind, I hope this issue is opened until services completely support IPv6.

what could be the recommendation for the 3 statements you highlighted with 0.0.0.0?

I prefer replacing 0.0.0.0 with [::] to support both IPv4 and IPv6. But I’m not an expert on IPv6 and Kubernetes, so I don’t know if it will not cause any problems. It may allow unintended IPv6 communication in IPv4 dedicated cluster.