gRPC load balancing doesn't appear to work

From @ryanolson

@richarddli by just increasing the replicas in the example i get:

root@7aad1319fc7d:/devel# python greeter_client.py Greeter client received: Hello, you from host=grpc-greet-f468d5c7d-6xdvh! Greeter client received: Hello, you from host=grpc-greet-f468d5c7d-6xdvh! Greeter client received: Hello, you from host=grpc-greet-f468d5c7d-6xdvh! ... root@7aad1319fc7d:/devel# python greeter_client.py Greeter client received: Hello, you from host=grpc-greet-f468d5c7d-qzlt9! Greeter client received: Hello, you from host=grpc-greet-f468d5c7d-qzlt9! Greeter client received: Hello, you from host=grpc-greet-f468d5c7d-qzlt9! ... root@7aad1319fc7d:/devel# python greeter_client.py Greeter client received: Hello, you from host=grpc-greet-f468d5c7d-6xdvh! Greeter client received: Hello, you from host=grpc-greet-f468d5c7d-6xdvh! Greeter client received: Hello, you from host=grpc-greet-f468d5c7d-6xdvh! ...
as you can see, i get load-balancing per-invocation of a client; essentially L4 load-balancing.

however, i’m looking for 1 client, i.e. 1 grpc stub, to load-balance over all backend services - L7 note: i had to modify the greeter_server to output it’s HOSTNAME in the response; similarly, the client makes 10 repeated calls using the same stub. (I shortened the output above).

Also reported by Jean-Christophe Baey @jcbaey_twitter

[plombardi89]

So we think we understand the problem and have a workaround:

Current Theory

When you create a Kubernetes v1.Service object you are creating a virtual host representing an iptables rule that randomly selects Pod addresses for you (see: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies).

Envoy asynchronously queries DNS and only ever receives a single IP address from the DNS server (the Kubernetes v1.Service objects cluster IP). The current working assumption is that because Envoy only ever sees a single address and it is never changing that a single persistent connection is established to a backend Pod which is why traffic does not get load balanced.

We need to perform more testing, but that will take some time. In the meantime we have a simple workaround detailed below:

Workaround

The workaround to this problem is use a a headless Kubernetes service. A headless Kubernetes service creates a DNS A record that points to the individual Pod IP addresses for a service. When Envoy performs one of its asynchronous DNS queries to populate its internal concept of a cluster then it receives <X> records from DNS where <X> represents the number running pods.

You can create a headless service using the clusterIP: None attribute on a Kubernetes v1.Service, for example:

---
apiVersion: v1
kind: Service
metadata:
  name: grpc-basic
  namespace: stable
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind: Mapping
      name: grpc-basic-stable
      grpc: true
      prefix: /helloworld.Greeter/
      rewrite: /helloworld.Greeter/
      service: grpc-basic:50051
      ---
      apiVersion: ambassador/v0
      kind: Mapping
      name: grpc-basic-stable-grpcreflect
      grpc: true
      prefix: /grpc.
      rewrite: /grpc.
      service: grpc-basic:50051
spec:
  type: ClusterIP
  clusterIP: None
  ports:
  - name: grpc
    port: 50051
    targetPort: grpc
  selector:
    app: grpc-basic

More information about headless services can be found in the Kubernetes docs: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services

[richarddli]

Closing this as it’s resolved with endpoint routing