Add watch on volumes for TLSContext

Background/Summary

The TLSContext allows for specifying the cert-chain, the ca-cert-chain, and the key for TLS. It only does this, however, when it is first initialized and does not update when the volumes change. This is most evident with Istio’s worker certificate rotation, where the /etc/istio-certs volume is updated, but the TLSContext is not.

Issue/Request

TLSContext needs to update whenever the volumes it’s pointing at are changed.

Replicator

Kubernetes Cluster: Kubeception 1.17 Ambassador Version: 1.6.2 via Helm

1. Follow the Istio how-to (excluding Prometheus and Distributed Tracing), but in the environment variables of the istio-proxy container, add

- name: SECRET_TTL
  value: "0h5m0s"

1. wait 5 mins to see 2020-08-24T16:13:24.532920Z info sds resource:default pushed key/cert pair to proxy in kubectl logs -n ambassador {{AMBASSADOR_POD}} -c istio-proxy.

2. try to get to a backend service in the mesh: curl -v https://{{AMBASSADOR_HOST}}/backend/

< HTTP/1.1 503 Service Unavailable
< content-length: 91
< content-type: text/plain
< date: Mon, 24 Aug 2020 16:14:35 GMT
< server: envoy
< 
* Connection #0 to host {{AMBASSADOR_HOST}} left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure* Closing connection 0

1. Restart the pod kubectl rollout restart deploy -n ambassador ambassador

2. Try again curl -v https://{{AMBASSADOR_HOST}}/backend/

< HTTP/1.1 200 OK
< content-type: application/json
< date: Mon, 24 Aug 2020 17:05:58 GMT
< content-length: 168
< x-envoy-upstream-service-time: 4
< server: envoy
< x-envoy-decorator-operation: quote.default.svc.cluster.local:80/*
< 
{
    "server": "quintessential-passionfruit-t11jxcdt",
    "quote": "A principal idea is omnipresent, much like candy.",
    "time": "2020-08-24T17:05:58.616336259Z"
* Connection #0 to host {{AMBASSADOR_HOST}} left intact
}* Closing connection 0

See also #2692 and #2587

[valeneiko]

I am facing the same issue. The certificate is renewed by Istio, but is not picked up by Ambassador.

• Kubernetes 1.18
• Istio 1.7.3
• Ambassador 1.8.1

I have tried setting SECRET_TTL to 5m and 24h (default), both fail until ambassador is restarted.

By default Istio rotates certificates at half-time, but at most once every 5 minutes (all configurable).

• With TTL of 24h the certificates are rotated every 12h but are valid for 24h. (app becomes unavailable after 24h)
• With TTL of 5m, certificates are rotated every 5min and are valid for 5 min. (app becomes unavailable after 5min)

[Arttii]

This is interesting, that the secret does not get created. Is it just the internal cached k8s client value that Ambassador is able to use in the end? Seems like a weird undocumented behaviour if this is the case. My problem was, that I was looking for the actual secret in the namespace.