Emissary sends gzip encoded payloads to backend services

Describe the bug

If we send certain payloads to Emissary, Emissary seems to be gzip encoding the payload and forwarding it to the upstream service. Most often, we send Content-Types application/json, text/html (and a few others) which all seem to be gzip encoded before forwarding upstream (which is in-line with what the docs say the default behaviour is).

Maybe we have something mis-configured, but I don’t think Emissary should gzip encode the payload before sending to the backend service?

To Reproduce

1. Deploy Ambassador to your cluster
2. Setup a Module like this:

---
apiVersion: getambassador.io/v2
kind: Module
metadata:
  name: ambassador
  namespace: ambassador
spec:
  ambassador_id: gzip-test
  config:
    gzip:
      enabled: true
    use_remote_address: true
    xff_num_trusted_hops: 1
    server_name: "gzip-test"
[ ... ]

1. Setup a service and mapping to pass the payload from Emissary to the backend.
2. Send a curl request like this, notice it is NOT gzip encoded:

curl -i -vv -X POST curl --silent -o /dev/null https://my-ingress.example.com -H "Content-Type: application/json" -d '{"foo":"bar"}'

Backend service request:

{
  body: '{"foo":"bar"}',
  headers: Headers {
    accept: "*/*",
    "content-type": "application/json",
  },
  [ ... ]
}

1. Send a curl request like this, with a little larger payload, notice the payload body is gzip encoded:

curl -i -vv -X POST curl --silent -o /dev/null https://my-ingress.example.com -H "Content-Type: application/json" -d '{"deadbeef":{"foo":150,"bar":100},"ea7beef":242,"foobaz":240}'

Backend service request:

{
  body: '\x1f�\b\x00\x00\x00\x00\x00\x00\x03�V*-N-R��V [ ... ]',
  headers: Headers {
    accept: "*/*",
    "accept-encoding": "gzip",
    "content-encoding": "gzip",
    "content-type": "application/json",
    "transfer-encoding": "chunked",
  },
  [ ... ]
}

Expected behavior

My understanding is that the gzip encode/decode should only happen between the client <-> Emissary, and Emissary should not forward that gzip encoded content (or gzip encode the data itself), and forward to the backend service.

Versions

• Ambassador - Helm Chart: emissary-ingress-7.1.2-ea
• Kubernetes - v1.19.9

Additional context

As a workaround we can disable gzip encoding in the Emissary config (not ideal), or put in a app middleware of some sort to decompress the content as it’s received from Emissary (less than ideal), but we weren’t sure if we had something mis-configured that may be causing the behaviour we are seeing?

[kmott]

Another thing I noticed–with a minimal Module config like this:

apiVersion: getambassador.io/v2
kind: Module
metadata:
  name: ambassador
  namespace: ambassador
spec:
  ambassador_id: gzip-test
  config:
    gzip:
      enabled: false
[ ... ]

Gzip encoding is still enabled in the deployment after the change. Not sure if I missed something in order for the change to take effect?

In order to workaround the issue, I had to explicitly set the content_type list to a single bogus entry, so it wouldn’t be triggered for standard Content-Type’s like application/json, etc.

apiVersion: getambassador.io/v2
kind: Module
metadata:
  name: ambassador
  namespace: ambassador
spec:
  ambassador_id: gzip-test
  config:
    gzip:
      enabled: true
      content_type:
        - example/enable-gzip-encoding-to-service
[ ... ]

[EvertonSA]

@khussey can you help me out pointing where was this fixed? I would be glad to see if I can contribute to this repository