v0.50.1 AuthService drops all but one `set-cookie` response header

Describe the bug When AuthService returns a non-200 response to Ambassador, only one set-cookie header can be sent back to the client, all other set-cookie headers are stripped. All cookies have the same domain, max_age, etc.

To Reproduce Steps to reproduce the behavior:

1. Setup a v1 (0.50.1) AuthService that adds more than one set-cookie header on auth responses
2. Send a non-200 response (302 in this case so that the request does not continue upstream) from AuthService with >1 set-cookie header on response
3. Verify all set-cookie headers are present on response from AuthService as the response passes back through Ambassador; there should be >1 set-cookie header on the response from the AuthService and only 1 set-cookie header on the response that Ambassador filters and sends back to the client

Expected behavior We expect all set-cookie headers to be returned by Ambassador when the AuthService returns a response. Rolling the service back to Ambassador v0.40.2 results in the expected behavior.

Versions (please complete the following information):

• Ambassador: [e.g. 0.32.1]
• Kubernetes environment [e.g. Minikube, bare metal, Google Kubernetes Engine]
• Version [e.g. 1.8.1]

Additional context Here is logging we captured of the issue. Notice that x-request-destination is preserved but session is removed in the final response.

[2019-02-07 20:41:58.736][000059][debug][http] [source/common/http/async_client_impl.cc:96] async http request response headers (end_stream=false):
':status', '302'
'server', 'nginx/1.15.0'
'date', 'Thu, 07 Feb 2019 20:41:58 GMT'
'content-type', 'text/html; charset=utf-8'
'content-length', '941'
'connection', 'keep-alive'
'location', 'https://dev-syapse.auth0.com/authorize?response_type=code&client_id=Y4EF4s3mw3IvLKFUUggL9Xgz64g0pH6h&redirect_uri=https%3A%2F%2Fambassador.dev.syapse.com%2Fauthz%2Fv1%2Fauth0%2Fcomplete&scope=openid+profile+email+user_metadata+app_metadata&state=u63YpJcjSweLIFyOQpqfFL1ZvRSLxI&audience=https%3A%2F%2Fdev-syapse.auth0.com%2Fuserinfo&prompt=none'
'set-cookie', 'x-request-destination=https://oncology-web-2.dev.syapse.com/; Domain=.syapse.com; Path=/'
'vary', 'Cookie'
'set-cookie', 'session=.eJyrVopPLC3JMACTOZlJ8cmJOTlJicnZ8UpWShklJQXFVvr6iblJicXFiSn5RXopqWV6xZWJBcWpesn5ufogXVX6ZYZghoE-UKggJ7UkVUkH3djiksSSVJCZpWbGkQVeyVnB5ak-nm6V_oEFhWluPoZRZUHBPhWeSrUAJe00QQ.XFyYFg.8BzD07RYRXiRz8Iz2s-9U3S3XUg; Domain=.syapse.com; HttpOnly; Path=/'
'x-content-type-options', 'nosniff'
'x-frame-options', 'SAMEORIGIN'
'x-xss-protection', '1; mode=block'
'strict-transport-security', 'max-age=15768000; includeSubDomains'
'x-envoy-upstream-service-time', '27'

[2019-02-07 20:41:58.740][000059][debug][client] [source/common/http/codec_client.cc:95] [C141708] response complete
[2019-02-07 20:41:58.740][000059][debug][filter] [source/extensions/filters/http/ext_authz/ext_authz.cc:177] [C141707][S17486080943075874799] ext_authz rejected the request
[2019-02-07 20:41:58.740][000059][debug][http] [source/common/http/conn_manager_impl.cc:1096] [C141707][S17486080943075874799] encoding headers via codec (end_stream=false):
':status', '302'
'content-length', '941'
'content-type', 'text/plain'
'location', 'https://dev-syapse.auth0.com/authorize?response_type=code&client_id=Y4EF4s3mw3IvLKFUUggL9Xgz64g0pH6h&redirect_uri=https%3A%2F%2Fambassador.dev.syapse.com%2Fauthz%2Fv1%2Fauth0%2Fcomplete&scope=openid+profile+email+user_metadata+app_metadata&state=u63YpJcjSweLIFyOQpqfFL1ZvRSLxI&audience=https%3A%2F%2Fdev-syapse.auth0.com%2Fuserinfo&prompt=none'
'set-cookie', 'x-request-destination=https://oncology-web-2.dev.syapse.com/; Domain=.syapse.com; Path=/'
'date', 'Thu, 07 Feb 2019 20:41:58 GMT'
'server', 'envoy'

[2019-02-07 20:41:58.741][000059][debug][pool] [source/common/http/http1/conn_pool.cc:209] [C141708] response complete
[2019-02-07 20:41:58.741][000059][debug][pool] [source/common/http/http1/conn_pool.cc:247] [C141708] moving to ready[2019-02-07 20:41:58.736][000059][debug][http] [source/common/http/async_client_impl.cc:96] async http request response headers (end_stream=false):
':status', '302'
'server', 'nginx/1.15.0'
'date', 'Thu, 07 Feb 2019 20:41:58 GMT'
'content-type', 'text/html; charset=utf-8'
'content-length', '941'
'connection', 'keep-alive'
'location', 'https://dev-syapse.auth0.com/authorize?response_type=code&client_id=Y4EF4s3mw3IvLKFUUggL9Xgz64g0pH6h&redirect_uri=https%3A%2F%2Fambassador.dev.syapse.com%2Fauthz%2Fv1%2Fauth0%2Fcomplete&scope=openid+profile+email+user_metadata+app_metadata&state=u63YpJcjSweLIFyOQpqfFL1ZvRSLxI&audience=https%3A%2F%2Fdev-syapse.auth0.com%2Fuserinfo&prompt=none'
'set-cookie', 'x-request-destination=https://oncology-web-2.dev.syapse.com/; Domain=.syapse.com; Path=/'
'vary', 'Cookie'
'set-cookie', 'session=.eJyrVopPLC3JMACTOZlJ8cmJOTlJicnZ8UpWShklJQXFVvr6iblJicXFiSn5RXopqWV6xZWJBcWpesn5ufogXVX6ZYZghoE-UKggJ7UkVUkH3djiksSSVJCZpWbGkQVeyVnB5ak-nm6V_oEFhWluPoZRZUHBPhWeSrUAJe00QQ.XFyYFg.8BzD07RYRXiRz8Iz2s-9U3S3XUg; Domain=.syapse.com; HttpOnly; Path=/'
'x-content-type-options', 'nosniff'
'x-frame-options', 'SAMEORIGIN'
'x-xss-protection', '1; mode=block'
'strict-transport-security', 'max-age=15768000; includeSubDomains'
'x-envoy-upstream-service-time', '27'

[2019-02-07 20:41:58.740][000059][debug][client] [source/common/http/codec_client.cc:95] [C141708] response complete
[2019-02-07 20:41:58.740][000059][debug][filter] [source/extensions/filters/http/ext_authz/ext_authz.cc:177] [C141707][S17486080943075874799] ext_authz rejected the request
[2019-02-07 20:41:58.740][000059][debug][http] [source/common/http/conn_manager_impl.cc:1096] [C141707][S17486080943075874799] encoding headers via codec (end_stream=false):
':status', '302'
'content-length', '941'
'content-type', 'text/plain'
'location', 'https://dev-syapse.auth0.com/authorize?response_type=code&client_id=Y4EF4s3mw3IvLKFUUggL9Xgz64g0pH6h&redirect_uri=https%3A%2F%2Fambassador.dev.syapse.com%2Fauthz%2Fv1%2Fauth0%2Fcomplete&scope=openid+profile+email+user_metadata+app_metadata&state=u63YpJcjSweLIFyOQpqfFL1ZvRSLxI&audience=https%3A%2F%2Fdev-syapse.auth0.com%2Fuserinfo&prompt=none'
'set-cookie', 'x-request-destination=https://oncology-web-2.dev.syapse.com/; Domain=.syapse.com; Path=/'
'date', 'Thu, 07 Feb 2019 20:41:58 GMT'
'server', 'envoy'

[2019-02-07 20:41:58.741][000059][debug][pool] [source/common/http/http1/conn_pool.cc:209] [C141708] response complete
[2019-02-07 20:41:58.741][000059][debug][pool] [source/common/http/http1/conn_pool.cc:247] [C141708] moving to ready

Here’s the AuthService annotation that we are using:

  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: AuthService
      name: authentication
      auth_service: {auth-service-url-and-port}
      path_prefix: "/v1/validate"
      allowed_authorization_headers:
      - "set-cookie"
      - "session"
      ---
      apiVersion: ambassador/v1
      kind: Mapping
      name: authz_mapping
      prefix: /authz/
      service: {auth-service-url-and-port}
      tls: true

We also verified this behavior against normal non-AuthService request/response flows, and did not see Ambassador filtering any response headers.

[ankitpec72]

Any update on this issue? When it going to be fixed? We can not update to 50 version because of this issue.

[woraphol-j]

Hi @gsagula there is only one set-cookie in the request to the echo server

"set-cookie": [
        "baz=baz"
],

I think the flow is (Please correct me if I am wrong)

1. A request reaches Ambassador.
2. Ambassador passes it on to the AuthService with requested-cookie: foo, bar, baz header.
3. AuthService generates a response with 3 set-cookie headers and send back to AuthService.
4. Ambassador receives a response from the AuthService with 3 set-cookie headers.
5. Ambassador passes this on to the echo server, however, it only passes 1 set-cookie (the last one) instead of 3 set-cookie headers. Note that it still passes requested-cookie: foo, bar, baz header.
6. Because of the passed requested-cookie: foo, bar, baz header, Ambassador still returns the correct result in the end with 3 set-cookie response headers.