set-cookie seems to override the cookie header instead of appending the new cookies

I’m trying to test my website where users authenticate via session cookie without having to automate the whole manual logging process using the Chrome debugger solution suggested here https://github.com/GoogleChrome/lighthouse/blob/master/docs/readme.md#testing-on-a-site-with-authentication, so I’m trying to directly set the cookies using the --extra-headers flag.

Provide the steps to reproduce

1. Run LH on an authenticated via cookie website: lighthouse https://authenticated-website.com --chrome-flags="--headless" --extra-headers=./headers.json

The content of headers.json is as follows:

{ "Cookie": "session_id=whatever-session-id" }

What is the current behavior?

When lighthouse accesses the website, there’s a call to an authentication API that returns a set-cookie header with a csrf token. From that request onwards, the user is unauthenticated. All of the requests are sending the cookie with the csrf token (returned via set-cookie) but are completely disregarding the headers I set in the beginning, overriding the original “Cookie” header instead of appending the value. I enabled extra logging and see on the network responses that the headers received are not the expected ones:

{ "method": "Network.responseReceived", "params": { "requestId": "1000019523.17", "loaderId": "5D025634B8EEDC4CF37CCC23430142BB", "timestamp": 52153.046111, "type": "Fetch", "response": { "url": "https://authenticated-website.com/whatever.json", "status": 200, "statusText": "", "headers": { "date": "Tue, 09 Oct 2018 16:44:06 GMT", "content-encoding": "gzip", "x-content-type-options": "nosniff", "x-permitted-cross-domain-policies": "none", "status": "200", "vary": "Accept-Encoding", "content-length": "219", "x-xss-protection": "1; mode=block", "pragma": "no-cache", "last-modified": "Tue, 09 Oct 2018 15:52:20 GMT", "server": "none", "x-frame-options": "SAMEORIGIN", "x-download-options": "noopen", "strict-transport-security": "max-age=31536000; includeSubdomains; preload\nmax-age=31536000; includeSubdomains;", "content-type": "application/json", "region": "EU", "cache-control": "max-age=0, no-cache, no-store, must-revalidate", "accept-ranges": "bytes", "x-robots-tag": "none", "expires": "Wed, 11 Jan 1984 05:00:00 GMT" }, "mimeType": "application/json", "requestHeaders": { ":path": "/whatever.json", "accept-encoding": "gzip, deflate", "user-agent": "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/71.0.3559.0 Mobile Safari/537.36", "accept": "*/*", "referer": "https://authenticated-website.com", ":authority": "authenticated-website.com", "cookie": "csrf_token=whatever-csrf-token", ":scheme": "https", ":method": "GET" }, "connectionReused": true, "connectionId": 9 ....

What is the expected behavior?

Cookie returned in set-cookie should be appended, making the headers from that point onwards look like this:

{ "Cookie": "session_id=whatever-session-id; csrf_token=whatever-session-token" }

Environment Information

• Affected Channels: CLI
• Lighthouse version: 3.2.1
• Node.js version: 9.5.0
• Operating System: Mac OS X (also reproduced on Ubuntu 18.04.1)

I’m running Chrome 69.0.3497.100.

Related issues https://github.com/GoogleChrome/lighthouse/issues/1418

[RynatSibahatau]

I think this issue breaks the possibility of using lighthouse on any enterprise level project on CLI mode (i.e in CI) and I add +1 for the criticality of this issue. I came across exactly the same behavior: any Set-Cookie response from backend overwrites the Cookie header from extra-headers.

1. lighthouse http://localhost/ --extra-headers "{\"Cookie\":\"test=true\"}"
2. Request header: Cookie: test=true Response header: Set-Cookie: locale=en;
3. All subsequent requests headers: Actual request headers : Cookie: locale=en Excpeted request headers: Cookie: test=true; locale=en

So I decided to play with the code https://github.com/RynatSibahatau/lighthouse/commit/5a8b8ceb9fbd1ac8834d33116958ef24b2dcf6f7 and have introduced a new CLI parameter:

lighthouse http://localhost/ --extra-cookies "[{\"name\":\"test\", \"value\": \"true\", \"url\": \"http://localhost\"}]"

This command works as expected behaviour.

If you support me to have this parameter in the lighthouse’s master branch, I can do extra polishing/add extra tests and create a PR.

Thank you!

[patrickhulce]

Thanks @RynatSibahatau that looks like a great starting point for a PR already! Personally, I’m fairly convinced this is a worthwhile use case to support in aid of #1418 without resorting to #3837, especially in CI cases. I’d be happy to assist with your feature PR and make the case to the rest of the team.

The only issues I think I currently see is how this behavior is intended to interact with other cookie settings and the specific JSON format expected, but we can discuss those details in the PR if you wish to proceed 😃