HTTP Cookies are set but not signaled in the response headers reported by the CDTP

Dear all,

The European data protection authority uses puppeteer to collect data on the use of cookies (see: https://github.com/EU-EDPS/website-evidence-collector)

The software relies on puppeteer and Chromium. While playing with the tools, I found an inconstistancy: HTTP cookies are set, but the respective Set-Cookie headers are not displayed in Chromium or Puppeteer. In Firefox’ developer tools I can see the cookies.

Reproducing the issue

Step 1 (Check with curl)

curl -I "https://logs1407.xiti.com/hit.xiti?s=577958&ts=1580984520392&vtag=5.9.0&pta…S&idp=1122001791222&jv=0&at=0&p=home::homepage&s2=4&vrn=1&x7=EN&x8=No&ref=";

HTTP/2 302 
date: Thu, 06 Feb 2020 15:07:53 GMT
content-type: text/html; charset=utf-8
location: /hit.xiti?s=577958&ts=1580984520392&vtag=5.9.0&pta%e2%80%a6S&idp=1122001791222&jv=0&at=0&p=home::homepage&s2=4&vrn=1&x7=EN&x8=No&ref=&Rdt=On
p3p: CP="NON DSP COR CURa PSA PSD OUR BUS NAV STA"
set-cookie: idrxvr=AFD16D12-ED88-41A0-896E-29FBA5037DB7; Path=/; Domain=xiti.com; Expires=Sat, 06 Mar 2021 15:07:53 GMT; HttpOnly
set-cookie: atidx=AFD16D12-ED88-41A0-896E-29FBA5037DB7; Path=/; Domain=xiti.com; Expires=Sat, 06 Mar 2021 15:07:53 GMT; HttpOnly; Secure
set-cookie: atid=AFD16D12-ED88-41A0-896E-29FBA5037DB7; Path=/; Domain=xiti.com; Expires=Sat, 06 Mar 2021 15:07:53 GMT; HttpOnly; Secure; SameSite=None
strict-transport-security: max-age=3600

Step 2 (Check with Chromium)

• open Chromium and its developer tools, tab Network
• go to https://multimedia.europarl.europa.eu/en/home
• filter the traffic for e.g. xiti.com and notice that the response does not show the Set-Cookie headers as curl does.

Step 3 (Check with Puppeteer)

MWE

#!/usr/bin/env node
// jshint esversion: 8

const puppeteer = require('puppeteer');

// const url = "https://logs1407.xiti.com/hit.xiti?s=577958&ts=1580984520392&vtag=5.9.0&pta…S&idp=1122001791222&jv=0&at=0&p=home::homepage&s2=4&vrn=1&x7=EN&x8=No&ref=";
const url = "https://multimedia.europarl.europa.eu/en/home";

(async () => {
  const browser = await puppeteer.launch();
  const page = (await browser.pages())[0];

  // record all requested hosts
  await page.on('response', (response) => {
    if(response.headers()['set-cookie'] ||
       response.request().url().match(/xiti\.com/)) {
      console.log(JSON.stringify(response.headers(), null, 2));
    }
  });

  await page.goto(url);

  const cookies = await page._client.send('Network.getAllCookies');

  console.log('Cookies:');
  console.log(JSON.stringify(cookies.cookies.filter(c => c.name != ''), null, 2));

  await browser.close();
})();

[josepharhar]

I found that in the Chrome 80 DevTools Network panel, filtering for “xiti” may show requests that don’t have set-cookies, but that seems to be because the request already sent a cookie to the server so the server didn’t reply with a set-cookie. Going to the Application panel and clicking “Clear site data” removed the cookie and cause the set-cookie to appear again in the Network panel in the first request to xiti.

When I ran the puppeteer script you provided, this was one of the cookies that was printed out, which looks like the one you’re interested in: { “name”: “atidx”, “value”: “5D94130E-0878-49D9-A68F-5EE07EB37065”, “domain”: “.xiti.com”, “path”: “/”, “expires”: 1617656737.903769, “size”: 41, “httpOnly”: true, “secure”: true, “session”: false },

[floort]

Hi Robert,

I’ve tried to reproduce the issue with Chrome Version 80.0.3987.132 (Official Build) (64-bit). Both through puppeteer and through the Chrome debug tool the cookies are visible.