Request interception with caching (rev 8695759) fails when loading stylesheet-initiated fonts (?)

0. Overview

TL;DR:

• The recent fix of request interception and caching in revision 8695759 (#6996) does not work properly in some conditions.
• The error may be stably reproduced.
• While I’m not 100% sure before understanding the relevant Chromium implement details, this seems to be caused by CSS-initiated font resources

1. Test Configuration

My tests are conducted with:

• GNU/Linux Debian 10 (buster)
• Node.js 14.16.0 and npm 7.7.6
• Puppeteer 8695759 (also its parent bf60a30 for comparison)
• Chromium r856583 (the bundled version from Puppeteer)

Here is a Dockerfile that may duplicate the exact test environment:

FROM node:14.16.0-buster
# Just for the dependencies.
RUN apt-get update && apt-get install -y chromium
# Puppeteer installation from git requires npm>=7
# See https://github.com/puppeteer/puppeteer/issues/7034
RUN npm install -g npm@7.7.6
WORKDIR /app
ARG PPTR_RVSN=8695759
RUN npm install git+ssh://[e-mail]/puppeteer/puppeteer.git#${PPTR_RVSN}
ADD test.js .

Here is the sample JavaScript code I used to trigger the issue:

(async () => {
  const puppeteer = require('puppeteer');
  const browser = await puppeteer.launch({
    // Needed only when inside a Docker container
    args: ['--no-sandbox'],
  });
  const page = await browser.newPage();
  const enableCaching = process.env.TEST_ENABLE_CACHING == 'true';
  await page.setRequestInterception(true, enableCaching);
  page.on('request', (request) => {
    request.continue();
  });
  const url = process.env.TEST_URL || 'https://github.com/';
  try {
    await page.goto(url);
  } catch (err) {
    console.error(err);
  }
  await browser.close();
})();

This script accesses https://github.com/ by default, while in practice I observed the same issue from several other frequently visited websites.

2. Tests and Results

From the above configurations one may set up the test environment and launch the quick test. Here is an example of triggering the issue:

$ docker build -t tmptest-pptr -f Dockerfile --build-arg PPTR_RVSN=8695759 . &> /dev/null
$ docker run --entrypoint=/bin/bash --env TEST_ENABLE_CACHING=true tmptest-pptr -c "node test.js"
TimeoutError: Navigation timeout of 30000 ms exceeded
    at /app/node_modules/puppeteer/lib/cjs/puppeteer/common/LifecycleWatcher.js:106:111

Here are the respective results per test:

Puppeteer ↓ \ caching →	enabled	disabled
8695759	error	no error
bf60a30 (previous one)	N/A	no error

Therefore the implement of 8695759 seems not fully functional, at least not for rendering https://github.com/ as tested.

3. Further Analysis

Thanks to Puppeteer’s debugging ability, I tried to dump the CDP communication of the failed test via the below command:

$ docker run --entrypoint=/bin/bash --env TEST_ENABLE_CACHING=true --env DEBUG="puppeteer:*" tmptest-pptr -c "node test.js" &> 8695759_caching_on.log

In case that may save you some time, I’ve uploaded the recorded 8695759_caching_on.log file here: https://gist.github.com/starrify/e1f5ad68c358b932890933be3fb2b736

By comparing the URLs in the log file, it is observed that there are four .woff resources for fonts, all initiated by the stylesheets, that had the Network.requestWillBeSent event but not a Network.responseReceived:

$ tmp_filter() { grep -Po "(?<=protocol:RECV ◀ ).*" | jq 'select(.method=="'$1'")|.params|(.response // .request)|.url' | sort; }
$ diff <(cat 8695759_caching_on.log | tmp_filter Network.requestWillBeSent) <(cat 8695759_caching_on.log | tmp_filter Network.responseReceived)
37,40d36
< "https://github.githubassets.com/static/fonts/alliance/Alliance-No-1-Bold.woff"
< "https://github.githubassets.com/static/fonts/alliance/Alliance-No-1-ExtraBold.woff"
< "https://github.githubassets.com/static/fonts/alliance/Alliance-No-1-Regular.woff"
< "https://github.githubassets.com/static/fonts/alliance/Alliance-No-1-SemiBold.woff"

These four font resources are apparently not cached since the Docker container has a cold start. Also that’s confirmed from the extracted CDP log file (e.g. no Network.requestServedFromCache event observed).

Taking one resource Alliance-No-1-SemiBold.woff as an example for further inspection, I’ve got these from the log file:

$ cat 8695759_caching_on.log | grep "Alliance-No-1-SemiBold.woff" | grep -Po "(?<=protocol:RECV ◀ ).*" | jq '.params|(.requestId//.networkId)' -r
interception-job-28.0
51.164
interception-job-38.0

$ cat 8695759_caching_on.log | grep -P '\b(51\.164|interception-job-28\.0|interception-job-38\.0)\b'
2021-03-30T22:25:48.770Z puppeteer:protocol:RECV ◀ {"method":"Fetch.requestPaused","params":{"requestId":"interception-job-28.0","request":{"url":"https://github.githubassets.com/static/fonts/alliance/Alliance-No-1-SemiBold.woff","method":"GET","headers":{"sec-ch-ua":"","Origin":"https://github.com","sec-ch-ua-mobile":"?0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4427.0 Safari/537.36","Accept":"*/*","Referer":"https://github.githubassets.com/assets/site-7c8868efa7b2b508d6f89c0e94f94864.css"},"initialPriority":"VeryHigh","referrerPolicy":"strict-origin-when-cross-origin"},"frameId":"0979EF16C3B9EFE96B9BD45E76E24204","resourceType":"Font","networkId":"51.164"},"sessionId":"E5290B3CCB2B7FB6607E6186D7644353"}
2021-03-30T22:25:48.935Z puppeteer:protocol:RECV ◀ {"method":"Network.requestWillBeSent","params":{"requestId":"51.164","loaderId":"5423DC2D96468E21561099D7F1FA7D90","documentURL":"https://github.com/","request":{"url":"https://github.githubassets.com/static/fonts/alliance/Alliance-No-1-SemiBold.woff","method":"GET","headers":{"sec-ch-ua":"","Referer":"https://github.githubassets.com/assets/site-7c8868efa7b2b508d6f89c0e94f94864.css","Origin":"https://github.com","sec-ch-ua-mobile":"?0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4427.0 Safari/537.36"},"mixedContentType":"none","initialPriority":"VeryHigh","referrerPolicy":"strict-origin-when-cross-origin"},"timestamp":781832.362451,"wallTime":1617143148.769689,"initiator":{"type":"parser","url":"https://github.githubassets.com/assets/site-7c8868efa7b2b508d6f89c0e94f94864.css"},"type":"Font","frameId":"0979EF16C3B9EFE96B9BD45E76E24204","hasUserGesture":false},"sessionId":"E5290B3CCB2B7FB6607E6186D7644353"}
2021-03-30T22:25:48.935Z puppeteer:protocol:SEND ► {"sessionId":"E5290B3CCB2B7FB6607E6186D7644353","method":"Fetch.continueRequest","params":{"requestId":"interception-job-28.0"},"id":46}
2021-03-30T22:25:48.966Z puppeteer:protocol:RECV ◀ {"method":"Fetch.requestPaused","params":{"requestId":"interception-job-38.0","request":{"url":"https://github.githubassets.com/static/fonts/alliance/Alliance-No-1-SemiBold.woff","method":"GET","headers":{"sec-ch-ua":"","Origin":"https://github.com","sec-ch-ua-mobile":"?0","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4427.0 Safari/537.36","Accept":"*/*","Referer":"https://github.githubassets.com/assets/site-7c8868efa7b2b508d6f89c0e94f94864.css"},"initialPriority":"VeryHigh","referrerPolicy":"strict-origin-when-cross-origin"},"frameId":"0979EF16C3B9EFE96B9BD45E76E24204","resourceType":"Font","networkId":"51.164"},"sessionId":"E5290B3CCB2B7FB6607E6186D7644353"}

This suggests something interesting: for the same request (as identified by the networkId field), Chrome actually issued twice the Fetch.requestPaused event for some reason I do not yet know.

While Puppeteer only manages to resume the first pause, such requests are therefore blocked forever.

In addition, since such resources are initiated in the stylesheets, having the requests pending indefinitely would block the browser from issuing the Page.loadEventFired event, therefore Puppeteer would also keep blocking on the page.goto line.

[Androbin]

Someone offered a workaround upstream: https://bugs.chromium.org/p/chromium/issues/detail?id=1196004#c10

… for the time being, please try --disable-feature=WebFontsCacheAwareTimeoutAdaption as a workaround.

This seems to confirm that this issue is specific to fonts.

[Androbin]

During testing I observed the following: Stylesheet, cache disabled: Network.requestWillBeSent, Fetch.requestPaused Stylesheet, cache enabled: Fetch.requestPaused, Network.requestWillBeSent Font, cache disabled: Network.requestWillBeSent, Fetch.requestPaused Font, cache enabled: Fetch.requestPaused, Network.requestWillBeSent, Fetch.requestPaused Seems like the caching does play an essential role.