Page timeout due to script loading error/timeout

• Puppeteer version: 1.13 and 1.15
• Platform / OS version: Ubuntu 18.04
• URLs (if applicable): https://www.echt-susteren.nl/ and https://www.noordwijk.nl/
• Node.js version: v8.16.0

I use Puppeteer to check 360 sites for cookies. 358 of them work with a 9s timeout. 2 of them don’t work at all, not even with a 90s timeout.

Script:

const browser = await puppeteer.launch(/*{executablePath: '/usr/bin/chromium-browser'}*/);
const page = await browser.newPage();

try {
	await page.goto(url, {timeout: 90000});
}
catch (ex) {
	console.log(ex.constructor.name + ': ' + ex.message);
	process.exit();
}

Only for those 2 URLs, it keeps crashing with TimeoutError: Navigation Timeout Exceeded: 90000ms exceeded.

A separately installed chromium (via executablePath) (73.0.3683.86) doesn’t change anything.

I can’t see why it times out. Only that it times out. Can’t it connect? Is it loading too much? They work fine in my real (Windows) Chrome browser. The other 358 sites work fine too. Is there a more debug mode? A Network panel?

[rudiedirkx]

In my case the goto timeout happens because of a forever-loading blocking resource (js or css). That’ll never trigger the page’s load or domcontentloaded. A bug in Puppeteer IMO, but whatever.

My fix (FINALLY!) is to do what Lighthouse does in its Driver: a Promise.race() for a custom ‘timeout’-ish. The shorter version I used:

const LOAD_FAIL = Math.random();
const sleep = options => new Promise(resolve => {
	options.timer = setTimeout(resolve, options.ms, options.result === undefined ? true : options.result);
});

const sleepOptions = {ms: TIMEOUT - 1000, result: LOAD_FAIL};
const response = await Promise.race([
	sleep(sleepOptions),
	page.goto(url, {timeout: TIMEOUT + 1000}),
]);
clearTimeout(sleepOptions.timer);
const success = response !== LOAD_FAIL;

The sleep() and page.goto() race. If it’s the timeout, the result is clearly a fail (LOAD_FAIL). Otherwise it’s whatever page.goto() returns. Like this, it doesn’t throw a TimeoutException, and the page is still queryable.

That was my issue. I don’t know all your semi-related issues.

[Paul-DS]

Is the normal behavior of puppeteer not supposed to be the same as a “normal” browser ? In this case I think this issue should be reopen.