Request content was evicted from inspector cache ??

Hello,

I’m using Puppeteer to do an offline mirror of a site. Problem is, I can’t save large files. The following code is a simplified version of the mirroring bot, and it fails on the .flv (larger) file. My actual target site contains similar flash players and flv files.

const puppeteer = require('puppeteer');
const url = require('url');
const fs = require('fs');
const path = require('path');
const fse = require('fs-extra');

(async () => {
  const savePath = './site';
  const browser = await puppeteer.launch({
    args: [
      '--no-sandbox',
      '--ppapi-flash-version=28.0.0.126',
      '--ppapi-flash-path=/usr/lib/PepperFlash/libpepflashplayer.so',
      '--user-data-dir=/home/howe/.config/chromium'
    ],
    headless: false,
  });
  const page = await browser.newPage();
  page.on('response', async (res) => {
    const parsed = url.parse(res.url);
    try {
      const fn = path.join(savePath, parsed.pathname);
      console.log(`Saving file ${fn}...`);
      await fse.ensureDir(path.dirname(fn));
      await fse.writeFile(fn, await res.buffer());
    }
    catch (e) {
      console.error('Error:', e)
    }
  });
  await page.goto('http://flash.flowplayer.org/demos/plugins/flash/viralvideos-extconfig.html', { waitUntil: 'networkidle2' });
  // browser.close();
})();

After downloading a few smaller files, it throws the following cryptic error:

Protocol error (Network.getResponseBody): Request content was evicted from inspector cache undefined

Any clues? Is this a bug? I have no idea why it works on smaller but not larger files.

Thanks, Howe

[howesteve]

Ok, It seems this works (but I didn’t test thoroughly):

await page._client.send('Network.enable', {
      maxResourceBufferSize: 1024 * 1204 * 100,
      maxTotalBufferSize: 1024 * 1204 * 200,
    })

Hope that helps someone else. Thanks @aslushnikov .

[aslushnikov]

There’s a limitation on the resource content size that devtools keep in memory. It’s 10Mb for resource, and there’s also a total of 100Mb limit to store all resources: InspectorNetworkAgent.cpp

There’s experimental support in the protocol to amend these limits: Network.enable.