Memory heap : best use of page "newPage" and urls "goto"

Tell us about your environment:

• Puppeteer version: puppeteer@1.11.0
• Platform / OS version: Centos 7
• URLs (if applicable): 500000 urls crawled one by one
• Node.js version: v10.14.2

I’m using puppeteer to crawl web pages. Should I create only one page to go to all urls (url by url)? or should I create a new page for each url and close it once the page is loaded (always url by url, one after the other).

The problem with the first solution (creating only one page) is the high memory use. The memory used by the page is constantly increasing. This error is : FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory This occurs when 1.5 GB of memory is reached.

The problem with the second solution (creation one page for each url) could be the creation/closing of 500000 pages (one page per url). Is it optimized,? Is it the the best practice?

Thanks

[mikeyoon]

It’s been over two years, but I’ve run into a similar (same?) issue and found a workaround that isn’t too tough to implement. The leak seems to happen in the Browser object or something related, so disconnecting and reconnecting it worked pretty well in my case. After reconnecting, you will need to reassign your Page references.

const browser = await puppeteer.launch(/* options */);
const endpoint = browser.wsEndpoint();

while (true) {
  // do your page load, etc
  browser.disconnect();
  browser = await puppeteer.connect({
    browserWSEndpoint: endpoint,
    defaultViewport: null,
  });
  page = (await browser.pages())[0];
}

With this, I could reload the url using the same page indefinitely, whereas before it would run out of memory in about 2 days.

[Devhercule]

https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/qwirZaQ27E4