[Info] Beta versions available for the new `puppeteer-extra` & `playwright-extra`

The rewrite of puppeteer-extra is available for beta testing, to gather some final feedback before we make the switch. This issue is meant as a canonical reference on how to install those packages (also please report bugs/feedback here). 😄

edit: playwright-extra has landed: https://github.com/berstend/puppeteer-extra/tree/master/packages/playwright-extra

---

👉 We will follow a different approach than a full rewrite with a shared code base between puppeteer-extra and playwright-extra, more info can be found in this comment

(Click for previous (now outdated) info)

❌❌❌ The information below is outdated and does not apply anymore

Context

• A major new version (rewrite) of puppeteer-extra is close to public release 🎉
• The new plugin framework will support both Puppeteer and Playwright (adding playwright-extra)
• Every existing puppeteer-extra-plugin-* should continue working with the new puppeteer-extra
• In addition new plugins (@extra/*) are being released that support both Puppeteer and Playwright

More info can be found in the PR: #303

How to install (must read ⚡)

Important:

• ⚡ The temporary tagged beta packages have issues with npm, please use yarn to install those. ⚡
• The beta versions are published under the @next tag, you must add this tag when installing them.

Available packages

Important:

• ⚡ The documentation links below point to the unreleased automation-extra branch, the installation instructions for the new packages there are written from the perspective of being released and don’t mention the @next tag. Please install the packages as instructed in this issue. ⚡

puppeteer-extra

yarn add puppeteer@5 puppeteer-extra@next

• Supports existing puppeteer-extra-plugin-* as well as the new @extra/* plugins

playwright-extra

yarn add playwright@1.8.0 playwright-extra@next

• Supports Chrome, Firefox and Webkit and the new @extra/* plugins

New plugins

• These plugins use the new base plugin and are compatible with both Playwright & Puppeteer.

@extra/recaptcha

yarn add @extra/recaptcha@next

• A plugin for playwright-extra & puppeteer-extra to solve reCAPTCHAs and hCaptchas automatically.
• Supports Playwright & Puppeteer, Chrome, Firefox and Webkit.

@extra/humanize

yarn add @extra/humanize@next

• A plugin for playwright-extra & puppeteer-extra to humanize input (mouse movements, etc)
• Supports Playwright & Puppeteer, Chrome, Firefox and Webkit.

Existing plugins

• All existing puppeteer-extra plugins are meant to stay compatible with the new puppeteer-extra. Please report any issues you might experience.

Notes

• Existing puppeteer-extra-plugin-* will work with puppeteer-extra, not playwright-extra.
• An updated version of the popular stealth plugin with playwright support is not yet available.
• The target audience of those beta packages are developers interested in testing them and providing feedback before the public release. I don’t advise using them in production unless you really know what you’re doing 😃
• Puppeteer broke typings support in their latest releases, use puppeteer@5 when using TypeScript

[berstend]

hey @berstend! hope all is well, i was just wondering when we can expect to use newer versions of playwright with this

📙 TL;DR: Progress on the switch to the new codebase had stalled but we’re back at it now.

---

A little more context:

Apologies for the delay on this - puppeteer unfortunately breaking TypeScript typings a while back took the wind out of the sails of the planned release of the new branch and I’ve been waiting a bit for the dust to settle. 😅

Given the projects popularity I’m a bit cautious about replacing the old versions until I’m satisfied it’ll be a smooth and backwards compatible transition for everyone, hence we haven’t made the switch yet 😃

I haven’t updated the @next packages in the meantime as the packaging/deployment of those is a bit brittle and cumbersome (our monorepo tool lerna unfortunately fails to resolve their dependencies automatically, which means I need to bump all internal dependencies manually)

I’m not a huge fan of the current limbo situation though and want us to switch to the new codebase as soon as possible.

Things I have on my shortlist in this regard:

• Figure out the definitive best way how we want to deal with typings in our packages (peerDependencies are a mess, if we don’t ship with them as a dependency regular pptr < v5 JS users don’t get to enjoy Intellisense hints, if we do ship with a specific version TS users with a different pptr/pw version might run into conflicts, puppeteer switched to TS/built-in types themselves a while ago, etc.)
• Backport some recent changes made in the old recaptcha plugin to the new @extra/recaptcha
• Optimize the plugin API to allow for easy script injection in workers as well
• See if I can find usage numbers on older puppeteer versions, dropping support for some older versions would make the migration a lot easier

Regarding playwright + stealth: The “hacks” discussed here are fine 😄 Unfortunately they only cover JS based evasions and don’t handle launch args or more importantly CDP commands, which is the main issue I ran into when working on the playwright stealth port. Playwright only allows to create a new CDP session whereas we need to hook into the existing one. I did however find a promising workaround I’m currently fleshing out, so a stealth plugin with full playwright support is on the horizon again. 😃

[terion-name]

Existing puppeteer-extra-plugin-* will work with puppeteer-extra, not playwright-extra.

BTW, I use puppeteer-extra-plugin-stealth with playwrite for a long time with such hack:

const enabledEvasions = [/*list of my requred evasions*/];
    const evasions = enabledEvasions.map(e => new require(`puppeteer-extra-plugin-stealth/evasions/${e}`));
    const stealth = {
      callbacks: [],
      async evaluateOnNewDocument(...args) {
        this.callbacks.push({cb: args[0], a: args[1]})
      }
    }
    evasions.forEach(e => e().onPageCreated(stealth));
    for (let evasion of stealth.callbacks) {
      await browserContext.addInitScript(evasion.cb, evasion.a);
    }