[BUG] ElectronApp crash and failure to detach debugger

Context:

• Playwright Version: 1.17, 1.18
• Operating System: Linux, macOS Monetery 12.0
• Node.js version: 17.3.1, 16.11.1
• Browser: Electron
• Test runner: Jest 27.3.1
• Extra: macOS-arm64

System:

• OS: macOS 12.1
• Memory: 95.39 MB / 16.00 GB

Binaries:

• Node: 16.11.1 - ~/.nvm/versions/node/v16.11.1/bin/node
• Yarn: 3.1.1 - /usr/local/bin/yarn
• npm: 8.0.0 - ~/.nvm/versions/node/v16.11.1/bin/npm

Languages:

• Bash: 3.2.57 - /bin/bash

Code Snippet

Help us help you! Put down a short code snippet that illustrates your bug and that we can run and debug locally. For example:

const path = require( 'path' );
const { _electron: electron } = require( 'playwright' );

const APP_PATH = path.join(
	__dirname,
	'../../../release/mac-arm64/WordPress.com.app/Contents/MacOS/WordPress.com'
);

describe( 'User Can log in', () => {
	jest.setTimeout( 5000 );

	let electronApp;

	beforeAll( async () => {
		electronApp = await electron.launch( {
			executablePath: APP_PATH,
		} );
	} );

	it( 'Test Step', async function () {
		console.log( 'hi!' );
	} );

	afterAll( async () => {
		await electronApp.context().close();
	} );
} );

Describe the bug

Hi 👋 I’ve been working on upgrading the Playwright version used in our CI (1.16) to a newer version (1.17/1.18).

We have a task that builds the WordPress.com desktop app, then runs a basic login test against the app. The app is an electron app.

Issue

The issue arose with upgrade to Playwright 1.17/1.18, where Jest would report that suites failed to run:

 ● Test suite failed to run

    thrown: "Exceeded timeout of 5000 ms for a hook.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      24 | 	} );
      25 |
    > 26 | 	afterAll( async () => {
         | 	^
    ...

What is observed is as follows:

1. electron app is launched
2. navigation to login endpoint is completed
3. authentication occurs
4. afterAll hook fails

After some debugging with DEBUG=pw:browser,api, the following are observed after the test step is run (3):

 pw:browser [pid=32097][out] [2022-02-07 16:22:24.464] [platform:mac] [info] Application quit triggered +209ms
  pw:browser [pid=32097][out] [2022-02-07 16:22:24.464] [desktop:runapp] [info] Closing main window, last location: 'https://wordpress.com/' +1ms
  pw:browser [pid=32097][out] [2022-02-07 16:22:24.465] [desktop:runapp] [info] Window closed +1ms
  pw:browser [pid=32097][err] Waiting for the debugger to disconnect... +9ms
  pw:browser [pid=32097][err] 2022-02-07 16:22:24.689 WordPress.com Helper (Renderer)[32101:6094352] CoreText note: Client requested name ".HiraKakuInterface-W7", it will get Times-Roman rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:]. +214ms
  pw:browser [pid=32097][err] 2022-02-07 16:22:24.689 WordPress.com Helper (Renderer)[32101:6094352] CoreText note: Set a breakpoint on CTFontLogSystemFontNameRequest to debug. +1ms
  pw:browser [pid=32097][err] 2022-02-07 16:22:24.691 WordPress.com Helper (Renderer)[32101:6094352] CoreText note: Client requested name ".HiraKakuInterface-W7", it will get Times-Roman rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:]. +1ms
  pw:browser [pid=32097][err] 2022-02-07 16:22:24.691 WordPress.com Helper (Renderer)[32101:6094352] CoreText note: Client requested name ".HiraKakuInterface-W7", it will get Times-Roman rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:]. +1ms
  pw:browser [pid=32097][err] 2022-02-07 16:22:24.692 WordPress.com Helper (Renderer)[32101:6094352] CoreText note: Client requested name ".HiraginoSansGBInterface-W6", it will get Times-Roman rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:]. +1ms
  pw:browser [pid=32097][err] 2022-02-07 16:22:24.693 WordPress.com Helper (Renderer)[32101:6094352] CoreText note: Client requested name ".HiraKakuInterface-W7", it will get Times-Roman rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:]. +0ms
  pw:browser [pid=32097][err] 2022-02-07 16:22:24.693 WordPress.com Helper (Renderer)[32101:6094352] CoreText note: Client requested name ".HiraKakuInterface-W7", it will get Times-Roman rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:]. +1ms
  pw:browser [pid=32097][err] 2022-02-07 16:22:24.693 WordPress.com Helper (Renderer)[32101:6094352] CoreText note: Client requested name ".HiraKakuInterface-W7", it will get Times-Roman rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:]. +1ms
  pw:browser [pid=32097][err] 2022-02-07 16:22:24.693 WordPress.com Helper (Renderer)[32101:6094352] CoreText note: Client requested name ".HiraKakuInterface-W7", it will get Times-Roman rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:]. +0ms
  pw:browser [pid=32097][err] 2022-02-07 16:22:24.694 WordPress.com Helper (Renderer)[32101:6094352] CoreText note: Client requested name ".HiraKakuInterface-W7", it will get Times-Roman rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:]. +0ms
  pw:browser [pid=32097][err] 2022-02-07 16:22:24.697 WordPress.com Helper (Renderer)[32101:6094352] CoreText note: Client requested name ".AppleSDGothicNeoI-Bold", it will get Times-Roman rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:].

These messages are not present when the exact same test is run using Playwright 1.16. Logs from both Playwright 1.16 and 1.18 are attached to this report.

Steps to reproduce

1. check out the https://github.com/Automattic/wp-calypso repository.
2. yarn up playwright@1.18
3. cd desktop
4. USE_HARD_LINKS=false yarn run build
5. DEBUG=pw:browser,api yarn run test:e2e a. if your architecture is not macOS arm64, change the APP_PATH path to the appropriate output directory.

Other info & logs

Looking around, I see that https://github.com/microsoft/playwright/issues/8257 also experienced something similar, but in that report the OP never returned to provide information.

Attached below are the logs from a successful (1.16) run against an unsuccessful (1.18) run demonstrating the difference.

playwright-1-16.log playwright-1-18.log

[worldomonation]

For this issue on macOS arm64 it turns out the version of Electron for the app was the likely culprit - upon updating electron version to 17.0.0 the demonstration script and the original test file are now running as expected:

Parts of the log after the quit event is triggered:

 pw:browser [pid=72645][out] [2022-02-08 13:10:07.064] [platform:mac] [info] Application quit triggered +186ms
  pw:browser [pid=72645][out] [2022-02-08 13:10:07.065] [desktop:runapp] [info] Closing main window, last location: 'https://wordpress.com/log-in?redirect_to=https%3A%2F%2Fwordpress.com%2Flog-in' +0ms
  pw:browser <ws disconnecting> ws://127.0.0.1:55928/24788399-c2d5-40be-bcf7-c9aba7e4dbe8 +103ms
  pw:browser [pid=72645][out] [2022-02-08 13:10:07.168] [desktop:runapp] [info] Window closed +3ms
  pw:browser [pid=72645][err] Debugger ending on ws://127.0.0.1:55928/24788399-c2d5-40be-bcf7-c9aba7e4dbe8 +1ms
  pw:browser [pid=72645][err] For help, see: https://nodejs.org/en/docs/inspector +0ms
  pw:browser <ws disconnected> ws://127.0.0.1:55928/24788399-c2d5-40be-bcf7-c9aba7e4dbe8 +1ms
  pw:browser [pid=72645] <process did exit: exitCode=0, signal=null> +282ms
  pw:browser [pid=72645] starting temporary directories cleanup +1ms
  pw:browser <ws disconnected> ws://127.0.0.1:55930/devtools/browser/a1f5c49b-9838-4cf0-8405-0df499275a2f +1ms
 PASS  test/e2e/specs/login.js (9.257 s)
  User Can log in
    ✓ Log in (2370 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        9.48 s
Ran all test suites.

[pavelfeldman]

We do support Electron 12, but you need the latest revision there (12.2.3) for it to function. Closing as per above, please file a new issue if this is not addressed in full.