Easier debugging / test-specific configurations

Is your feature request related to a problem? Please describe.

I typically run all tests headless, but I often have failing tests that that require visually inspecting the page. Currently, this involves the following:

1. Find my jest-playwright.config.js file
2. Modify my browsers to include only one browser (since I generally don’t need to visually inspect in multiple browsers)
3. Modify my launchOptions to set headless:false and occasionally devtools:true

Once the test passes, I then have to undo all of the changes made above.

What would greatly speed up this process is a way to launch a single test in non-headless mode, possibly with devtools enabled, for one-or-more browsers and/or devices. Ideally, this would be as quick as adding .only or .skip to a test, although this would obviously take some pre-configuration.

Describe the solution you’d like

The verbose-but-flexible way would be to allow passing configuration options with each test. Using the existing jestPlaywright.skip implementation as inspiration, perhaps something like this could work:

jestPlaywright.config({ 
  browser: 'chromium',
  launchOptions: {
    headless: false,
    devtools: true,
  }
}, () => {
  // Tests...
})

This could be made less obnoxious by storing the non-headless/debug configuration separately and passing it to the .config() method:

const jestPlaywrightDebugConfig = {
  browser: 'chromium',
  launchOptions: {
    headless: false,
    devtools: true,
  }
};

jestPlaywright.config(jestPlaywrightDebugConfig, () => {
  // Tests...
})

Better yet, the non-headless/debug configuration could be set in jest-playwright.config.js, allowing a much simpler .debug() method to be used without passing additional parameters:

// jest-playwrite.config.js
module.exports = {
  browsers: [
    'chromium',
    'firefox',
    'webkit',
  ],
  debugOptions: {
    browsers: [
      'chromium'
    ],
    launchOptions: {
      headless: false,
      devtools: true
    }
  }
};

// my.test.js
jestPlaywright.debug(() => {
  // Tests...
})

[mmarkelov]

@jhildenbiddle thank you for your suggestion. Seems like that it can be very helpful. I’ll take it

[creage]

@mmarkelov, @jhildenbiddle Hey mates, sorry for interrupting your discussion, but to me the entire flow looks a bit broken. I mean, you should not modify your code to enter debug mode. You should probably modify it when you actually want to fix it.

Saying that, implementing most of the API presented here is a bit overkill to me.

Let me explain how could one address the actual issue in the OP.

Instead of keeping all of your test run parameters in jest-playwright.config.js, you should keep some/most of them in some JSON file, which will present your environment.

{
	"APP_PROTOCOL": "http:",
	"APP_HOST": "app.in.test",
	"APP_PATH": "app/client",

	"SLOW_MO": 0,
	"VIEWPORT": "1920x1080",

	"HEADLESS": true,
	"SCREEN_FAILED_TESTS": true,
	"DEFAULT_TIMEOUT": 600,
	"RETRY_TIMES": 3,

	"TYPING_DELAY": 0,

	"BROWSERS": "chromium,firefox,webkit",
	"DEVICES": "",
	"TEST_NAME_PATTERN": ""
}

And you can read that JSON in your jest-playwright.config.js, and set it’s values as env vars, but only those that are not yet defined:

const defaults = require('./tests.config.json');
// push config variables to env
Object.keys(defaults).forEach(key => {
	if (!process.env[key]) {
		process.env[key] = defaults[key];
	}
});
// here you consume your env vars, something like
const [width, height] = (process.env.VIEWPORT || '1920x1080').split('x').map(v => +v);

module.exports = {
	launchOptions: {
		headless: process.env.HEADLESS === 'true',
		slowMo: +process.env.SLOW_MO
	},
	contextOptions: {
		ignoreHTTPSErrors: true,
		bypassCSP: true,
		viewport: {
			width,
			height
		}
	},	
	browsers: process.env.BROWSERS ? process.env.BROWSERS.split(',') : ['chromium'],
	devices: process.env.DEVICES ? process.env.DEVICES.split(',') : []
};

This approach gives you an ability to set env vars for you particular test run, and run your tests with no code modification.

set BROWSERS=chromium
set HEADLES=false
set RETRY_TIMES=0
jest -t "my test name I want to run"

So far I made zero changes to run my particular test with particular parameters. And you can parametrize your setup up to your needs.

Again, if you want to skip a test - can’t you use Jest’s it.skip(), and it.only() for selected run? Why you introduce new hooks in this runner?

And in order to inject a breakpoint, since you mentioned VSCode - it has super nice JavaScript Debug Terminal, which attaches to any Node code/process, and you can place your breakpoints directly in your test sources. No need for explicit debugger statements or similar.

What do you think?