[Feature] file order with multiple workers

From looking at the documentation there doesn’t seem to be a way to specify the order that tests are run while also allowing multiple workers.

I’m currently using playwright to test a fairly complex web hosting application that has a lot of platform based settings. These settings can take a long time to update (over 1 minute) and can cause other tests to fail as they affect all users. At the moment we’re just using the single worker in the CI pipeline but as the number of tests grow this will mean the test suite will take a very long time to complete.

I was thinking of a more fully featured “test list” file where you can specify items to run sequentially and items that ok to run in parallel.

// test.list.ts
import './feature-b.spec.ts';
import './feature-a.spec.ts';

// new.test.list.ts
[
   './global-setting-a.spec.ts',  // changes to global setting
   ['./depends-on-a-1.spec.ts', './depends-on-a-2.spec.ts', '.depends-on-a-3.spec.ts', ...etc], // runs in parallel
   './global-setting-b.spec.ts'
   // runs any other tests in parallel that are not specified.
]

[rwoll]

The timing here is very nice! Please watch: https://github.com/microsoft/playwright/issues/14895#issuecomment-1207665459. Folding into that issue.

[TJHdev]

It looks like the documentation was recently updated for “test list” files.

I was thinking something along the lines of syntax like this, but it’s just a rough idea.

import { test } from '@playwright/test';

import addEmailRole from 'components/platform/addEmailRole.spec'
import addBackupRole from 'components/platform/addBackupRole.spec'
import removeEmailRole from 'components/platform/removeEmailRole.spec'
import removeBackupRole from 'components/platform/removeBackupRole.spec'

import addEmails from 'components/addEmails.spec';
import emailsBulkActionBar from 'components/emailsBulkActionBar.spec';

import addWebsite from 'components/addWebsite.spec';
import websitesBulkActionBar from 'components/websitesBulkActionBar.spec';
import backups from 'components/website/backups.spec'

test.describe(addEmailRole);  // won't start the next line until all tests have completed.
test.describe([emailsBulkActionBar, addEmail]);  // workers can be assigned to any tests within these files as the same time. Again the next line won't be started until all tests have completed.
test.describe(removeEmailRole);
test.describe(addBackupRole);
test.describe([backups, addWebsite, websitesBulkActionBar]);
test.describe(removeBackupRole);

Important point is that multiple workers can be assigned. In the example above if 6 workers are allocated:

1. While addEmailRole is being worked through only 1 worker is being used, while 5 sit idle.
2. Once [emailsBulkActionBar, addEmail] begins all 6 workers can be allocated.
3. removeEmailRole won’t start until all the tests have completed in [emailsBulkActionBar, addEmail]