[BUG] Failing or flaky tests when run in parallel

Context:

• Playwright Version: [what Playwright version do you use?]
• Operating System: Mac
• Node.js version: 16.14.0
• Browser: Chrome

Code Snippet

Help us help you! Put down a short code snippet that illustrates your bug and that we can run and debug locally. For example:

// our test file
test.describe.configure({ mode: 'parallel' })

// playwright.config.ts

{
  workers: 3
}

We have an issue potentially and a question, the potential issue we’re seeing is whenever we run in parallel mode we’re getting failures in which normally running in 1 worker will have all the tests passing consistently. We’ve tested this in a couple ways:

1. running our suite with 3 workers (running each file through a process)
2. running a single test file and configuring to run each test case to run in parallel

The results are the same, we get some cases that fail that normally don’t under 1 worker. Additionally, when we’re running a single file in parallel we noticed the time it takes to run the file doesn’t seem to improve:

File run in parallel with 3 workers:

File run not in parallel with 1 worker:

If we compare the screen shots, the screen shot with all the passing test (1 worker) has a total run time of 4.0m, theoretically it (3 workers) should cut the run time down to ~1/3 or 1.3m. However, looking at the screen shot with the flaky case (run single file in parallel with 3 workers) which adds an extra 1.1 min, if we took away the additional time caused by the flaky case, it would still be around 4m or so for the entire file to run? Any help would greatly be appreciated, thank you!

[tanmdoan]

Hi @dgozman thank you so much for checking in on this. Turns out the extra workers hangs when using the same credentials, going to file an issue with our auth team to see if it’s something on our side. Will create another issue if needed once we do some debugging internally.

Thanks again!

[jamalebp-formant]

Has anyone figured out whats going on with flaky tests or ways to debug the tests? I’ve seen via trace that my tests are failing isVisible() when its clearly visible.