[BUG] Problems with parallelism and page.waitForSelector

Context:

• Playwright Version: 1.19.1
• Operating System: Linux
• Node.js version: 14
• Browser: Chromium
• Extra: Our tests run on an Azure Devops pipeline agent with 2 cores and 7gb of RAM.

page.waitForSelector('[data-test-hook="home-page-container"]', { state: 'attached', timeout: 1_000 });

Describe the bug

While migrating our E2E tests from Puppeteer to Playwright, I’ve noticed that as we increase the amount of parallelism to speed up our tests, our calls to page.waitForSelector() get flakier. To the point that in our trace files we can see that the desired selector is already attached in the ‘before’ tab, and yet we still get timeout errors.

Below I have attached a screenshot showing the issue. If it is helpful, I can provide the full trace file internally at Microsoft.

It is worth noting that if I use page.$('[data-test-hook="home-page-container"]') to test for the element’s presence, it does not seem to be affected by the parallelism problem, but it comes with the major downside that it doesn’t put records of the action into the trace file, which is often essential to our debugging.

[dgozman]

@joshuacc Model B is almost exactly how it works. When you say “on the next turn of the event loop”, that’s not entirely accurate though. Checking something in the page involves going to the other OS process (browser process vs playwright’s node process), so the check is very much asynchronous as well. This also means that going to the Model A is impossible, because the very first iteration of checking for the element may take arbitrary amount of time.

In addition, I’d say page.waitForSelector() is rarely used. Most of the time you just do page.locator().click() and that waits for the element inside. If you had to resort of page.waitForSelector(), you have something unusual going on, and most likely the element is not immediately present in the page. Note this is just an opinion, no hard statistics behind it 😄

[dgozman]

@joshuacc There could definitely be a bug, but we’ll need to debug locally with a repro to find it 😄

When I use page.$() to check for the presence of that element, this failure doesn’t occur.

That’s because page.$() does not have a timeout, so no matter how slow the machine is, it will eventually respond with an element. If you bump your waitForSelector timeout, you’ll see less flakiness.

Overall, we recommend to run without any timeouts for specific actions (or with a rather large timeout like 30s), and instead rely on a single test timeout to find issues. This way, you usually do not have flakiness due to process scheduling under heavy load.