[BUG] Test steps spill over into after hooks and fail even though they passed

Context:

• Playwright Version: 1.22
• Operating System: Manjaro Linux
• Node.js version: v16.15.0
• Browser: Chromium

Today I’ve repeatedly seen a test fail even though all its steps passed (and in the video I can actually see the test running fine without any problems). The fail only occurs due to a 60s timeout (see attachment). However, what’s odd is that the test actually ran for an entire 1.7m even though it is not marked as .slow():

test('Edit Tutor Availabilities', async ({shared, webApp, webAppTutor}) => {
    await webAppTutor.goToCalendar();
    await webApp.closeCodeOfConductBannerIfNecessary(shared);
    await webAppTutor.moveCalendarByWeeks(4);
    await webAppTutor.removeAllTutorAvailabilitySlotsInCalendarIfPossible(webApp);
    await webAppTutor.addTutorAvailabilities(shared, webApp);
});

And just FYI the timeout in my config is set to 60s: timeout: 60000,

So it seems to me there is a bug regarding PW’s timeout. The test execution should stop after 60s but it doesn’t. But since it doesn’t and the test passes, it shouldn’t mark the test as failed either.

Upon a closer look, I noticed that the test step are wrongly “spillling over” into the after hooks (I haven’t defined any after hooks):

And at the end of the after hooks, browserContext.close actually takes 24 seconds! How can that be?

In some cases it also took -1ms:

This issue only occurs when I run my entire test suite (23 tests, 4 skipped, 4 minutes in total, 7 spec files & workers) though. When I run those tests individually, it doesn’t occur. So maybe the root cause is the same as in https://github.com/microsoft/playwright/issues/14686.

Playwright test passed but failed with timeout.zip

[pavelfeldman]

I’ll close this bug as non-actionable because there are no steps to follow, but feel free to ping me over Slack!

[azad-derakhshani-GS]

@ramam-v As far as we could figure out, it was happening because the test timeout was too short, because the duration of the test timeout also has to cover the teardown. To alleviate this, I suggested moving the teardown timeout out of the test timeout. However, that approach was not approved so instead (iirc) the errror message was adjusted a bit to avoid confusion.

Another related topic is the number of workers. We used to run our tests with 8 workers (on 8 physical CPU cores), which led to a lot of test timeouts. As it turned out, that’s because the resources on our machines were too scarce with that configuration. Just using the machine for anything else could lead to test timeouts/fails. Halving the number of workers to 4 made our tests much more stable and all but eliminated those timeouts for good. Other users on the PW Slack have observed the same. To make this more flexible, this feature request was created, approved and by now, released.

So what I would suggest as first steps to tackle your issue is:

• try changing the number of workers to 50%
• if that isn’t enough, increase the test timeouts for the tests that still exhibit that issue