Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[BUG] fixture failures in tests using `fixme` is reported as failed though on retry fixture executed without errors

See original GitHub issue

Context:

Playwright Version: 1.21.1
Operating System: Mac
Node.js version: 16.14.2
Browser: All

Code Snippet

Repo to replicate: https://github.com/vigneshrajsb/playwright-fixme-report-issue

npm install 
npx playwright test

Describe the bug

When a test uses test.fixme() at the beginning, the execution immediately stops and exits but the fixtures are executed(as expected).

When a test with setup fixture fails in the first run but passes in a retry for a fixme test, the HTML report still reports the test as failure.

Expected: On retry as the fixture execution was successful, the reporter should flag the test as flaky similar to tests without fixme

Should the reporter flag this test as flaky?

Issue Analytics

State:
Created a year ago
Comments:6 (3 by maintainers)

Top GitHub Comments

1reaction

vigneshrajsbcommented, Apr 28, 2022

ah!! gotchu!! apologies… yeah i think it makes sense now… we will override the lint rules for the playwright tests

1reaction

dgozmancommented, Apr 28, 2022

The issue here is that execution failed before the test.fixme() call had a chance to run. This makes the test a proper “failure”. Try using this variant of test.fixme to avoid the problem and let me know if that helps.