[BUG] When on serial mode, assertions prevent subsequent tests to run.

## System:
 - OS: macOS 11.4
 - Memory: 52.63 MB / 8.00 GB
## Binaries:
 - Node: 17.8.0 - /usr/local/bin/node
 - Yarn: 1.22.18 - /usr/local/bin/yarn
 - npm: 8.5.5 - /usr/local/bin/npm
## Languages:
 - Bash: 3.2.57 - /bin/bash
## npmPackages:
 - playwright: ^1.24.2 => 1.24.2 

Describe the bug:

1. Run multiple tests on serial mode
2. In one of the tests, have a soft assertion that fails
3. Expected (e.g. correct) result: The specific test is marked as failed but continues to run. And, subsequent tests continue to run as well and if successful marked as passed
4. Actual (e.g. incorrect) result: The specific test is marked as failed but continues to run But subsequent tests do not run

Code Snippet:

Help us help you! Put down a short code snippet that illustrates your bug and that we can run and debug locally.

const { test, expect } = require('@playwright/test');

test.beforeEach(async ({ page }) => {
  await page.goto('https://demo.playwright.dev/todomvc');
});

const TODO_ITEMS = [
  'buy some cheese',
  'feed the cat',
  'book a doctors appointment'
];

test.describe.serial('New Todo', () => {
  test('should allow me to add todo items', async ({ page }) => {
    // Create 1st todo.
    await page.locator('.new-todo').fill(TODO_ITEMS[0]);
    await page.locator('.new-todo').press('Enter');

    ########## This soft assertion will prevent next tests to run because the test runs on serial mode ############
    expect.soft(2).toEqual(1)


    // Make sure the list only has one todo item.
    await expect(page.locator('.view label')).toHaveText([
      TODO_ITEMS[0]
    ]);

    // Create 2nd todo.
    await page.locator('.new-todo').fill(TODO_ITEMS[1]);
    await page.locator('.new-todo').press('Enter');


    // Make sure the list now has two todo items.
    await expect(page.locator('.view label')).toHaveText([
      TODO_ITEMS[0],
      TODO_ITEMS[1]
    ]);

    await checkNumberOfTodosInLocalStorage(page, 2);
  });

  test('should clear text input field when an item is added', async ({ page }) => {
    // Create one todo item.
    await page.locator('.new-todo').fill(TODO_ITEMS[0]);
    await page.locator('.new-todo').press('Enter');

    // Check that input is empty.
    await expect(page.locator('.new-todo')).toBeEmpty();
    await checkNumberOfTodosInLocalStorage(page, 1);
  });

  test('should append new items to the bottom of the list', async ({ page }) => {
    // Create 3 items.
    await createDefaultTodos(page);

    // Check test using different methods.
    await expect(page.locator('.todo-count')).toHaveText('3 items left');
    await expect(page.locator('.todo-count')).toContainText('3');
    await expect(page.locator('.todo-count')).toHaveText(/3/);

    // Check all items in one call.
    await expect(page.locator('.view label')).toHaveText(TODO_ITEMS);
    await checkNumberOfTodosInLocalStorage(page, 3);
  });

  test('should show #main and #footer when items added', async ({ page }) => {
    await page.locator('.new-todo').fill(TODO_ITEMS[0]);
    await page.locator('.new-todo').press('Enter');

    await expect(page.locator('.main')).toBeVisible();
    await expect(page.locator('.footer')).toBeVisible();
    await checkNumberOfTodosInLocalStorage(page, 1);
  });
});

/**
 * @param {import('@playwright/test').Page} page
 * @param {number} expected
 */
 async function checkNumberOfTodosInLocalStorage(page, expected) {
  return await page.waitForFunction(e => {
    return JSON.parse(localStorage['react-todos']).length === e;
  }, expected);
}

/**
 * @param {import('@playwright/test').Page} page
 * @param {number} expected
 */
 async function checkNumberOfCompletedTodosInLocalStorage(page, expected) {
  return await page.waitForFunction(e => {
    return JSON.parse(localStorage['react-todos']).filter(i => i.completed).length === e;
  }, expected);
}

/**
 * @param {import('@playwright/test').Page} page
 * @param {string} title
 */
async function checkTodosInLocalStorage(page, title) {
  return await page.waitForFunction(t => {
    return JSON.parse(localStorage['react-todos']).map(i => i.title).includes(t);
  }, title);
}

[gal064]

Yep soft assertions may work according the dry definition but I think it’s a bug from a user perspective (i.e. a miss when defining soft assertions).

The intended use-case of soft assertions is for a failure to not stop the test suite from running. This is not the case with serial tests, and probably not intentionally. I don’t see any logic to why the current test should continue to run but subsequent tests stop.

With that said, I understand if it’s not top priority though

[michelle-choi1]

Yep soft assertions may work according the dry definition but I think it’s a bug from a user perspective (i.e. a miss when defining soft assertions).

The intended use-case of soft assertions is for a failure to not stop the test suite from running. This is not the case with serial tests, and probably not intentionally. I don’t see any logic to why the current test should continue to run but subsequent tests stop.

With that said, I understand if it’s not top priority though

agreeing with @gal064 - my use case is that I’m using serial mode to split filling out an order checkout process (which includes a lot of steps, expects, and page navigations) into multiple tests that have to run sequentially, and i just ran across a case in which i’d like an assertion to be soft (not to keep the soft failure permanently, but as a temporary measure to help me speed development) but i also understand there are other priorities. just wanted to add a data point!