[Question] Tests pass when running one file at a time but when all files are run, random ones start failing?

Hi, so I’m testing mostly links/buttons on the lucidmotors.com website, and some tests are really giving me a hard time - specifically the main nav, side nav and footer. When I run a minimal amount of tests at a time (like a handful or even just 1 file) on gitlab, everything passes. However, when I run all the tests at the same time tests start failing mostly because of timeouts while waiting for selectors or navigations and I know it’s not a bug on the website’s end because each time it’s random ones that are failing (a certain test will pass one time and then fail the next) and I’ll also go and try to reproduce the bug and nothing breaks. I’ve tried upping the workers and timeouts but that doesn’t fix the problem. It’s really hard trying to figure out what the problem is because I haven’t found much when googling this issue.

All my tests follow the same type of layout:

test.describe.parallel('Footer on home page', () => {

  test.beforeEach(async ({ page }) => {
    await page.goto('https://www.lucidmotors.com');
    await page.locator('text=Accept Cookies').click();
  });

  test.afterEach(async ({ page }) => {
    await page.close();
  });

  test('Discover Air', async ({ page }) => {
    await page.locator('footer >> text=Discover Air').click();
    await expect(page).toHaveURL('https://www.lucidmotors.com/air')
  });

  test('Design Yours', async ({ page }) => {
     await page.locator('footer >> text=Design Yours').click();
     await expect(page).toHaveURL('https://www.lucidmotors.com/air/configure')
  });

obviously with more links, and then I thought it might be a network issue so I refactored to have them look like this:

test.describe.parallel('Footer on home page', () => {

  test.beforeEach(async ({ page }) => {
    await page.goto('https://www.lucidmotors.com');
    await page.locator('text=Accept Cookies').click();
  });

  test.afterEach(async ({ page }) => {
    await page.close();
  });

  test('Discover Air', async ({ page }) => {
       await expect(page.locator('footer >> a[href="/air"]')).toContainText("Discover Air");
  });

  test('Design Yours', async ({ page }) => {
       await expect(page.locator('footer >> a[href="/air/configure"]')).toContainText("Design Yours");
  });

I’m wondering if these are the correct way to be setting up the tests or if I’m missing something. Or if it’s really due to some slow backend/network issues.

[haramchang]

Hi thanks for the response! Yeah I wasn’t sure if the after hooks were truly necessary but it didn’t seem to have an adverse effect on the tests and it also seemed like the pages for each test weren’t closing until the whole test run was finished (?) so I thought it’d be best to leave it in there to avoid saturation. And weird!! I’ll try seeing if I can get a Trace for a failed test case and I’ll post it here! Thank you 😃

[jakebanks]

@haramchang thanks for raising this issue, it sounds like what I’m seeing - were you able to work out the root cause? I am yet to work toward a CI environment that supports artifacts so haven’t been able to investigate properly yet.