[BUG] not.toBeVisible() assertion incorrectly fails at mobile screen resolution (width <= 575px)

Context:

• Playwright Version: 1.27.1
• Operating System: Linux Mint 20.3 Cinnamon 5.2.7
• Node.js version: 16.15.0
• Browser: All

<!-- CLI to auto-capture this info -->

    Running 1 test using 1 worker
    1) [chromium] › ngx-admin/homepage.spec.ts:62:6 › Validate Home Page loads correctly in different resolutions 

    Error: expect(received).not.toBeVisible()
    Call log:
      - expect.toBeVisible with timeout 5000ms
      - waiting for selector "li.menu-item:has-text("FEATURES") >> nth=0"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"
      -   selector resolved to <li nbmenuitem="" _ngcontent-rsv-c116="" class="menu…>…</li>
      -   unexpected value "true"


      70 |     await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).not.toBeVisible();
      71 |     await page.setViewportSize({ width: 375, height: 812 }); // iPhone X
    > 72 |     await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).not.toBeVisible();
         |                                                                                 ^
      73 | });
      74 |

<!-- npx envinfo --preset playwright --markdown -->
## System:
 - OS: Linux 5.4 Linux Mint 20.3 (Una)
 - Memory: 19.67 GB / 29.40 GB
 - Container: Yes
## Binaries:
 - Node: 16.15.0 - /usr/local/bin/node
 - Yarn: 1.22.19 - /usr/bin/yarn
 - npm: 8.11.0 - /usr/local/bin/npm
## Languages:
 - Bash: 5.0.17 - /usr/bin/bash
## npmPackages:
 - playwright: ^1.27.1 => 1.27.1 

Code Snippet Validating www.akveo.com/ngx-admin application homepage at different screen resolutions:

test('Validate Home Page loads correctly in different resolutions', async ({ page }) => {
    await page.goto('https://www.akveo.com/ngx-admin/themes')
    await page.locator('nb-card-body img[alt="Material Light Theme"]').click()
    await page.setViewportSize({ width: 1920, height: 1080 }); // Full HD
    await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).toBeVisible();
    await page.setViewportSize({ width: 1280, height: 800 }); // macbook-13
    await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).toBeVisible();
    await page.setViewportSize({ width: 810, height: 1080 }); // iPad
    await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).not.toBeVisible();
    await page.setViewportSize({ width: 375, height: 812 }); // iPhone X
    await expect(page.locator('li.menu-item:has-text("FEATURES")').first()).not.toBeVisible();
});

Describe the bug The www.akveo.com/ngx-admin is used as a demo application under test If you run the code from the snippet, the first 3 validations will pass, and the last one will fail — that is an incorrect, because the li.menu-item:has-text("FEATURES") element is definitely not visible.

NOTE: the 3rd validation (at the iPad resolution) passes correctly, as the element is already not visible at this resolution. But as soon as the width of the viewport becomes 575 pixels or less — Playwrigth starts to see this element again despite it’s absence on the screen.

The bug was discovered while building a pet test automation project for 1:1 comparison with Cypress (which didn’t have this issue)

[SkaterRambo]

https://www.akveo.com/ngx-admin/themes @skater_rambo.com

[pavelfeldman]

I don’t think it is an app bug per se - menu area is occluded by an opaque element, so the user taps won’t trigger the menu. But we can’t known that before we start clicking - the element is technically visible, everything above it can be transparent and pointer-events: none-y. I would not know why it is not caught by Cypress.