question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[py] CI tests are intermittently failing for pull requests [list of culprit tests]

See original GitHub issue

What happened?

A fair few of the python tests in CI are a little flaky; resulting in red builds in most python pull requests at the minute; we are merging a few knowing they are ‘flaky’ so let’s get that fixed up.

Edit: There are quite a few; here is a compiled lists of tests that suffer in CI; getting information for them all is pretty time consuming; but they could use some work. Some are harder to recreate than others and they could be due to latency in a CI system; most cases are hitting TimeoutErrors:

  • ~test_full_virtual_authenticator (Todo: Add links)~ (fixed)
  • test_click_should_timeout (Todo: Add links)
  • test_should_timeout_on_page_load_taking_too_long (Todo: Add links)
  • test_collect_log_mutations (BIDI/async) (Todo: Add links)
  • test_should_be_able_to_click_element_that_is_out_of_view_in_aframe_that_is_out_of_view[chrome] output
  • test_touch_pointer_properties - output

os: ubuntu 20.04 chromedriver: 100.0.4896.75 chrome: 100

Issue Analytics

  • State:open
  • Created a year ago
  • Comments:6 (3 by maintainers)

github_iconTop GitHub Comments

1reaction
SinghHrmncommented, May 31, 2022

@symonk test_full_virtual_authenticator: (outdated)

0reactions
nnreposcommented, Oct 13, 2022

hey @symonk can you give some examples of failures in git actions? by looking at the actions of the last few days, i only see tox-lint fail in selenium/webdriver/firefox/firefox_binary.py. thanks 😄

Read more comments on GitHub >

github_iconTop Results From Across the Web

[py] CI tests are intermittently failing for pull requests [list of culprit ...
A browser automation framework and ecosystem. Contribute to SeleniumHQ/selenium development by creating an account on GitHub.
Read more >
Fixing Flaky Time Based Unit Tests | by EG Tech - Medium
Circle CI Build report showing a single test failing. Though retesting usually solves the issue, these flaky failures increase the amount of time...
Read more >
How to auto-reject a pull request if tests are failing (Github ...
The best way to "reject" the pull request is to prevent it from being merged unless the tests pass.
Read more >
Finding test coupling - Ned Batchelder
The test failure itself was a UnicodeError while trying to log a warning message involving a username with a non-ASCII character in it....
Read more >
7. Regression Testing — VisIt User Manual 3.2.2 documentation
VisIt's test suite involves a combination of python scripts in src/test , raw data in archives in the top-level data directory and data...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found