Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Tests don't pass if run in random order (bug?)
See original GitHub issueSteps to reproduce:
rm -rf .tox- add
pytest-randomlytorequirements-dev.txt3 runtox -e pya few times
Example of order which fails:
tests/test_find_root.py::test_find_project_root[src0] PASSED [ 1%]
tests/test_find_root.py::test_find_project_root_no_root PASSED [ 2%]
tests/test_find_root.py::test_find_project_root[src1] PASSED [ 4%]
tests/test_no_inits.py::test_no_inits PASSED [ 5%]
tests/test_setupcfg.py::test_nbqa_ini_works PASSED [ 7%]
tests/test_cmdline.py::test_cli_command_str PASSED [ 8%]
tests/tools/test_black.py::test_black_multiple_files PASSED [ 9%]
tests/tools/test_black.py::test_black_works PASSED [ 11%]
tests/tools/test_black.py::test_successive_runs_using_black PASSED [ 12%]
tests/tools/test_black.py::test_black_works_with_multiline PASSED [ 14%]
tests/tools/test_black.py::test_black_works_with_trailing_semicolons PASSED [ 15%]
tests/test_map_python_line_to_nb_lines.py::test_black_unparseable_output PASSED [ 16%]
tests/test_map_python_line_to_nb_lines.py::test_map_python_line_to_nb_lines PASSED [ 18%]
tests/test_get_notebooks.py::test_get_notebooks[_build] PASSED [ 19%]
tests/test_get_notebooks.py::test_get_notebooks[venv] PASSED [ 21%]
tests/test_get_notebooks.py::test_get_notebooks[.git] PASSED [ 22%]
tests/tools/test_doctest.py::test_doctest_invalid_import PASSED [ 23%]
tests/tools/test_doctest.py::test_doctest_works PASSED [ 25%]
tests/test_create_blank_init_files.py::test_create_blank_init_files PASSED [ 26%]
tests/test_runtime_errors.py::test_unable_to_parse PASSED [ 28%]
tests/test_runtime_errors.py::test_directory_without_notebooks PASSED [ 29%]
tests/test_runtime_errors.py::test_unable_to_parse_output PASSED [ 30%]
tests/test_runtime_errors.py::test_unable_to_reconstruct_message_pythonpath FAILED [ 32%]
tests/test_runtime_errors.py::test_missing_root_dir PASSED [ 33%]
tests/test_runtime_errors.py::test_unable_to_reconstruct_message PASSED [ 35%]
tests/test_runtime_errors.py::test_missing_command PASSED [ 36%]
tests/test_include_exclude.py::test_cli_files PASSED [ 38%]
tests/test_include_exclude.py::test_config_files PASSED [ 39%]
tests/test_include_exclude.py::test_config_exclude PASSED [ 40%]
tests/test_include_exclude.py::test_cli_exclude PASSED [ 42%]
tests/test_pyproject_toml.py::test_pyproject_toml_works PASSED [ 43%]
tests/test_nbqa_diff.py::test_diff_present PASSED [ 45%]
tests/test_nbqa_diff.py::test_diff_and_mutate PASSED [ 46%]
tests/test_ipython_magics.py::test_ignore_cells[[tool.nbqa.ignore_cells]\nflake8=["%%custommagic", "%%anothercustommagic"]-_validate_ignore_cells_without_warning-pyproject.toml] PASSED [ 47%]
tests/test_ipython_magics.py::test_ignore_cells[[flake8]\nignore_cells=%%%%custommagic,%%%%anothercustommagic-_validate_ignore_cells_without_warning-.nbqa.ini] PASSED [ 49%]
tests/test_ipython_magics.py::test_indented_magics[[tool.nbqa.mutate]\nblack=1-_validate_magics_with_black-pyproject.toml] PASSED [ 50%]
tests/test_ipython_magics.py::test_ignore_cells[--nbqa-ignore-cells=%%custommagic-_validate_ignore_cells_with_warning-None] PASSED [ 52%]
tests/test_ipython_magics.py::test_ignore_cells[[tool.nbqa.ignore_cells]\nflake8=["%%custommagic"]-_validate_ignore_cells_with_warning-pyproject.toml] PASSED [ 53%]
tests/test_ipython_magics.py::test_magics_with_flake8[--extend-ignore=F821-_validate_magics_flake8_warnings] PASSED [ 54%]
tests/test_ipython_magics.py::test_ignore_cells[[flake8]\nignore_cells=%%%%custommagic-_validate_ignore_cells_with_warning-.nbqa.ini] PASSED [ 56%]
tests/test_ipython_magics.py::test_ignore_cells[--nbqa-ignore-cells=%%custommagic,%%anothercustommagic-_validate_ignore_cells_without_warning-None] PASSED [ 57%]
tests/tools/test_mypy_works.py::test_mypy_works PASSED [ 59%]
tests/test_configs_work.py::test_configs_work_in_setupcfg PASSED [ 60%]
tests/test_configs_work.py::test_configs_work PASSED [ 61%]
tests/test_configs_work.py::test_nonexistent_config PASSED [ 63%]
tests/test_configs_work.py::test_setupcfg_is_preserved PASSED [ 64%]
tests/test_configs_work.py::test_configs_work_in_nbqaini PASSED [ 66%]
tests/tools/test_isort_works.py::test_isort_trailing_semicolon PASSED [ 67%]
tests/tools/test_isort_works.py::test_old_isort PASSED [ 69%]
tests/tools/test_isort_works.py::test_isort_separated_imports[notebook_with_separated_imports_other.ipynb] PASSED [ 70%]
tests/tools/test_isort_works.py::test_isort_works PASSED [ 71%]
tests/tools/test_isort_works.py::test_isort_separated_imports[notebook_with_separated_imports.ipynb] PASSED [ 73%]
tests/tools/test_isort_works.py::test_isort_initial_md PASSED [ 74%]
tests/tools/test_isort_works.py::test_old_isort_separated_imports PASSED [ 76%]
tests/test_file_not_found.py::test_file_not_found PASSED [ 77%]
tests/tools/test_autoflake.py::test_autoflake_toml PASSED [ 78%]
tests/tools/test_autoflake.py::test_autoflake_cli PASSED [ 80%]
tests/test_running_from_different_dir.py::test_running_in_different_dir_works[arg1-cwd1] PASSED [ 81%]
tests/test_running_from_different_dir.py::test_running_in_different_dir_works[arg2-cwd2] PASSED [ 83%]
tests/test_running_from_different_dir.py::test_running_in_different_dir_works[arg0-cwd0] PASSED [ 84%]
tests/test_running_from_different_dir.py::test_running_in_different_dir_works[arg3-cwd3] PASSED [ 85%]
tests/test_return_code.py::test_pylint_return_code PASSED [ 87%]
tests/test_return_code.py::test_black_return_code PASSED [ 88%]
tests/test_return_code.py::test_flake8_return_code PASSED [ 90%]
tests/test_return_code.py::test_check_ast_return_code PASSED [ 91%]
tests/test_return_code.py::test_autoflake_return_code PASSED [ 92%]
tests/tools/test_pyupgrade.py::test_pyupgrade PASSED [ 94%]
tests/test_allow_mutation.py::test_allow_mutation PASSED [ 95%]
tests/test_version.py::test_version PASSED [ 97%]
tests/tools/test_pylint_works.py::test_pylint_works PASSED [ 98%]
tests/tools/test_flake8_works.py::test_flake8_works PASSED
example of order which passes:
tests/test_return_code.py::test_pylint_return_code PASSED [ 1%]
tests/test_return_code.py::test_autoflake_return_code PASSED [ 2%]
tests/test_return_code.py::test_flake8_return_code PASSED [ 4%]
tests/test_return_code.py::test_black_return_code PASSED [ 5%]
tests/test_return_code.py::test_check_ast_return_code PASSED [ 7%]
tests/test_running_from_different_dir.py::test_running_in_different_dir_works[arg0-cwd0] PASSED [ 8%]
tests/test_running_from_different_dir.py::test_running_in_different_dir_works[arg3-cwd3] PASSED [ 9%]
tests/test_running_from_different_dir.py::test_running_in_different_dir_works[arg2-cwd2] PASSED [ 11%]
tests/test_running_from_different_dir.py::test_running_in_different_dir_works[arg1-cwd1] PASSED [ 12%]
tests/test_setupcfg.py::test_nbqa_ini_works PASSED [ 14%]
tests/test_allow_mutation.py::test_allow_mutation PASSED [ 15%]
tests/tools/test_flake8_works.py::test_flake8_works PASSED [ 16%]
tests/test_create_blank_init_files.py::test_create_blank_init_files PASSED [ 18%]
tests/tools/test_doctest.py::test_doctest_works PASSED [ 19%]
tests/tools/test_doctest.py::test_doctest_invalid_import PASSED [ 21%]
tests/test_map_python_line_to_nb_lines.py::test_black_unparseable_output PASSED [ 22%]
tests/test_map_python_line_to_nb_lines.py::test_map_python_line_to_nb_lines PASSED [ 23%]
tests/test_get_notebooks.py::test_get_notebooks[_build] PASSED [ 25%]
tests/test_get_notebooks.py::test_get_notebooks[venv] PASSED [ 26%]
tests/test_get_notebooks.py::test_get_notebooks[.git] PASSED [ 28%]
tests/tools/test_isort_works.py::test_isort_separated_imports[notebook_with_separated_imports_other.ipynb] PASSED [ 29%]
tests/tools/test_isort_works.py::test_isort_trailing_semicolon PASSED [ 30%]
tests/tools/test_isort_works.py::test_isort_initial_md PASSED [ 32%]
tests/tools/test_isort_works.py::test_isort_separated_imports[notebook_with_separated_imports.ipynb] PASSED [ 33%]
tests/tools/test_isort_works.py::test_old_isort PASSED [ 35%]
tests/tools/test_isort_works.py::test_isort_works PASSED [ 36%]
tests/tools/test_isort_works.py::test_old_isort_separated_imports PASSED [ 38%]
tests/test_include_exclude.py::test_cli_exclude PASSED [ 39%]
tests/test_include_exclude.py::test_cli_files PASSED [ 40%]
tests/test_include_exclude.py::test_config_exclude PASSED [ 42%]
tests/test_include_exclude.py::test_config_files PASSED [ 43%]
tests/tools/test_black.py::test_successive_runs_using_black PASSED [ 45%]
tests/tools/test_black.py::test_black_works_with_multiline PASSED [ 46%]
tests/tools/test_black.py::test_black_works PASSED [ 47%]
tests/tools/test_black.py::test_black_multiple_files PASSED [ 49%]
tests/tools/test_black.py::test_black_works_with_trailing_semicolons PASSED [ 50%]
tests/test_ipython_magics.py::test_ignore_cells[[flake8]\nignore_cells=%%%%custommagic,%%%%anothercustommagic-_validate_ignore_cells_without_warning-.nbqa.ini] PASSED [ 52%]
tests/test_ipython_magics.py::test_ignore_cells[[tool.nbqa.ignore_cells]\nflake8=["%%custommagic", "%%anothercustommagic"]-_validate_ignore_cells_without_warning-pyproject.toml] PASSED [ 53%]
tests/test_ipython_magics.py::test_ignore_cells[--nbqa-ignore-cells=%%custommagic,%%anothercustommagic-_validate_ignore_cells_without_warning-None] PASSED [ 54%]
tests/test_ipython_magics.py::test_ignore_cells[[flake8]\nignore_cells=%%%%custommagic-_validate_ignore_cells_with_warning-.nbqa.ini] PASSED [ 56%]
tests/test_ipython_magics.py::test_ignore_cells[[tool.nbqa.ignore_cells]\nflake8=["%%custommagic"]-_validate_ignore_cells_with_warning-pyproject.toml] PASSED [ 57%]
tests/test_ipython_magics.py::test_magics_with_flake8[--extend-ignore=F821-_validate_magics_flake8_warnings] PASSED [ 59%]
tests/test_ipython_magics.py::test_indented_magics[[tool.nbqa.mutate]\nblack=1-_validate_magics_with_black-pyproject.toml] PASSED [ 60%]
tests/test_ipython_magics.py::test_ignore_cells[--nbqa-ignore-cells=%%custommagic-_validate_ignore_cells_with_warning-None] PASSED [ 61%]
tests/tools/test_mypy_works.py::test_mypy_works PASSED [ 63%]
tests/test_pyproject_toml.py::test_pyproject_toml_works PASSED [ 64%]
tests/tools/test_autoflake.py::test_autoflake_toml PASSED [ 66%]
tests/tools/test_autoflake.py::test_autoflake_cli PASSED [ 67%]
tests/tools/test_pylint_works.py::test_pylint_works PASSED [ 69%]
tests/test_file_not_found.py::test_file_not_found PASSED [ 70%]
tests/test_runtime_errors.py::test_unable_to_parse PASSED [ 71%]
tests/test_runtime_errors.py::test_missing_command PASSED [ 73%]
tests/test_runtime_errors.py::test_unable_to_parse_output PASSED [ 74%]
tests/test_runtime_errors.py::test_directory_without_notebooks PASSED [ 76%]
tests/test_runtime_errors.py::test_missing_root_dir PASSED [ 77%]
tests/test_runtime_errors.py::test_unable_to_reconstruct_message PASSED [ 78%]
tests/test_runtime_errors.py::test_unable_to_reconstruct_message_pythonpath PASSED [ 80%]
tests/test_find_root.py::test_find_project_root[src0] PASSED [ 81%]
tests/test_find_root.py::test_find_project_root[src1] PASSED [ 83%]
tests/test_find_root.py::test_find_project_root_no_root PASSED [ 84%]
tests/test_version.py::test_version PASSED [ 85%]
tests/test_configs_work.py::test_configs_work PASSED [ 87%]
tests/test_configs_work.py::test_nonexistent_config PASSED [ 88%]
tests/test_configs_work.py::test_setupcfg_is_preserved PASSED [ 90%]
tests/test_configs_work.py::test_configs_work_in_nbqaini PASSED [ 91%]
tests/test_configs_work.py::test_configs_work_in_setupcfg PASSED [ 92%]
tests/tools/test_pyupgrade.py::test_pyupgrade PASSED [ 94%]
tests/test_cmdline.py::test_cli_command_str PASSED [ 95%]
tests/test_nbqa_diff.py::test_diff_and_mutate PASSED [ 97%]
tests/test_nbqa_diff.py::test_diff_present PASSED [ 98%]
tests/test_no_inits.py::test_no_inits PASSED [100%]
In the failing case, here’s the error:
FAILED tests/test_runtime_errors.py::test_unable_to_reconstruct_message_pythonpath - ModuleNotFoundError: Command `remove_comments` not found by nbqa.
Issue Analytics
- State:
- Created 3 years ago
- Comments:12 (9 by maintainers)
Top Results From Across the Web
Ability to run tests within a file in a random order #4386 - GitHub
To isolate tests in Jest, you can use jest.resetModules() and then re-require modules for each individual tests. If you are strict about this, ......
Read more >How can I make my JUnit tests run in random order?
What I want is to randomize the running order of tests so I will make sure they are not dependent to each other....
Read more >How to Fix Flaky Tests - Semaphore CI
Randomly failing tests are the hardest to debug. Here's a framework you can use to fix them and keep your test suite healthy....
Read more >Why tests pass locally but fail in Jenkins | Object Partners
Symptom: Tests pass locally when run in a certain order, but fail in Jenkins when run in a different order (or tests pass/fail...
Read more >Troubleshooting Continuous Integration, or How to Debug ...
Or, if the tests pass on the CI after a relaunch, it means there is a random factor that you haven't identified, and...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Oh, another option to fix the test would be to call
python -m nbqawith the proper flags + env and parse the output.If the latter (subprocess.run) works, then that might be better - thanks! The first is fine too though