Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Ultest incorrectly marking failing tests as "passed"

See original GitHub issue

Describe the bug

When using pytest.paramterize in python (which enables code reuse to test a list of input),ultest incorrectly gives “false negative”, meaning it says tests are passing when one of the tests are failing. In the screenshots above, test_parametrize should be marked as failed as opposed to passed

Logs

22:07:58 | INFO | MainThread | logging.py:create_logger:101 | Logger created
22:07:58 | INFO | MainThread | __init__.py:__init__:26 | Using local threaded child watcher
22:07:58 | DEBUG | MainThread | __init__.py:__init__:44 | Handler created
22:07:58 | INFO | MainThread | __init__.py:get_attach_script:227 | Creating script to attach to process 
22:07:58 | INFO | MainThread | __init__.py:run_nearest:128 | Running nearest test in tests/test_ultest.py at line 0
22:07:58 | INFO | MainThread | __init__.py:run_nearest:133 | No tests found for tests/test_ultest.py, rerunning after processing positions
22:07:58 | INFO | MainThread | tracker.py:_init_test_file:97 | Initialising test file /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:62 | Starting job with group update_positions
22:07:58 | INFO | Thread-1 | tracker.py:_async_update:57 | Updating positions in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | file.py:parse_file_structure:25 | Converted pattern {'test': ['\\v^\\s*%(async )?def (test_\\w+)'], 'namespace': ['\\v^\\s*class (\\w+)']} to {'test': [re.compile('^\\s*(?:async )?def (test_\\w+)')], 'namespace': [re.compile('^\\s*class (\\w+)')]}
22:07:58 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py found in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test test_success-4911137721759419562 found in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test test_failure-4911137721759419562 found in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | tracker.py:_async_update:84 | New test test_parametrize-4911137721759419562 found in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | tracker.py:_remove_old_positions:129 | No tests removed
22:07:58 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:62 | Starting job with group update_positions
22:07:58 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:82 | Finished job with group update_positions
22:07:58 | INFO | Thread-1 | tracker.py:_async_update:57 | Updating positions in /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | file.py:parse_file_structure:25 | Converted pattern {'test': ['\\v^\\s*%(async )?def (test_\\w+)'], 'namespace': ['\\v^\\s*class (\\w+)']} to {'test': [re.compile('^\\s*(?:async )?def (test_\\w+)')], 'namespace': [re.compile('^\\s*class (\\w+)')]}
22:07:58 | DEBUG | Thread-1 | tracker.py:_remove_old_positions:129 | No tests removed
22:07:58 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:82 | Finished job with group update_positions
22:07:58 | INFO | MainThread | __init__.py:run_nearest:128 | Running nearest test in tests/test_ultest.py at line 0
22:07:58 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py as started
22:07:58 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering test_success-4911137721759419562 as started
22:07:58 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering test_failure-4911137721759419562 as started
22:07:58 | DEBUG | MainThread | __init__.py:_register_started:288 | Registering test_parametrize-4911137721759419562 as started
22:07:58 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:62 | Starting job with group /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py
22:07:58 | DEBUG | Thread-1 | processes.py:run:51 | Starting test process /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py with command ['poetry', 'run', 'pytest', '-s', '-v', 'tests/test_ultest.py'], cwd = /user/home/allee/feature/allee/batch_id_proper, env = None
22:07:59 | DEBUG | Thread-1 | processes.py:run:77 | Process /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py complete with exit code: 1
22:07:59 | INFO | Thread-1 | output.py:parse_failed:71 | Found failed test in output test_failure in namespaces None of runner python#pytest
22:07:59 | INFO | Thread-1 | output.py:parse_failed:71 | Found failed test in output test_parametrize[5] in namespaces None of runner python#pytest
22:07:59 | DEBUG | Thread-1 | __init__.py:_register_result:299 | Registering /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py as exited with result {"id": "/user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py", "file": "/user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py", "code": 1, "output": "/tmp/ultestlct3tsj2/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py_out"}
22:07:59 | DEBUG | Thread-1 | __init__.py:_register_result:299 | Registering test_success-4911137721759419562 as exited with result {"id": "test_success-4911137721759419562", "file": "/user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py", "code": 0, "output": "/tmp/ultestlct3tsj2/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py_out"}
22:07:59 | DEBUG | Thread-1 | __init__.py:_register_result:299 | Registering test_failure-4911137721759419562 as exited with result {"id": "test_failure-4911137721759419562", "file": "/user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py", "code": 1, "output": "/tmp/ultestlct3tsj2/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py_out"}
22:07:59 | DEBUG | Thread-1 | __init__.py:_register_result:299 | Registering test_parametrize-4911137721759419562 as exited with result {"id": "test_parametrize-4911137721759419562", "file": "/user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py", "code": 0, "output": "/tmp/ultestlct3tsj2/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py/__user__home__allee__feature__allee__batch_id_proper__tests__test_ultest_py_out"}
22:07:59 | DEBUG | Thread-1 | __init__.py:_handle_coroutine:82 | Finished job with group /user/home/allee/feature/allee/batch_id_proper/tests/test_ultest.py

To Reproduce Steps to reproduce the behavior:

Use the code

"""Ultest bug reprex"""
import pytest


def test_success():
    """Normal case."""
    a = 3
    assert a == 3


def test_failure():
    """Normal case."""
    assert 3 == 5


@pytest.mark.parametrize("answer", [3, 5])
def test_parametrize(answer):
    """pytest parameterize"""
    assert answer == 3

Expected behavior

test_parametrize should be marked as failed as opposed to passed

Screenshots

Attached

Additional context

I love this plugin!

Issue Analytics

State:
Created 2 years ago
Comments:9 (4 by maintainers)

Top GitHub Comments

1reaction

grepinsightcommented, Dec 5, 2021

Thank you! The issue is now fixed with the recent commit.

0reactions

rcarrigacommented, Dec 4, 2021

Ah it’s due to a python bug https://bugs.python.org/issue35621 which I had removed the workaround for due to no longer using a separate loop. Thought I had tested but must have misconfigured something. Should be fine in latest commit, sorry for the trouble! 3.7 is a bit troublesome 😅

Top Results From Across the Web

Marking failed tests as passed · Issue #400 - GitHub

The problem is that when we change it to 'Manual Pass' – the test will not appear in the widgets as passed. We...

Testing in Go: Failing Tests - Ilija Eftimov ‍

As you can see above, the test returns PASS because there was no test that got marked as failed. If we would like...

Failed test step reported in next test - TestProject Forum

One of the tests fails with a NoSuchElementException but when checking the reports, the failed test is marked as passed and the test...

Advanced googletest Topics

The ::testing::AssertionResult class solves this problem. An AssertionResult object represents the result of an assertion (whether it's a success or a failure, ...

Cypress: interrupt all tests on first failure - Stack Overflow

In my case the Cypress tests were left pending indefinitely on the CI (Github action workflow) but with this fix they interrupt properly....