Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Pytest: Unable to run tests for non-installed modules

See original GitHub issue

Environment data

  • VS Code version: 1.15.2
  • Extension version (available under the Extensions sidebar): 2020.12.42445261
  • OS and version: Manjaro (up to date @ 2020-12-27); Linux x64 5.9.11
  • Python version (& distribution if applicable, e.g. Anaconda): 3.8.6
  • Type of virtual environment used (N/A | venv | virtualenv | conda | …): n/a
  • Relevant/affected Python packages and their versions: Pytest 6.2.1
  • Relevant/affected Python-related VS Code extensions and their versions: (probably not relevant) Pylance 2020.12.2
  • Value of the python.languageServer setting: Pylance

Setup

Consider the following project layout:

├ .vscode/
│  └ settings.json
├ somepkg/
│  └ __init__.py
└ tests/
   └ test_foo.py

The contents of those files don’t really matter, expect that tests/test_foo.py imports somepkg.

Example .vscode/settings.json:

{
    "python.testing.pytestArgs": ["."],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}

Example somepkg/__init__.py:

def foo():
    return "foo"

Example tests/test_foo.py:

import somepkg
def test_foo():
    assert somepkg.foo() == "foo"

With such project layout, it is impossible to setup VS Code to run those tests without also installing somepkg (for example in a venv).

Here are solutions that I have tried:

Case 0: Do nothing

Obviously, this doesn’t work, because invoking bare pytest doesn’t add current directory to sys.path

VS Code fails to discover tests:

Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== no tests collected, 1 error in 0.06s =====================

Traceback (most recent call last):
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

And similarly command pytest fails:

$ pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error                                                                                         

====================================================== ERRORS =======================================================
________________________________________ ERROR collecting tests/test_foo.py _________________________________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
============================================== short test summary info ==============================================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================= 1 error in 0.06s ==================================================

Case 1: python -m pytest

This is the solution from pytest docs. I added "python.testing.pytestPath": "python -m pytest" to .vscode/settings.json and tried to launch tests.

VS Code, however, still fails on discovery.

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir ~/Documents/VSCode-Pytest-Bug -s --cache-clear .
cwd: ~/Documents/VSCode-Pytest-Bug
Error 2020-12-27 19:55:08: Failed to parse discovered Test [r [Error]: ============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
===================== no tests collected, 1 error in 0.06s =====================

Traceback (most recent call last):
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 2)

	at ChildProcess.<anonymous> (/home/mikolaj/.vscode/extensions/ms-python.python-2020.12.424452561/out/client/extension.js:9:567859)
	at Object.onceWrapper (events.js:313:26)
	at ChildProcess.emit (events.js:223:5)
	at maybeClose (internal/child_process.js:1021:16)
	at Socket.<anonymous> (internal/child_process.js:430:11)
	at Socket.emit (events.js:223:5)
	at Pipe.<anonymous> (net.js:664:12)]

And running pytest manually works fine.

$ python -m pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Case 2: Absolute PYTHONPATH

Here, two new keys were added to the original settings.json: "python.envFile": "${workspaceFolder}/.vscode/.env" and "terminal.integrated.env.linux": { "PYTHONPATH": "${workspaceFolder}" }; and the .vscode/.env file was created with the following content. The path is hard-coded to circumvent issue #13749.

# Somehow using ${workspaceFolder} or ${env:PWD} doesn't work here
PYTHONPATH = /home/mikolaj/Documents/VSCode-Pytest-Bug/

And now test discovery works! But then, running the tests errors out.

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/Documents/VSCode-Pytest-Bug --junit-xml=/tmp/tmp-12233yWmGcDH2XD1w.xml .
cwd: ~/Documents/VSCode-Pytest-Bug

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
-------------- generated xml file: /tmp/tmp-10236qdrGjj8MkHVf.xml --------------
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.07s ===============================

And of course, pytest works like a charm:

$ PYTHONPATH=/home/mikolaj/Documents/VSCode-Pytest-Bug/ pytest
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Case 3: relative PYTHONPATH

This time settings.json looks like this:

{
    "python.testing.pytestArgs": [
        "."
    ],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestPath": "python -m pytest",
    "python.envFile": "${workspaceFolder}/.vscode/.env",
    "terminal.integrated.env.linux": {
        "PYTHONPATH": "."
    },
    "python.testing.cwd": "${workspaceFolder}"
}

And .vscode/.env contains only PYTHONPATH=.

VS Code now correctly discovers the tests, but crashes when trying to run them:

> /usr/bin/python ~/.vscode/extensions/ms-python.python-2020.12.424452561/pythonFiles/pyvsc-run-isolated.py pytest --override-ini junit_family=xunit1 --rootdir ~/Documents/VSCode-Pytest-Bug --junit-xml=/tmp/tmp-10236MyRbp3P5xcEN.xml .
cwd: ~/Documents/VSCode-Pytest-Bug

============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 0 items / 1 error

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_foo.py ______________________
ImportError while importing test module '/home/mikolaj/Documents/VSCode-Pytest-Bug/tests/test_foo.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_foo.py:1: in <module>
    import somepkg
E   ModuleNotFoundError: No module named 'somepkg'
-------------- generated xml file: /tmp/tmp-10236whE7Tokmzock.xml --------------
=========================== short test summary info ============================
ERROR tests/test_foo.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.07s ===============================

And, yet again, pytest works fine:

$ PYTHONPATH=. pytest                                         
================================================ test session starts ================================================
platform linux -- Python 3.8.6, pytest-6.2.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mikolaj/Documents/VSCode-Pytest-Bug
collected 1 item                                                                                                    

tests/test_foo.py .                                                                                           [100%]

================================================= 1 passed in 0.01s =================================================

Expected behavior

I would expect that modifications from cases 1, 2 and 3 allow testing non-installed modules.

Related issues

#6891 - that’s not the case. starting from case 3, I have created a separate test in tests/test_env.py:

import os

def test_pythonpath():
    assert os.getenv("PYTHONPATH") == "."

And it passes correctly.

#12420 - there “Run tests” work, here it doesn’t

Issue Analytics

  • State:open
  • Created 3 years ago
  • Reactions:31
  • Comments:17 (5 by maintainers)

github_iconTop GitHub Comments

7reactions
MKuranowskicommented, Jan 28, 2021

After some time I also found that workaround, so yes, adding tests/__init__.py fixes the issue. Another workaround is to add an empty conftest.py to project root.

4reactions
phloosecommented, Feb 11, 2021

I am having the same issue with an editable install (pip install -e). To make the extension run the tests i have to add an __init__.py as @luabud has suggested and others have also successfully tried. Tests discovery works fine without.

I am just wondering why i can easily run pytest from the command line WITHOUT adding an __init__.py in the tests folder. Personally i don’t want to clutter the test folder with unnecessary files only to make the extension being able to run the tests.

How does the extension invoke pytest?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Cant run tests with pytest, module not found - python
What am I doing wrong? UPDATE. I've tried this too and failed: import sys sys.path.insert(0, "~/project").
Read more >
Full pytest documentation
Start here¶ · Install pytest · Create your first test · Run multiple tests · Assert that a certain exception is raised ·...
Read more >
pytest Documentation - Read the Docs
How to monkeypatch/mock modules and environments . ... How to re-run failed tests and maintain state between test runs .
Read more >
[Fixed] No module named 'pytest' - Finxter
This likely happens because pip is installed but doesn't reside in the path you can use. Although pip may be installed on your...
Read more >
pytest: ModuleNotFoundError: No module named 'requests'
pytest is an outstanding tool for testing Python applications. ... This issue is not the module or package you're trying to use.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Selecting Python interpreter in conda environment has no effect

Next page

imports sorted on save even when formatOnSave = false