question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

`pytest --collect-only --quiet` output is not "machine readable"

See original GitHub issue

What’s the problem this feature will solve?

I would like to programmatically collect the list of test ids from pytest.

current pytest --collect-only --quiet gets close, but has some non-machine-readable output at the end which is not desirable.

$ pytest --collect-only --quiet tests | tail -5
tests/astpretty_test.py::test_typedast_support_cmdline_27
tests/astpretty_test.py::test_typedast_support_cmdline_3
tests/astpretty_test.py::test_pformat_py38_type_comments

28 tests collected in 0.02s

Describe the solution you’d like

I would like to add either a --json output to this so it can be machine-parsed. perhaps a structure like:

$ pytest --collect-only --json
{
    "testids": [
        "...",
        "...",
        "..."
    ]
}

Alternative Solutions

I could write a plugin, but…

Additional context

I’m writing a tool to detect test pollution automatically. also I suspect this would be useful for those attempting to distribute tests by id across runners – for instance pytest-select

Issue Analytics

  • State:open
  • Created 2 years ago
  • Reactions:1
  • Comments:7 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
asottilecommented, Apr 6, 2022

@elbehery95 head -2 and strip are both hacks indicating that there is not structured output which is entirely what this issue is asking for – you’ll note that the original issue shows exactly the command you’re using and doesn’t help

adding a plugin of course makes this possible but the ask is for pytest to do this itself

1reaction
elbehery95commented, Apr 6, 2022

I would like to programmatically collect the list of test ids from pytest.

I believe there are many ways to do this E.g.

Shell approach:

pytest --collect-only -q tests/test_hello.py | head -n -2 | python -c 'import sys; tests=[i.strip() for i in sys.stdin.readlines()]; print(tests)'

Python approach:

import pytest

class MyCollectionPlugin:
  def __init__(self):
    self.tests = []
  def pytest_report_collectionfinish(self, items):
    self.tests = [item.nodeid for item in items]


if __name__ == "__main__":
  p = MyCollectionPlugin()
  pytest.main(["--collect-only"], plugins=[p])
  print(p.tests)
  # generate_json_from_items_nodeid_list(p.tests)
Read more comments on GitHub >

github_iconTop Results From Across the Web

Managing pytest's output — pytest documentation
By default no output will be shown (because KeyboardInterrupt is caught by pytest). ... To create plain-text machine-readable result files you can issue:....
Read more >
using pytest --collect-only to only return individual test names
Why should I supply --quiet to be able to list the full name of my tests - it makes no sense. I believe...
Read more >
Improving pytest's --collect-only Output | Hugo Martins
I have made a few changes in the way pytest processes the results of test ... pytest --collect-only ==== no tests ran in...
Read more >
pytest Documentation - Read the Docs
Execute the test function with “quiet” reporting mode: ... By default, pytest will not show test durations that are too small (<0.005s) ...
Read more >
Testing — transformers 3.4.0 documentation - Hugging Face
pytest -k "ada and not adam" tests/test_optimization.py. Copy to clipboard ... To have no color (e.g., yellow on white background is not readable):....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found