Memory usage in 3.3.0

Bug Description

TL;DR: memory usage seems to have increased several fold in 3.3.0. With the default log capturing enabled, memory usage has the potential to increase to ludicrous levels.

I was troubleshooting issues with tests hanging on our CI server and massive swap utilization, and I found it to be due to an upgrade to pytest 3.3.0. Running our test suite with 3.3.0 quickly consumed all of the memory and began swapping.

Looking at the changelog, the capturing of log output seemed to be the most likely cause. Our application uses SQLAlchemy, which has verbose logging, and has rather verbose logging itself, so I attempted to disable log capturing via both of the options specified here, and while the first seemed to decrease the massive consumption of virtual memory seen with the default options, neither was able to reduce memory utilization enough for our test suite to run.

I have collected the memory utilization of various configurations below. For all of the 3.3.0 runs, the CI server began swapping and the test suite hung, although some configurations resulted in more virtual memory consumption than others.

In all cases, regardless of attempting to disable log capturing, the virtual memory output jumped significantly when hitting tests that produced a large amount of log output. However, even prior to hitting those tests, virtual memory utilization was around twice that of its peak in version 3.2.5.

The tests were running in a python:3.5.2 docker container, via an Ubuntu 16.04 OS. The system has 2 GB of RAM and 2 GB of swap space. In all cases, pytest was invoked via tox, with the command coverage run --source <project directory> -m py.test {posargs}

pytest version	additional options	peak virtual memory consumed	peak resident memory consumed	test suite completed
3.2.5	none	2685 MB	243 MB	yes
3.3.0	none	11.9 GB !!!	418 MB	no
3.3.0	addopts=-p no:logging	9105 MB	366 MB	no
3.3.0	--no-print-logs	12.2 GB !!!	431 MB	no
3.3.0	addopts=-p no:logging and --no-print-logs	12.2 GB !!!	415 MB	no

pytest is not included in the pip output below, since its version varied by test. All other packages remained consistent between tests. Some private distributions have been redacted.

We have had to pin our version of pytest to <3.3.0 for the moment. Is there anything else I can try to reduce the base memory footprint of version 3.3.0?

I’m happy to gather more data as needed.

Thanks!

Pip List

aiohttp (2.2.5) alembic (0.9.5) amqp (2.2.2) aniso8601 (1.3.0) apipkg (1.4) apispec (0.25.4) appnope (0.1.0) asn1crypto (0.23.0) astroid (1.5.3) async-timeout (2.0.0) attrs (17.3.0) autopep8 (1.3.3) Babel (2.5.1) billiard (3.5.0.3) bleach (2.1.1) celery (4.1.0) certifi (2017.7.27.1) cffi (1.11.2) chardet (3.0.4) click (6.7) coverage (4.4.2) cryptography (2.1.1) debtcollector (1.18.0) decorator (4.1.2) dill (0.2.7.1) docutils (0.14) entrypoints (0.2.3) execnet (1.5.0) flake8 (3.5.0) flasgger (0.8.0) Flask (0.12.2) Flask-RESTful (0.3.6) Flask-Script (2.0.6) frozendict (1.2) html5lib (1.0b10) httplib2 (0.10.3) idna (2.6) ipdb (0.10.3) ipykernel (4.6.1) ipython (6.2.1) ipython-genutils (0.2.0) ipywidgets (7.0.5) iso8601 (0.1.12) isort (4.2.15) itsdangerous (0.24) jedi (0.11.0) Jinja2 (2.9.6) jsonschema (2.6.0) jupyter (1.0.0) jupyter-client (5.1.0) jupyter-console (5.2.0) jupyter-core (4.4.0) keystoneauth1 (3.2.0) kombu (4.1.0) lazy-object-proxy (1.3.1) loggly-python-handler (1.0.0) Mako (1.0.7) MarkupSafe (1.0) marshmallow (2.13.6) marshmallow-sqlalchemy (0.13.1) mccabe (0.6.1) mistune (0.7.4) monotonic (1.3) msgpack-python (0.4.8) multidict (3.2.0) mypy (0.550) mysqlclient (1.3.12) nbconvert (5.3.1) nbformat (4.4.0) netaddr (0.7.19) netifaces (0.10.6) notebook (5.2.1) oslo.config (4.13.1) oslo.i18n (3.18.0) oslo.policy (1.28.1) oslo.serialization (2.21.1) oslo.utils (3.30.0) pandocfilters (1.4.2) parso (0.1.0) passlib (1.7.1) pbr (3.1.1) pexpect (4.2.1) pickleshare (0.7.4) pika (0.11.0) pip (9.0.1) pluggy (0.6.0) positional (1.2.1) prompt-toolkit (1.0.15) psutil (5.4.1) psycopg2 (2.7.3.1) ptyprocess (0.5.2) py (1.5.2) pycodestyle (2.3.1) pycparser (2.18) pydecor (1.1.3) pydocstyle (2.1.1) pyflakes (1.6.0) Pygments (2.2.0) PyJWT (1.5.3) pylint (1.7.4) pymongo (3.5.1) pyparsing (2.2.0) pyrabbit (1.1.0) pytest-aiohttp (0.1.3) pytest-cov (2.5.1) pytest-forked (0.2) pytest-xdist (1.20.1) python-dateutil (2.6.1) python-editor (1.0.3) python-keystoneclient (3.13.0) python-memcached (1.58) pytz (2017.2) PyYAML (3.12) pyzmq (16.0.3) qtconsole (4.3.1) redis (2.10.6) requests (2.18.4) requests-futures (0.9.7) restructuredtext-lint (1.1.2) rfc3986 (1.1.0) setuptools (20.10.1) simplegeneric (0.8.1) simplejson (3.11.1) six (1.11.0) snowballstemmer (1.2.1) SQLAlchemy (1.1.14) SQLAlchemy-Utils (0.32.18) stevedore (1.27.1) stripe (1.67.0) terminado (0.8.1) testpath (0.3.1) tornado (4.5.2) tox (2.9.1) traitlets (4.3.2) typed-ast (1.1.0) typing (3.6.2) urllib3 (1.22) vine (1.1.4) virtualenv (15.1.0) wcwidth (0.1.7) webargs (1.8.1) webencodings (0.5.1) Werkzeug (0.12.2) widgetsnbextension (3.0.8) wrapt (1.10.11) yarl (0.13.0)

Submission Checklist

Thanks for submitting an issue!

Here’s a quick checklist in what to include:

• Include a detailed description of the bug or suggestion
• pip list of the virtual environment you are using
• pytest and operating system versions
• Minimal example if possible

[The-Compiler]

I haven’t really had time to follow this discussion, but FWIW I’ve had good experiences tracking down memory leaks using tracemalloc and/or pympler.muppy.

[nicoddemus]

Thanks for the patience as well!

out test suite outputs about 54000 log messages, which is about 10 MB

With the fixed tear down code, could you please check the memory consumption of your test suite with the logging plugin active?