processor plot tests succeed but return error

Describe the bug

Running tests for qutip 4.6.2, on linux (Debian unstable), all tests pass successfully, but overall tests return error code 1, which would be treated as test failure. For instance, running tests via python3 -c "import qutip.testing; qutip.testing.run()"

============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.0.2, py-1.10.0, pluggy-0.13.0 -- /usr/bin/python3.9
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/drew/projects/misc/build/qutip/.pybuild/cpython3_3.9_qutip/build-test/.hypothesis/examples')
rootdir: /home/drew/projects/misc/build/qutip
plugins: cov-2.10.1, mpi-0+unknown, doctestplus-0.9.0, remotedata-0.3.2, asyncio-0.14.0, filter-subpackage-0.1.1, arraydiff-0.3, astropy-header-0.1.2, hypothesis-5.43.3, openfiles-0.5.0, xvfb-1.2.0
collecting ... collected 1996 items / 67 deselected / 1 skipped / 1928 selected

qutip/tests/test_basis_transformation.py::test_transformation_to_eigenbasis_is_reversible[2-real_hermitian] PASSED [  0%]
qutip/tests/test_basis_transformation.py::test_transformation_to_eigenbasis_is_reversible[2-imaginary_hermitian] PASSED [  0%]
qutip/tests/test_basis_transformation.py::test_transformation_to_eigenbasis_is_reversible[2-complex_hermitian] PASSED [  0%]

...
qutip/tests/test_wigner.py::test_wigner_fft_comparse_ket PASSED          [ 99%]
qutip/tests/test_wigner.py::test_wigner_fft_comparse_dm PASSED           [ 99%]
qutip/tests/test_wigner.py::test_wigner_clenshaw_iter_dm PASSED          [ 99%]
qutip/tests/test_wigner.py::test_wigner_clenshaw_sp_iter_dm PASSED       [100%]XIO:  fatal IO error 0 (Success) on X server ":1029"
      after 201 requests (201 known processed) with 4 events remaining.

As far as I can tell this XIO error is related to the xvfb module (pytest-xvfb) used to manage graphical tests without display.

Some relevant discussion is found at https://github.com/The-Compiler/pytest-xvfb/issues/11 . They indicate there that this error can be triggered if windows managed by Xvfb are not properly closed before completing tests. So it’s likely it’s related to tests using matplotlib.

The only test using matplot lib is testPlot in test_processor.py. Sure enough, if testPlot is skipped then test_processor.py passes successfully without triggering the XIO error.

To Reproduce

Run test_processor.py directly:

python3 -m pytest /usr/lib/python3/dist-packages/qutip/tests/test_processor.py -v

The terminal output is

$ python3 -m pytest /usr/lib/python3/dist-packages/qutip/tests/test_processor.py -v
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.9.2, pytest-6.0.2, py-1.10.0, pluggy-0.13.0 -- /usr/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/drew/.hypothesis/examples')
rootdir: /usr/lib/python3/dist-packages/qutip/tests, configfile: pytest.ini
plugins: cov-2.10.1, mpi-0+unknown, doctestplus-0.9.0, remotedata-0.3.2, asyncio-0.14.0, filter-subpackage-0.1.1, arraydiff-0.3, astropy-header-0.1.2, hypothesis-5.43.3, openfiles-0.5.0, xvfb-1.2.0
collected 11 items                                                                                                                                                                                                

../../usr/lib/python3/dist-packages/qutip/tests/test_processor.py::TestCircuitProcessor::test_modify_ctrls PASSED                                                                                           [  9%]
../../usr/lib/python3/dist-packages/qutip/tests/test_processor.py::TestCircuitProcessor::test_save_read PASSED                                                                                              [ 18%]
../../usr/lib/python3/dist-packages/qutip/tests/test_processor.py::TestCircuitProcessor::test_id_evolution PASSED                                                                                           [ 27%]
../../usr/lib/python3/dist-packages/qutip/tests/test_processor.py::TestCircuitProcessor::test_id_with_T1_T2 PASSED                                                                                          [ 36%]
../../usr/lib/python3/dist-packages/qutip/tests/test_processor.py::TestCircuitProcessor::testPlot PASSED                                                                                                    [ 45%]
../../usr/lib/python3/dist-packages/qutip/tests/test_processor.py::TestCircuitProcessor::testSpline PASSED                                                                                                  [ 54%]
../../usr/lib/python3/dist-packages/qutip/tests/test_processor.py::TestCircuitProcessor::testGetObjevo PASSED                                                                                               [ 63%]
../../usr/lib/python3/dist-packages/qutip/tests/test_processor.py::TestCircuitProcessor::testNoise PASSED                                                                                                   [ 72%]
../../usr/lib/python3/dist-packages/qutip/tests/test_processor.py::TestCircuitProcessor::testMultiLevelSystem PASSED                                                                                        [ 81%]
../../usr/lib/python3/dist-packages/qutip/tests/test_processor.py::TestCircuitProcessor::testDrift PASSED                                                                                                   [ 90%]
../../usr/lib/python3/dist-packages/qutip/tests/test_processor.py::TestCircuitProcessor::testChooseSolver PASSED                                                                                            [100%]

================================================================================================ warnings summary =================================================================================================
/usr/lib/python3/dist-packages/_pytest/stepwise.py:122
  /usr/lib/python3/dist-packages/_pytest/stepwise.py:122: PytestCacheWarning: could not create cache path /usr/lib/python3/dist-packages/qutip/tests/.pytest_cache/v/cache/stepwise
    self.config.cache.set("cache/stepwise", [])

/usr/lib/python3/dist-packages/_pytest/cacheprovider.py:396
  /usr/lib/python3/dist-packages/_pytest/cacheprovider.py:396: PytestCacheWarning: could not create cache path /usr/lib/python3/dist-packages/qutip/tests/.pytest_cache/v/cache/nodeids
    config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================================= 11 passed, 2 warnings in 0.84s ==========================================================================================
XIO:  fatal IO error 0 (Success) on X server ":1029"
      after 366 requests (352 known processed) with 19 events remaining.
$ echo $?
1

Expected behavior

The XIO error should not be triggered. Exit value should be 0 not 1.

Your Environment

Linux (Debian unstable)

$ python3 -c "import qutip; qutip.about()"

QuTiP: Quantum Toolbox in Python
================================
Copyright (c) QuTiP team 2011 and later.
Current admin team: Alexander Pitchford, Nathan Shammah, Shahnawaz Ahmed, Neill Lambert, Eric Giguère, Boxi Li, Jake Lishman and Simon Cross.
Board members: Daniel Burgarth, Robert Johansson, Anton F. Kockum, Franco Nori and Will Zeng.
Original developers: R. J. Johansson & P. D. Nation.
Previous lead developers: Chris Granade & A. Grimsmo.
Currently developed through wide collaboration. See https://github.com/qutip for details.

QuTiP Version:      4.6.2
Numpy Version:      1.19.5
Scipy Version:      1.7.0
Cython Version:     0.29.21
Matplotlib Version: 3.3.4
Python Version:     3.9.2
Number of CPUs:     4
BLAS Info:          OPENBLAS
OPENMP Installed:   True
INTEL MKL Ext:      False
Platform Info:      Linux (x86_64)
Installation path:  /usr/lib/python3/dist-packages/qutip

Additional context

The first thing to check for is whether plots are being closed. But TestPlot already uses plt.clf(). Evidently it is not enough. Replacing plt.clf() with plt.close('all') seems to work, allowing tests to be registered as successful without error.

[hodgestar]

@drew-parsons I imagine MPLBACKEND=agg avoids the issue by not using X at all. I have used it in the past for matplotlib tests, and it’s not a bad idea perhaps to switch the QuTiP visualization tests to the agg backend, but it’s not something I’m going to tackle right now I don’t think (too many other big QuTiP changes up in the air at the moment).

[drew-parsons]

Curious cross-reference from mdanalysis. Their tests trigger the same error (likely again because of pytest-xvfb),

XIO:  fatal IO error 0 (Success) on X server ":1029"

Their error is not so simple to clear, since they use the object oriented matplotlib API (e.g. using AxesSubplot matplotlib.axes._subplots.AxesSubplot) rather than matplotlib.pyplot. So they don’t have access to plt.close() and axes.cla() doesn’t prevent the error, probably for the same reason that plt.clf() didn’t work for qutip.

One point they have which is worth noting: they recommend using the environment variable MPLBACKEND=agg for headless CI testing. And sure enough the XIO error does not occur with mdanalysis if MPLBACKEND=agg is set.

For qutip the plt.close(fig) patch is still the best (more robust) solution though.