Problems running tuf unit tests against local version of securesyslib

Description of issue or feature request: While I was working on issue https://github.com/secure-systems-lab/securesystemslib/issues/220 I wanted to approach it that way:

1. create and use editable venv in tuf
2. install the development dependencies
3. make the necessary changes in securesyslib
4. adapt tuf with the changes made in securesyslib
5. submit a pr to securesyslib
6. submit a pr to tuf

On step 4 I ran the tests with the tox command I received ConnectionRefusedError: [Errno 111] Connection refused errors when running the unit tests. Then I stepped back, stashed all my changes and ran all of the tests again and the same happened. I will paste more details from the logs below.

This error doesn’t encounter when running the tests with tox -e py37 for example. It seems to me that I am missing something when creating my editable venv.

Steps to reproduce the error:

1. Clone securesyslib when the top of the master branch is: https://github.com/secure-systems-lab/securesystemslib/commit/caf029ea61339dc5fa9beedf230ca1d026129169
2. Clone tuf when the top of the master branch is: https://github.com/theupdateframework/tuf/commit/e095112be7d57b26406469b767b7ef8561a8f6a9
3. In the tuf folder run:

$ python3.6 -m venv . .editable_venv
$ source .editable_venv/bin/activate

1. From the securesyslib folder run pip3.6 install -r requirements-dev.txt
2. From the tuf folder run pip3.6 install -r requirements-dev.txt
3. Finally, from the tuf directory, without changing the source code of securesyslib or tuf, run:

(.editable_venv) $ cd ./tests
(.editable_venv) $ python3.6 aggregate_tests.py

and then you will see the following errors:

..................x..............................................................................E
Stderr:
DEBUG:tuf.client.updater:Adding roles delegated from 'targets'.
DEBUG:tuf.client.updater:Adding delegated role: role1.
DEBUG:tuf.client.updater:Adding roles delegated from 'targets'.
DEBUG:tuf.client.updater:Adding delegated role: role1.
DEBUG:tuf.client.updater:Interrogating repository: 'test_repository1'
DEBUG:tuf.client.updater:Interrogating repository: 'test_repository1'
DEBUG:tuf.client.updater:Found local directory for 'test_repository1'
DEBUG:tuf.client.updater:Interrogating repository: 'test_repository1'
DEBUG:tuf.client.updater:Found local directory for 'test_repository1'
DEBUG:tuf.client.updater:Found local Root file at '/home/mvrachev/src/tuf/tests/tmpi_0e23e9/TestMultiRepoUpdater_lys9irpf/test_repository1/metadata/current/root.json'
DEBUG:tuf.client.updater:Interrogating repository: 'test_repository2'
DEBUG:tuf.client.updater:Found local directory for 'test_repository2'
DEBUG:tuf.client.updater:Found local Root file at '/home/mvrachev/src/tuf/tests/tmpi_0e23e9/TestMultiRepoUpdater_lys9irpf/test_repository2/metadata/current/root.json'
DEBUG:tuf.client.updater:Interrogating mappings..{'paths': ['*1.txt'], 'repositories': ['test_repository1', 'test_repository2'], 'terminating': False, 'threshold': 1}
DEBUG:tuf.client.updater:Interrogating pattern '*1.txt'for target: 'non-existent.txt'
DEBUG:tuf.client.updater:Continue searching for relevant paths.
DEBUG:tuf.client.updater:None of the paths are relevant.
DEBUG:tuf.client.updater:Interrogating mappings..{'paths': ['*3.txt'], 'repositories': ['test_repository2'], 'terminating': True, 'threshold': 1}
DEBUG:tuf.client.updater:Interrogating pattern '*3.txt'for target: 'non-existent.txt'
DEBUG:tuf.client.updater:Continue searching for relevant paths.
DEBUG:tuf.client.updater:None of the paths are relevant.
DEBUG:tuf.client.updater:Did not find valid targetinfo for 'non-existent.txt'
DEBUG:tuf.client.updater:Interrogating repository: 'test_repository1'
DEBUG:tuf.client.updater:Found local directory for 'test_repository1'
DEBUG:tuf.client.updater:Found local Root file at '/home/mvrachev/src/tuf/tests/tmpi_0e23e9/TestMultiRepoUpdater_lys9irpf/test_repository1/metadata/current/root.json'
DEBUG:tuf.client.updater:Interrogating repository: 'test_repository2'
DEBUG:tuf.client.updater:Found local directory for 'test_repository2'
DEBUG:tuf.client.updater:Found local Root file at '/home/mvrachev/src/tuf/tests/tmpi_0e23e9/TestMultiRepoUpdater_lys9irpf/test_repository2/metadata/current/root.json'
DEBUG:tuf.client.updater:Interrogating mappings..{'paths': ['*1.txt'], 'repositories': ['test_repository1', 'test_repository2'], 'terminating': False, 'threshold': 1}
DEBUG:tuf.client.updater:Interrogating pattern '*1.txt'for target: 'non-existent.txt'
DEBUG:tuf.client.updater:Continue searching for relevant paths.
DEBUG:tuf.client.updater:None of the paths are relevant.
DEBUG:tuf.client.updater:Interrogating mappings..{'paths': ['*3.txt'], 'repositories': ['test_repository2'], 'terminating': False, 'threshold': 1}
DEBUG:tuf.client.updater:Interrogating pattern '*3.txt'for target: 'non-existent.txt'
DEBUG:tuf.client.updater:Continue searching for relevant paths.
DEBUG:tuf.client.updater:None of the paths are relevant.
DEBUG:tuf.client.updater:Did not find valid targetinfo for 'non-existent.txt'
DEBUG:tuf.client.updater:Interrogating repository: 'test_repository1'
DEBUG:tuf.client.updater:Found local directory for 'test_repository1'
DEBUG:tuf.client.updater:Found local Root file at '/home/mvrachev/src/tuf/tests/tmpi_0e23e9/TestMultiRepoUpdater_lys9irpf/test_repository1/metadata/current/root.json'
DEBUG:tuf.client.updater:Interrogating repository: 'test_repository2'
DEBUG:tuf.client.updater:Found local directory for 'test_repository2'
DEBUG:tuf.client.updater:Found local Root file at '/home/mvrachev/src/tuf/tests/tmpi_0e23e9/TestMultiRepoUpdater_lys9irpf/test_repository2/metadata/current/root.json'
DEBUG:tuf.client.updater:Interrogating mappings..{'paths': ['*1.txt'], 'repositories': ['test_repository1', 'test_repository2'], 'terminating': True, 'threshold': 1}
DEBUG:tuf.client.updater:Interrogating pattern '*1.txt'for target: 'bad3.txt'
DEBUG:tuf.client.updater:Continue searching for relevant paths.
DEBUG:tuf.client.updater:None of the paths are relevant.
DEBUG:tuf.client.updater:Interrogating mappings..{'paths': ['*3.txt'], 'repositories': ['test_repository2'], 'terminating': True, 'threshold': 1}
DEBUG:tuf.client.updater:Interrogating pattern '*3.txt'for target: 'bad3.txt'
DEBUG:tuf.client.updater:Found a match for 'bad3.txt'
DEBUG:tuf.client.updater:Retrieving targetinfo for 'bad3.txt' from repository...
DEBUG:tuf.client.updater:Adding updater for 'test_repository2'
DEBUG:tuf.client.updater:Adding roles delegated from 'targets'.
DEBUG:tuf.client.updater:Adding delegated role: role1.
INFO:tuf.download:Downloading: 'http://localhost:30002/metadata/2.root.json'
DEBUG:tuf.download:url: http://localhost:30002/metadata/2.root.json
DEBUG:tuf.download:session index: http+localhost
DEBUG:tuf.download:Reusing session for http+localhost
ERROR:tuf.download:Could not download URL: 'http://localhost:30002/metadata/2.root.json'
Traceback (most recent call last):
  File "/home/mvrachev/src/tuf/.editable_venv/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/mvrachev/src/tuf/.editable_venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/home/mvrachev/src/tuf/.editable_venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

There are more exceptions after that, but this is the initial one. Then the tests are failing with:

======================================================================
ERROR: test_13__get_file (test_updater.TestUpdater)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mvrachev/src/tuf/tests/test_updater.py", line 1797, in test_13__get_file
    file_type, file_size, download_safely=True)
  File "/home/mvrachev/src/tuf/tuf/client/updater.py", line 1705, in _get_file
    raise tuf.exceptions.NoWorkingMirrorError(file_mirror_errors)
tuf.exceptions.NoWorkingMirrorError: No working mirror was found:
  'localhost:41481': ConnectionError(MaxRetryError("HTTPConnectionPool(host='localhost', port=41481): Max retries exceeded with url: /tmplmmecg9h/TestUpdater_1rv6zrui/repository/metadata/targets.json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8c2ff6ae48>: Failed to establish a new connection: [Errno 111] Connection refused',))",),)

Stderr:
DEBUG:tuf.client.updater:Adding roles delegated from 'targets'.
DEBUG:tuf.client.updater:Adding delegated role: role1.
INFO:tuf.download:Downloading: 'http://localhost:41481/tmplmmecg9h/TestUpdater_1rv6zrui/repository/metadata/targets.json'
DEBUG:tuf.download:url: http://localhost:41481/tmplmmecg9h/TestUpdater_1rv6zrui/repository/metadata/targets.json
DEBUG:tuf.download:session index: http+localhost
DEBUG:tuf.download:Reusing session for http+localhost
ERROR:tuf.download:Could not download URL: 'http://localhost:41481/tmplmmecg9h/TestUpdater_1rv6zrui/repository/metadata/targets.json'
Traceback (most recent call last):
  File "/home/mvrachev/src/tuf/.editable_venv/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/mvrachev/src/tuf/.editable_venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/home/mvrachev/src/tuf/.editable_venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

Expected behavior:

I want to be able to run the tuf unit tests against my local version of securesyslib without any problems.

[lukpueh]

Looks like https://github.com/theupdateframework/tuf/commit/842f8432102b24f5c6a62f2521591a66c2052ca9 indeed broke this (as bisected by @MVrachev 👍).

More specifically, running the test http server as python -m tests.simple_server might be a smart idea if the “tuf” directory is the only (or at least first) directory on the path that has a “tests” subdirectory. But this is not the case if we install securesystemslib in editable mode before you install tuf. 🤦

Moreover, passing a pipe to the subprocess, but not reading from it conceals helpful error messages like: No module named tests.simple_server.

-> And thus the http timeouts in the build logs that break the tests.

So, if we manage to actually start our test http server where needed, the tip of the tuf develop branch indeed works with the tip of the securesystemslib master branch.

A quick way to fix this would be to just run the script as python simple_server.py (which we already do in a few places). The downside is that it requires running the tests from within tests directory (which we already require).

A more flexible way would be assign the path to a tests wide-constant in __init__.py, e.g. as:

SIMPLE_SERVER = os.path.join(os.path.dirname(os.path.realpath(__file__)), "simple_server.py").

I’m curious what others think.

Regardless, I think we should remove the stderr PIPEs to better debug these problems in the future.

[trishankatdatadog]

Good work, guys!