import-style prevents vendoring

I just tried to vendor TUF into pip (in other words embed the tuf sources inside pip sources) and it fails (even with #1160 fixed) because of the import style used in TUF source code itself:

import tuf.download

Using this (instead of relative imports) means tuf needs to be installed to python path for the imports to work.

I’ve looked at the dozen other projects pip vendors and the system seems to cope with two styles:

1. Most projects use relative imports like

from . import treebuilders
from .treebuilders.base import Marker

1. The other method is

from toml.decoder import InlineTableDict

The second style works because vendoring (the tool) modifies import NAME to from vendored.path import NAME and from NAME.module import something to from vendored.path.NAME.module import something. I assume this is mostly done so vendored dependencies of vendored projects work (so e.g. vendored tuf can “import securesystemslib”). In any case the TUF style cannot be automatically modified as far as I can tell.

I’ll try to add reproduction instructions that don’t require my pip branch…

[jku]

All of TUF dependencies are already pip dependencies (so succesfully vendored) except SSL and iso8601 (which is a single file library).

If we have to choose between relative imports and importing names (functions, globals, etc) directly, I too vote for the former. At least this requirement will make it easy to choose an import style convention. 😃

I’m researching more clever alternatives as both of these options mean a massive PR: ~700 actual code lines would need to change and the possibility of a namespace mistake is real. I’ll try to have a possible solution for discussion by tomorrow.

[lukpueh]

Regarding the other pain points you mentioned …

repository_tool, developer_tool and repository_lib could live somewhere where they would not be confused for library parts (assuming they are not parts of the library like I think): tuf/scripts/ maybe or a directory of their own?

Agreed but probably not worth to change before the planned refactor? Definitely something to consider together with #1136 and #881. Maybe this warrants a separate over-arching module architecture design issue? Also https://github.com/theupdateframework/tuf/issues/1134#issuecomment-690091176 seems related.

unittest_toolbox should just be moved to tests/

Agreed but IMO low-prio.unittest_toolbox should probably also be subjected to a revision.

download and mirrors should probably be moved to tuf/client/

Agreed but see both comments above and #1135.

pylint does not understand the tuf import style at least in the unused-imports check: as a result we have some unused internal imports in our code.

I know. 😕 I’ve been meaning to at least submit an issue for this for a long time (I will do this right now). FWIW, I’ve fixed this a long time ago in in-toto and have created a ticket in securesystemslib, which both adopted their linter config from tuf (see https://github.com/secure-systems-lab/securesystemslib/issues/243).

we are not very consistent in imports: lots of files use e.g. securesystemslib modules without importing them (and trust that someone else is importing them)

I know. 😕 I think a sensible linter config would catch this (see above).