Migrating module dependencies to proper PIP packages

Just figured I should document this… At first, I tried to keep data access layers as minimal as possible (i.e. ideally in a single file), but it really seems to make more trouble than convenience:

• somewhat annoying to keep track of that in the config
• dependencies need to be installed separately via requirements.txt
• non-transparent to static analysis, and to check it with mypy, still need a proper environment

With proper setup.py

• one can simply git pip install <github repo url>, without cloning anything in a temporary location
• you can use virtualenv, if you prefer, to avoid mixing HPI dependencies and rest of the packages
• with --editable install, you can develop as if it was a symlink
• and you still can manually symlink code if you prefer for some reason

Basically, the only downside is maintaining setup.py. I keep it very minimal, merely with the package name, py.typed file for MyPy and the dependencies, since I’m not planning to upload to PIP (and no one really looks at the classifiers/reads documentation on PyPi anyway).

UPD: also it seems that for proper mypy integration it’s necessary to have __init__.py (see my comment here https://github.com/karlicoss/endoexport/blob/be5084aa45aaac206ff86624244f40c08b439340/src/endoexport/__init__.py#L1-L5 ). If anyone knows how to get around this, please let me know!

Related discussions:

• https://github.com/karlicoss/rexport/issues/9
• https://github.com/karlicoss/ghexport/issues/3

Migrated:

• https://github.com/karlicoss/emfitexport
• https://github.com/karlicoss/endoexport
• https://github.com/karlicoss/rescuexport
• @seanbreckenridge started with PIP package structure straightaway, for example: https://github.com/seanbreckenridge/pushshift_comment_export

Tagging @seanbreckenridge as you were interested in that too, let me know if you have some thoughts!

[karlicoss]

All right, I converted a few more modules:

• https://github.com/karlicoss/instapexport/pull/3
• https://github.com/karlicoss/rexport/pull/11
• https://github.com/karlicoss/pockexport/pull/4
• https://github.com/karlicoss/ghexport/pull/5
• https://github.com/karlicoss/hypexport/pull/6

, and added HPI support with a backwards compatible fallback https://github.com/karlicoss/HPI/pull/83

I’ve tested for a bit and seems fine, but I’ll leave it for a couple of days just to be more sure, and then update the docs, and merge everything in one go.

[karlicoss]

Right! So, I added a thing to HPI that can parse the requirements (via ast module, so there won’t be any import errors or something). So now it’s possible to use something like hpi module install my.endomondo, and it will install the required dependencies. Or run hpi module requires my.endomondo which will dump the dependencies in stdout (in case the user has some custom install process for dependencies). Seems like a reasonable compromise without forcing any special plugin architecture, guess okay to close this now.