Poetry Buildpack Proposal

Hi, I recently made a repository that runs on Binder and installs a python environment via Poetry, and I’m considering writing a buildpack for Poetry and doing a PR.

To me, Poetry is a much nicer way to specify environments than conda or pip requirement files, as it properly concretises and hashes the dependencies and also tracks the version of packages installed from git repositories via the commit.

The contribution documentation says that when looking at adding a new buildpack you should consider “How easy it is to use a given setup without support from repo2docker natively.”, and the answer is that it’s very easy and can be done in three or four lines.

However, given how poetry environments are completely concretised via the lock files and will not change, it would be nice to include poetry as a build pack so that the environment layer doesn’t have to be rebuilt each time. Currently running the poetry install command in postBuild means that all the python packages have to be installed any time the repository updates even if the dependencies have not changed.

If the PR for this buildpack could be accepted then I’ll start work on this in March.

P.S. The feature request template didn’t work for me, so I’m not sure what to include here.

[aeturrell]

+1 to introducing support for poetry. My sense is that it has really taken off since this issue first opened.

[RobertRosca]

@betatim alright, have some time to start looking at this now, sorry for the (long) delay!

What is your impression of the community around poetry? Have they been around for a while and look like a project growing in support and popularity? Or something that people got excited about and now not so excited about maintaining for the long term?

I think the project is going to keep growing, it’s a very nice tool for locking down version dependencies in a deterministic way, it has a couple of benefits over pipenv and the development is a bit more active than pipenv as well (not that active development is the best metric of quality).

* how to deal with people wanting to use a Python version which isn't the default we install

* can we use conda to install the Python executable instead of using pyenv?

I’ve had a look through the pipenv buildpack and I see your point, using a single python provider would make sense. Personally I don’t really like using conda very much so I’d lean more towards a pure pyenv implementation, but given that conda buildpack already serves as the base class for pipenv, in this case I think it makes sense to stick with conda then.

Managing the python version via conda shouldn’t be a problem for this, poetry’s use of pyenv is only a suggestion.

* what happens if a poetry file doesn't/does specify a jupyter notebook version?

Good question, didn’t think of that!

For python3 where you’d have a single environment containing the kernel and jupyter, there’s a simple solution, you can run poetry add jupyter before the environment installation command. This would either (1) add a version of jupyter compatible with the existing environment or (2) say that jupyter is already available and no changes are needed.

For Python 2 based environments we already have a nice setup of creating one environment in which the notebook server runs (with its dependencies and based on Python 3) and a second environment in which the notebook kernel runs (with the dependencies specified by the repository and using Python 2). When a repository requests Python 3 we currently don’t have this two environment setup, but in the long term it would probably make sense to switch to it. There are pros and cons to doing this (I will try and find the relevant issue).

Splitting the environments definitely has its benefits, but in this case it could complicate things in some situations, especially if you use conda to manage the notebook server environment but use poetry to manage the kernel, most of the time it probably won’t matter but if the repo uses plugins/extensions a missmatch between the versions of packages in the notebook server environment and the kernel environment could cause issues.

Because repo2docker provides some “ready made infrastructure” like the Jupyter notebook server (repositories don’t have to explicitly list this to get it) the mental model I have is that repo2docker creates an environment consisting of the requested Python version and support packages and then we “update” the environment with the requests from the repository. A good example is the conda buildpack where we have an explicit conda create ... command not controlled by the repository followed by conda update ... with input from the repository. This works pretty well also with Pipfiles but there are some gnarly edge cases. Mostly around dealing with conflicting requests.

Using conda to deliver the Python executable has served us well and I think we should keep using this mechanism. It means we can keep sharing docker image layers with other build packs, we already maintain this infrastructure, there is no compiling involved as we can get binary packages.

Aha, I see your point. What kind of conflict issues were there with the Pipfile buildpack?

Overall I think this would be an interesting addition. There are a few edge cases and warts that we learnt about while adding the Pipfile buildpack that need some brain power to figure out/improve on and we should make this investment.

Just to get familiar with it I’ll start by modifying the pipfile buildpack to work with poetry, they’re similar enough that this shouldn’t take too long, I’ll try to make improvements along the way if anything obvious stands out to me.

Do you have any issues I cold look at related to the pipfile edge cases? I can think of a few but seeing some examples would be very helpful.

ps. I think we should judge buildpacks/package management approaches around their convenience and popularity (growing and looks like it has support from its maintainers, is it a stable tool humans can rely on for their work), not so much via “It is better at doing X than option Y”. Usually the latter claims get us into a discussion that starts with “Well, actually Y can do X just as well if you bothered to read the documentation!” and then spirals out of control into an argument where people are trying to convince each other of their respective superiority. This tends to stress everyone out and isn’t terribly productive in getting new features built 😃

Yeah… in these kind of situations I think that having requests for something to be added is enough.

When I click the “feature request” button after clicking “new issue” on GitHub I end up at https://github.com/jupyter/repo2docker/issues/new?assignees=&labels=needs%3A+discussion&template=feature_request.md&title= Do you know where you found the link you posted? Seems like we should update that to be more like the one above (unless that also doesn’t work for you, in which case we need more investigating).

I found the link on the docs page here: https://repo2docker.readthedocs.io/en/latest/contributing/contributing.html?highlight=template%3Dfeature_request.md%26title#process-for-making-a-contribution

Suggest a new feature. We know that there are lots of ways to extend repo2docker! If you’re interested in adding a feature then please open a feature request. That issue template will ask you a few questions described in detail below.