Managing abstract dependencies of setup.py

You say pipenv is a marriage of Pipfile, Pip, & Virtualenv. But there’s one depency related task that’s left to be done manually, that’s managing setup.py’s install_requires.

As @dstufft wisely says in his article, abstract depencies and concrete depencies are used in different contexts. But when you do your code, you’re managing them at the same time. If you change a depency, you’ll have to modify both setup.py or the Pipfile. I believe, it’d be nice to make it possible to manage the abstract dependencies using pipenv.

From a pipenv user perspective, I mean something that would look like:

# in Pipfile → [dev-packages]
pipenv install --dev pytest mock delegator.py toml Sphinx
# in Pipfile → [packages]
pipenv install pew>=0.1.26
# in setup.py → install_requires
pipenv install --abstract virtualenv pip pew>=0.1.26

Maybe could we extend the syntax so it has three options:

# in Pipfile → [dev-packages]
pipenv install --dev pytest mock delegator.py toml Sphinx
# in Pipfile → [packages] (only)
pipenv install --concrete foobar
# in Pipfile → [packages] and setup.py → install_requires
pipenv install pew>=0.1.26
# in setup.py → install_requires (only)
pipenv install --abstract virtualenv pip 

That way, pipenv is the entry point tool to manipulate everything related to dependency management and associated virtualenv. And from a trouple marriage, it’ll be a marriage of four tools 😁.

Now, on the question on how to implement this, I’m not much in favour of automatic edition of a .py file, because there are just too many ways it can go wrong. But instead, store the abstract depencies externally in a new [abstract-packages] section of Pipfile if it’s ok with @pypa, or in an abstract-requirements.txt file or any other smarter solution I haven’t think of yet.

What do you think?

[guyzmo]

well, the issue I’m posting here has 3 questions:

1. do you think pipenv would be the right tool to manage abstract dependencies?
2. what command line API would it be nice to expose?
3. how to implement the mutability of those abstract dependencies?
   • in Pipfile, by negociating with pypa?
   • in some file we parse with something looking like the following in setup.py? with open('abstract-req') as r: install_requires=r.read().splitlines()
   • by modifying the setup.py file, like it’s a config file (it wouldn’t be hard to parse it using ast, find a given variable and mutate the RHS of that variable assignation).

So, to me the first two questions are the most important ones we should discuss here, and on which a decision shall be made. Only then, we should discuss the third question, which is, IMHO, a technical detail. If people of the Pipfile project decide it’s nice to also manage abstract dependencies in Pipfile, it’s great, if they don’t, then we can figure out another way.

I did not say, but I’d be happy to write a patch to implement whatever gets decided in this discussion, and ask the Pipfile project for permission, if contributors of this project share my opinion that it’d be a great adition to pipenv, and help decide what’s the best approach for implementing the abstract dependencies in a way that’s maintainable by a tool like pipenv.

[pcattori]

@guyzmo yes i am planning on handling abstract vs concrete dependencies. specifically, the add command will update the entry in the project’s json config (which is doing the job that setup.py was doing) and though I haven’t added all the functionality yet, it will allow users to specify abstract vs concrete dependencies as necessary and will keep concrete deps in a .lock file (ala Pipfile).

I have briefly considered using existing standard files like Pipfile and the new pyproject.toml but not really sure what to use. It seems like Pipfile is the replacement for requirements.txt, not setup.py. I think pyproject.toml might be what I want, but for short-term prototyping I haven’t devoted the research to answer this question and instead just using something I’m familiar with (json) and something I have complete control over (my own format for that json file). Now that I’ve got basic functionality in, I’m trying to answer this question.

haven’t explored setup.cfg much yet, which is why I asked about anything you’ve learned on the matter. sounds like maybe setup.cfg is another candidate to replace my custom snek.json