New feature idea: create a "minimal" pip freeze option

What’s the problem this feature will solve? When I create the requirements file with the pip freeze command (pip freeze > requirements.txt) there are many packages, most of which are dependencies of other packages.

Describe the solution you’d like An additional option like pip freeze --min > requirements.txt which writes only the top-level packages needed in the project. I give an example.

I create a project and install the tensorflow and keras packages. What I would like is a command that only provides these 2 packages, because they already install the other required packages (their dependencies).

Current situation: pip install keras tensorflow && pip freeze > requirements.txt

requirements.txt file:

absl-py==0.10.0
astunparse==1.6.3
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
freeze==3.0
gast==0.3.3
google-auth==1.22.1
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.32.0
h5py==2.10.0
idna==2.10
importlib-metadata==2.0.0
Keras==2.4.3
Keras-Preprocessing==1.1.2
Markdown==3.3
numpy==1.18.5
oauthlib==3.1.0
opt-einsum==3.3.0
protobuf==3.13.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyYAML==5.3.1
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scipy==1.5.2
six==1.15.0
tensorboard==2.3.0
tensorboard-plugin-wit==1.7.0
tensorflow==2.3.1
tensorflow-estimator==2.3.0
termcolor==1.1.0
urllib3==1.25.10
Werkzeug==1.0.1
wrapt==1.12.1
zipp==3.3.0

With the new feature: pip install keras tensorflow && pip freeze --min > requirements.txt

requirements.txt file:

Keras==2.4.3
tensorflow==2.3.1

Additional context I wrote a simple python script that achieves this goal. Maybe this can help with development.

[pfmoore]

For what it’s worth, here’s a script that returns the information the OP wanted. Recent versions of Python only, but it should be possible to backport.

import importlib.metadata
from packaging.requirements import Requirement
import re

def normalize(name):
    return re.sub(r"[-_.]+", "-", name).lower()

dependencies = set()
all_packages = set()
for dist in importlib.metadata.distributions():
    name = normalize(dist.metadata["name"])
    all_packages.add(name)
    if dist.requires:
        for req in dist.requires:
            dep = normalize(Requirement(req).name)
            dependencies.add(dep)

top_level = all_packages - dependencies
for name in sorted(top_level):
    print(f"{name}=={importlib.metadata.version(name)}")

And before anyone comments, this is no more an argument for not supporting the OP’s request than the fact that you can do

import importlib.metadata

for dist in importlib.metadata.distributions():
    print(f"{dist.metadata['name']}=={dist.version}")

means that pip freeze is not needed. In both cases the script probably solves 90% of the problem, building the functionality into pip is for people who need the remaining 10%.

[pfmoore]

Agreed. For me, the feature is useful for the purpose of rebuilding a description of “what the project depends on”. If I were developing a package, that should be in install_requires. If I’m developing an application, it’s in a (manually maintained) requirements.txt (or a requirements.in in a pip-tools world). But if I’m doing an adhoc analysis of some data, or writing a script to automate/integrate some tools, or something like that, then it’s only really captured in “the virtual environment I used while developing”. That’s what pip freeze --min would recover, for me. And yes, sometimes it’s context dependent, and sometimes it’s not quite what I need. But it’s always better in practice than pip freeze for that purpose, because dependencies are mostly noise in that context (i.e., the context where I explicitly don’t want a lock file).

This is a hard problem, with many possible solutions. There are tools like pip-tools and pipenv that solve it for certain workflows, at the cost of being inappropriate for others. I don’t think that pip needs to compete with them ("if you need pip-tools, you know where to find it). But I do think that pip should enable users to build their own workflows.

If people are adamant that supporting this use case is too awful to contemplate, how about simply sorting the output of pip freeze so that packages are in increasing order of “things that depend on them”? That would be relatively easy to do, and would at least help with the OP’s use case.

To clarify further, pip freeze itself is a bit of a weird case. It’s not part of pip’s core functionality of installing packages, it’s more in the “environment management” area. It can easily be implemented in 3rd party code, as I demonstrated above. It’s only useful in virtual environments that don’t have include-system-site-packages set. So before we start arguing that enhancements to pip freeze are inappropriate, maybe we could be clear on why pip freeze exists at all?

What I will say, though, is that in my own personal experience I have essentially never needed or used pip freeze. I have, however, wanted some variation on the OP’s pip freeze --min on many, many occasions. If nothing else, this discussion prompted me to write my own script, which I will now keep and use for the moment, until pip gets the equivalent functionality.