How to handle production only dependencies. psycopg2

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Question

One issue with poetry, which also is a problem in something like Pipenv, are dependencies that should only be installed on production. In my case, this is psycopg2. When doing development/testing, it is easy enough to install psycopg2-binary under [tool.poetry.dev-dependencies] but the problem is, that the normal dependencies are not “production only”. They are “all environemnts”. So if I have:

[tool.poetry.dependencies]
psycopg2 = "*"

[tool.poetry.dev-dependencies]
psycopg2-binary = "*"

I still get the following error when installing on my local machine, where postgres is not installed.

$ poetry install
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file


Package operations: 2 installs, 0 updates, 0 removals

  - Installing psycopg2 (2.8.3)
                                                                                                                                                               
[EnvCommandError]                                                                                                                               
Command ['/home/shawn/.local/share/virtualenvs/spedsa-nMjnt-5Z/bin/python', '-m', 'pip', 'install', '--no-deps', 'psycopg2==2.8.3'] errored with the follow        
ing return code 1, and output:                                                                                                                               
Collecting psycopg2==2.8.3                                                                                                                                   
  Using cached https://files.pythonhosted.org/packages/5c/1c/6997288da181277a0c29bc39a5f9143ff20b8c99f2a7d059cfb55163e165/psycopg2-2.8.3.tar.gz              
    ERROR: Command errored out with exit status 1:                                                                                                           
     command: /home/shawn/.local/share/virtualenvs/spedsa-nMjnt-5Z/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cz2                
8dqk0/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-cz28dqk0/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.rea  
d().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info                          
         cwd: /tmp/pip-install-cz28dqk0/psycopg2/                                                                                                            
    Complete output (23 lines):                                                                                                                              
    running egg_info                                                                                                                                         
    creating pip-egg-info/psycopg2.egg-info                                                                                                                  
    writing pip-egg-info/psycopg2.egg-info/PKG-INFO                                                                                                          
    writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt                                                                          
    writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt                                                                                  
    writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'                                                                                       
                                                                                                                                                             
    Error: pg_config executable not found.                                                                                                                   
                                                                                                                                                             
    pg_config is required to build psycopg2 from source.  Please add the directory                                                                           
    containing pg_config to the $PATH or specify the full executable path with the                                                                           
    option:                                                                                                                                                  
                                                                                                                                                             
        python setup.py build_ext --pg-config /path/to/pg_config build ...                                                                                   
                                                                                                                                                             
    or with the pg_config option in 'setup.cfg'.                                                                                                             
                                                                                                                                                             
    If you prefer to avoid building psycopg2 from source, please install the PyPI                                                                            
    'psycopg2-binary' package instead.                                                                                                                       
                                                                                                                                                             
    For further information please check the 'doc/src/install.rst' file (also at                                                                             
    <http://initd.org/psycopg/docs/install.html>).                                                                                                           
                                                                                                                                                             
    ----------------------------------------                                                                                                                 
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.                                              
                                                                                                                                                               
install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

So i can’t leave psycopg2 to only be installed on production and the binary version otherwise. How should I handle this? Others have asked to have a prod-dependencies or something similar. But it doesnt seem like that will ever happen. Nor in Pipenv.

How should I make this work? Assuming both environments are on linux.

[bittner]

I hit the same issue, albeit for a different use case. (Or maybe it’s really the same use case, but we describe it differently.) – Here it goes:

For local development I use docker-compose, hence my dependencies are isolated in a Docker image (via the project’s Dockerfile). My developer machine is minimal, my system dependencies for psycopg2 (or for mysql-connector, depending on the database engine) are not installed locally, but only in the Dockerfile.

This is a perfect setup for working with tools like pip-compile or pipenv if I want to generate or update my project dependencies “from outside”, i.e. running one of those commands on my developer machine locally (to create, say, a requirements.txt file). Those tools resolve the dependencies without requiring me to have runtime dependencies of the packages installed.

Poetry seems to work differently. It fails, like above, because I don’t have the database client libraries (and headers) installed. This makes it impossible for me to use Poetry. It doesn’t support my usual development workflow. – And it’s not an unusual workflow, is it?

[abn]

@autoferrit you could make use of extras as a potential solution for your scenario.

poetry add --dev psycopg2-binary
poetry add --extras=production psycopg2

And when installing in your production environment, you can do the following.

# if installing from a package index
pip install somepackge[production]

# if installing from source
poetry install --no-dev --extras=production somepackage

However, this means that if installed without extras your package/application may not work as expected. An operation solution might be better for your particular case of using psycopg2. For example, using a private index (nexus etc.) that has a securely built psycopg2-binary wheel; or you can consider pre-creating a virtual environment and installing psycopg2-binary prior to using poetry install. Hope this helps.

EDIT: Just realised that this might not solve your particular issue as poetry installs optional packages too when installing in editable mode.