How to use pipenv with multistage docker builds?

Good day,

I’m exploring on how to use pipenv with multi-stage docker builds. In the nutshell, the idea is to “compile” stuff in base image and only copy the resulting artifacts to the final image.

With Python is gets tricky, since you need to copy package dependencies as well.

I’ve checked out several ideas and looks like pip install --user together with setting PYTHONUSERBASE is the simplest ways to install dependencies to a side directory, e.g.:

FROM alpine AS builder
# Install your gcc, python3-dev, etc. here
apk add --no-cache python3
COPY . /src/
WORKDIR /src
ENV PYROOT /pyroot
RUN PYTHONUSERBASE=$PYROOT pip3 install --user -r requirements.txt
RUN PYTHONUSERBASE=$PYROOT pip3 install --user .

# The final image
FROM alpine
apk add --no-cache python3
ENV PYROOT /pyroot
COPY --from=builder $PYROOT/lib/ $PYROOT/lib/

(The full story)

The problem is that pipenv disregards PYTHONUSERBASE:

$ docker run --rm -ti python:3.6-alpine sh
/ # pip install --upgrade pip; pip install pipenv==2018.10.13  # skipped output
/ # mkdir /tmp/foo; cd /tmp/foo
/tmp/foo # pipenv install requests  # skipped output
/tmp/foo # PYTHONUSERBASE=/pyroot pipenv install --system --deploy
Installing dependencies from Pipfile.lock (b14837)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 5/5 — 00:00:01
/tmp/foo # ls /pyroot
ls: /pyroot: No such file or directory

I found a workaround by using pipenv lock -r and then installing requirements.txt as in my original idea, but I’m not sure this is the best way to go, particularly if I have custom (private) sources defined in my Pipefile - I don’t want to replicate their configuration into pip.

Any other ideas?

[ClaytonJY]

can confirm this works on the non-alpine python:3.7-slim-stretch base image.

I only needed a simple script on top of the pipenv deps, I added an initial base step for the env vars, and I didn’t need to muck around with symlinks, console scripts, or entrypoints, so I love how simple this ended up, only 2-3 steps per stage:

FROM python:3.7-slim-stretch AS base

ENV PYROOT /pyroot
ENV PYTHONUSERBASE $PYROOT


FROM base AS builder

RUN pip install pipenv

COPY Pipfile* ./

RUN PIP_USER=1 PIP_IGNORE_INSTALLED=1 pipenv install --system --deploy --ignore-pipfile


FROM base

COPY --from=builder $PYROOT/lib/ $PYROOT/lib/
COPY myscript.py ./

CMD ["python","myscript.py"]

many thanks for the hard work here @haizaar!

[haizaar]

@derPuntigamer It’s all here: https://tech.zarmory.com/2018/09/docker-multi-stage-builds-for-python-app.html (scroll down for pipenv version). Questions are welcome.