Poetry and CI builds -- best practices, knowledge share (travisci), and discussion

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Question

Hi! I’ve been working with poetry recently and am happy that I can publish packages to pypi or my own hosted package repository with it. I would like to share my current setup with TravisCI in hopes that it could help others. I would also like to learn from those who have more experience with poetry to get tips on how I might make this setup better, or see how it works with other continuous integration platforms, or other more in depth CI/CD requirements.

My requirements for travis-ci are fairly common.

1. Use poetry to install my project dependencies and dev dependencies.
2. Run lint and unit tests for my project.
3. When the project is git tagged, to build and publish my project to my companies internal pypi registry.

.travis.yml

language: python
python:
- 3.6
env:
  global:
  - secure: "<encrypted MYPYPI_USER=username>"
  - secure: "<encrypted MYPYPI_PASS=p@ssword>"
before_install:
- pip install poetry
install:
- poetry install
script:
- poetry run flake8 my_package test
- poetry run coverage run --source=my_package -m unittest discover -b
before_deploy:
- poetry config repositories.mypypi http://mypypi.example.com/simple
- poetry config http-basic.mypypi $MYPYPI_USER $MYPYPI_PASS
- poetry build -f sdist
deploy:
  provider: script
  script: poetry publish -r mypypi
  skip_cleanup: true
  on:
    tags: true

I have in the past used the built in travis pypi deployment, but it requires a setup.py ( which I don’t have anymore! 🙌). So instead I’m running my poetry publish as a script deployment when I tag my repo.

So when master is at a spot where I want to deploy a new version of the package. I do something like.

poetry version minor
git commit -am 'bumped the version'
git tag <version>
# SIDE NOTE: it would be nice to be able to do `git tag $(poetry version --show)`
# or possibly have the bump command output the NEW_VERSION=poetry version minor --output
git push --tags

In order to configure poetry with the credentials to push to our repository I have set $MYPYPI_USER and $MYPYPI_PASS encrypted environment variables in travis.

That’s what I have. Cheers 🍺

[pawamoy]

Can I continue the sharing 🙂 ?

Here is my typical .gitlab-ci.yml:

# Global --------------------------

variables:
  PIP_CACHE_DIR: "${CI_PROJECT_DIR}/.cache/pip"

cache:
  key: "${CI_JOB_NAME}"
  paths:
    - .cache/pip
    - .venv

stages:
  - quality
  - tests

# Jobs templates ------------------

.install-deps-template: &install-deps
  before_script:
    - pip install poetry
    - poetry --version
    - poetry config settings.virtualenvs.in-project true
    - poetry install -vv

.quality-template: &quality
  <<: *install-deps
  image: python:3.6
  stage: quality

.test-template: &test
  <<: *install-deps
  stage: tests
  coverage: '/TOTAL.*\s(\d+\.\d+\%)/'
  script: make test
  artifacts:
    paths:
      - tests/logs
    when: always
    expire_in: 1 week

# Quality jobs ----------------------

check-bandit:
  <<: *quality
  script: make check-bandit

check-black:
  <<: *quality
  script: make check-black

check-flake8:
  <<: *quality
  script: make check-flake8

check-isort:
  <<: *quality
  script: make check-isort

check-safety:
  <<: *quality
  script: make check-safety

# Tests jobs ------------------------

python3.6:
  <<: *test
  image: python:3.6

python3.7:
  <<: *test
  image: python:3.7

python3.8:
  <<: *test
  image: python:3.8

[br3ndonland]

Poetry tips for GitHub Actions workflows

Use caching to speed up workflows

Use actions/cache with a variation on their pip cache example to cache Poetry dependencies for faster installation.

- name: Set up Poetry cache for Python dependencies
  uses: actions/cache@v2
  if: startsWith(runner.os, 'Linux')
  with:
    path: ~/.cache/pypoetry
    key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
    restore-keys: ${{ runner.os }}-poetry-

Use the custom installer

Installing Poetry via pip can lead to dependency conflicts, so the custom installer is recommended. The command listed in the docs exits in GitHub Actions with 127 (not on $PATH).

There are some additional modifications needed for GitHub Actions:

• Add -y to avoid prompts.
• Add Poetry to $GITHUB_PATH (note that the ::set-env syntax has been deprecated).
• Move poetry install to separate step to ensure Poetry is on $GITHUB_PATH.

- name: Install Poetry
  run: |
    curl -fsS -o get-poetry.py https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
    python get-poetry.py -y
    echo "$HOME/.poetry/bin" >> $GITHUB_PATH
- name: Install dependencies
  run: poetry install --no-interaction

Use environment variables for config

Poetry allows config from the poetry config command, or by environment variables. Environment variables are a more dependable way to configure Poetry in CI.

env:
  POETRY_VIRTUALENVS_CREATE: false

Build and publish in one step

• Create a PyPI token.
• Add it to the GitHub Secrets store for the repo (Settings -> Secrets).
• Use the secret in your workflow with ${{ secrets.PYPI_TOKEN }} (secret name is PYPI_TOKEN in this example, and username for PyPI tokens is __token__).
• Use poetry publish --build to build and publish in one step.

- name: Build Python package and publish to PyPI
  if: startsWith(github.ref, 'refs/tags/')
  run: poetry publish --build -u __token__ -p ${{ secrets.PYPI_TOKEN }}

That’s why they call it Poetry. Beautiful.

Example workflow

Expand this details element for an example workflow from br3ndonland/inboard that uses these tips.

name: builds

on:
  push:
    branches: [develop, master]
    tags:
      - "[0-9v]+.[0-9]+.[0-9a-z]+"
  workflow_dispatch:

jobs:
  python:
    runs-on: ubuntu-latest
    env:
      POETRY_VIRTUALENVS_CREATE: false
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - name: Set up Poetry cache for Python dependencies
        uses: actions/cache@v2
        if: startsWith(runner.os, 'Linux')
        with:
          path: ~/.cache/pypoetry
          key: ${{ runner.os }}-poetry-${{ hashFiles('**/poetry.lock') }}
          restore-keys: ${{ runner.os }}-poetry-
      - name: Set up pre-commit cache
        uses: actions/cache@v2
        if: startsWith(runner.os, 'Linux')
        with:
          path: ~/.cache/pre-commit
          key: ${{ runner.os }}-pre-commit-${{ hashFiles('.pre-commit-config.yaml') }}
          restore-keys: ${{ runner.os }}-pre-commit-
      - name: Install Poetry
        run: |
          curl -fsS -o get-poetry.py https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py
          python get-poetry.py -y
          echo "$HOME/.poetry/bin" >> $GITHUB_PATH
      - name: Install dependencies
        run: poetry install --no-interaction -E fastapi
      - name: Run pre-commit hooks
        run: pre-commit run --all-files
      - name: Run unit tests
        run: pytest
      - name: Build Python package and publish to PyPI
        if: startsWith(github.ref, 'refs/tags/')
        run: poetry publish --build -u __token__ -p ${{ secrets.PYPI_TOKEN }}
  docker:
    runs-on: ubuntu-latest
    needs: [python]
    steps:
      - uses: actions/checkout@v2
      - name: Log in to Docker registry
        run: docker login ghcr.io -u ${{ github.actor }} -p ${{ secrets.PAT_GHCR }}
      - name: Build Docker images
        run: |
          docker build . --rm --target base -t ghcr.io/br3ndonland/inboard:base --cache-from python:3.8
          docker build . --rm --target starlette -t ghcr.io/br3ndonland/inboard:starlette
          docker build . --rm --target fastapi -t ghcr.io/br3ndonland/inboard:fastapi
      - name: Push Docker images to registry
        run: |
          docker push ghcr.io/br3ndonland/inboard:base
          docker push ghcr.io/br3ndonland/inboard:starlette
          docker push ghcr.io/br3ndonland/inboard:fastapi
      - name: Add Git tag to Docker images
        if: startsWith(github.ref, 'refs/tags/')
        run: |
          GIT_TAG=$(echo ${{ github.ref }} | cut -d / -f 3)
          docker tag ghcr.io/br3ndonland/inboard:base ghcr.io/br3ndonland/inboard:base-"$GIT_TAG"
          docker tag ghcr.io/br3ndonland/inboard:starlette ghcr.io/br3ndonland/inboard:starlette-"$GIT_TAG"
          docker tag ghcr.io/br3ndonland/inboard:fastapi ghcr.io/br3ndonland/inboard:fastapi-"$GIT_TAG"
          docker push ghcr.io/br3ndonland/inboard:base-"$GIT_TAG"
          docker push ghcr.io/br3ndonland/inboard:starlette-"$GIT_TAG"
          docker push ghcr.io/br3ndonland/inboard:fastapi-"$GIT_TAG"
      - name: Tag and push latest image
        run: |
          docker tag ghcr.io/br3ndonland/inboard:fastapi ghcr.io/br3ndonland/inboard:latest
          docker push ghcr.io/br3ndonland/inboard:latest

Bonus: automated dependency updates with Dependabot

Dependabot now offers automated version updates, with (preliminary) support for Poetry 🎉. If you have access to the Dependabot beta, set up .github/dependabot.yml as described in the docs:

version: 2
updates:
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "weekly"

Dependabot will now send you PRs when dependency updates are available. Although package-ecosystem must be set to pip, it will pick up the pyproject.toml and poetry.lock. Check the status of the repo at Insights -> Dependency graph -> Dependabot.