[RFC] Revisiting our CI setup: GH Actions, RTD

In #687 , we started to discuss our more general setup in terms of Azure Pipelines vs GitHub Actions (GH Actions). A couple of us support the switch to GH Actions, and there’s no strong objection to it. I’d like to try that out.

The setup I want to try would be something like a reusable workflow to run tox, in the repo, and then a job named build which runs the tox workflow over a matrix of python version, os, and tox env names. Now that reusable workflows supports using a local file, we can develop something in webargs until we like the result. If we then want to move it to a separate project like marshmallow-code/reusable-github-workflows, we can discuss and/or do that.

There ~are two notable jobs~ is one job in Azure Pipelines which I don’t think belongs in the move to GH Actions:

• tox -e docs
• ~pypi publish~ (done in #690)

For the docs, ReadTheDocs (RTD) offers a relatively new feature to build on PRs; I recall we discussed it when it was still in beta. I like this primarily because it’s driven by the .readthedocs.yml in the repo, so we don’t run into trouble where our tox -e docs run in CI differs from what happens in RTD. It also means that if we have a PR to update .readthedocs.yml config, it will be tested in the PR build. All we have to do is enable this feature in the readthedocs.org admin pane and make sure that the RTD webhook is set to send Pull Request events. A trial PR could demonstrate that this works – e.g. by updating the python version used for RTD.

~For pypi publishing, I think we can leave this in Azure for the initial move to GitHub Actions. It should be possible to setup a tag->publish reusable workflow, but I want to get more “basic” CI worked out before taking a crack at this.~ (done in #690)

---

With #690 now merged, we have a short “remaining” TODO list:

• Turn on ReadTheDocs PR builds (Settings > Advanced > PR builds)
• Remove testpypi releasing once we’ve tested that it works
• evaluate reusable workflows so that we can reduce duplication in config across repos

[lafrech]

Also related, we’re facing CI limitations in apispec so I’m probably going to move to GHA as soon as I get the time.

https://github.com/marshmallow-code/apispec/pull/747

[davidism]

the versions of tox and pre-commit are actually not the version used during the test

We now use pre-commit.ci which uses a specific pre-commit image, but pre-commit has other guarantees about compatibility and pinning hooks.

Now that you mention it, tox not being pinned was an oversight. With pip-compile-multi, it’s absolutely fine to split up into lots of small requirements files, like one for a pre-commit env (again, check out pre-commit.ci), one for tox, etc. Although I guess it turns out it wasn’t a big deal that it was unpinned, it hasn’t caused any issues since I started doing all this.

The nice part about having all dev dependencies pinned is that it’s much easier to ensure new contributors have a known-good environment. This was important at conference sprints, where we’d need to get lots of people set up quickly and not run into weird version differences.

Not perfect for Pallets either

Just to be clear, we’re fine with pip-tools, pip-compile-multi is a wrapper around it that automates updating lots of requirements files. What I was unhappy with was Dependabot, which was noisy and also only worked specifically with the output of plain pip-compile, not the slightly different output of pip-compile-multi.