Actions and input needed towards CI/CD for upcoming release

Background

Towards making a release (#381), it would be great to have automation for various things setup.

While working towards automation to build/push docker images to ghcr.io, I realize it ties into automation of publishing the Helm chart as well. I’m trying to plan ahead and raise questions for input as I’d like to solve many goals and not just pushing the images.

Making goals explicit

Important goals

1. Release automation To have git ops based publishing of artifacts when we push a git tag.
   • Documentation (#495)
   • PyPI packages (#494)
     • Ensure dask-gateway-server gets wheel populated with compiled Go code for various architectures etc. https://github.com/dask/dask-gateway/issues/496
   • Docker images (amd64 and arm64) (#493)
     • Validate this works properly.
   • Helm chart

Bonus goals

1. Development release automation To have git obs based publishing of some artifacts when we push to main branch:
   • Docker images (#493)
   • Helm chart With this in place, it would make it far easier for me to for example make a manual end-to-end test of the Helm chart before making a release in a production deployment setting.

Action points

There are plenty of action points that I lack permissions to do, but I’ve outlined below what I think should be done and how to do it for anyone with relevant permissions. If you are a Dask GitHub organization admin you can do a lot of the steps below!

• PyPI release preparations We need deployment keys for both PyPI projects available to this repo’s GitHub workflows (dask-gateway, dask-gateway-server). This can only be done by a PyPI project maintainer, currently that must be @jcrist.
  • Visit https://pypi.org/manage/account/token/ and create a token for one project
  • Visit https://github.com/dask/dask-gateway/settings/secrets/actions and create a GitHub actions secret including the package name, like pypi_token__dask_gateway and pypi_token__dask_gateway_server. It would be good to have an additional owner from the Dask team added to those PyPI projects.
  • Add one or more owner/maintainer from the Dask team to both projects, direct links are: dask-gateway on PyPI, dask-gateway-server on PyPI.
• conda-forge release preparations Following a release has been made to PyPI, an automated PR will show up at conda-forge/dask-gateway-feedstock. Currently only @jcrist among the Dask team has permissions to merge those. Adding more maintainers would be good.
  • Add a code suggestion to add yourself as a maintainer by the conda-forge feedstock to this PR: https://github.com/conda-forge/dask-gateway-feedstock/pull/22
• dask/helm-chart release preparations
  • Alternative strategy option: To publish the dask-gateway Helm chart, we rely on the dask/helm-chart repo’s gh-pages branch. So, we will need a GitHub repo deployment key for that setup as a GitHub actions secret in this repo.
    1. Generating a private/public key pair:

       ssh-keygen -t ed25519 -C "dask/dask-gateway" -f /tmp/id_ed25519
       

    2. Registering the private key (/tmp/id_ed25519) as a secret named “gh_deploy_key__dask_helm_chart” for this repo in https://github.com/dask/dask-gateway/settings/secrets/actions
    3. Registering the public key (/tmp/id_ed25519.pub) as a deploy key with push rights for the dask/helm-chart repo at https://github.com/dask/helm-chart/settings/keys
  • Alternative strategy option: To make available a Private Access Token (PAT) from an account with access to push to the dask/helm-chart repo, such as from the dask bot account.
• ghcr.io release preparation
  • BLOCKING UPSTREAM: It appears that the instructions at https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility#inheriting-access-for-a-container-image-from-a-repository are outdated, and I’ve failed to manage to push a public image to ghcr.io. I will investigate this further. UPDATE: This was resolved, the UI had changed, but there was a “package settings” after all but a bit tricky to find.
  • Publish a dummy ghcr.io/dask/dask-gateway and ghcr.io/dask-gateway-server container manually so we can configure settings.
    1. You must be a Dask organization admin
    2. You must create a personal access token (PAT) for your account with write:packages permissions at https://github.com/settings/tokens/new.
    3. Login to the ghcr.io container registry with the PAT:

       docker login ghcr.io -u your-username
       

    4. Push a dummy image with a label to couple it with a github repo and enable us to configure permissions

       echo "FROM alpine" > Dockerfile
       docker build . --tag ghcr.io/dask/dask-gateway --label org.opencontainers.image.source=https://github.com/dask-gateway
       docker build . --tag ghcr.io/dask/dask-gateway-server --label org.opencontainers.image.source=https://github.com/dask-gateway-server
       docker push ghcr.io/dask/dask-gateway
       docker push ghcr.io/dask/dask-gateway-server
       

    5. Declare permissions/access for each container package For each container package: a) grant permissions to github actions to push to it, b) declare that access should be inherited, c) declare that it should be public. All settings should be found in the same page.
       • Visit https://github.com/orgs/dask/packages/container/dask-gateway/settings and do the steps above.
       • Visit https://github.com/orgs/dask/packages/container/dask-gateway-server/settings and do the steps above.

UPDATE: Removed Quay steps as we are no longer going down that road.

Input wanted for decisions

• To git ops automate publishing to PyPI We’ve done it across the JupyterHub ecosystem and I personally love it because it makes cutting regular releases far more sustainable for us.

  • Decision: ok to go for it here?

• To not pin image versions in Helm chart If we pin the versions images we reference in our helm chart, we will need to push git changes about that whenever they should change. I’ve thought a lot about this and concluded that if we managed the helm chart in a separate repository from the images, this would be fine. But, when they are part of the same, we should let them have placeholder values and let skaffold or chartpress update the image tags just before packaging and publishing a Helm chart.

  • Decision: ok to remove hard-coded references in the helm chart and rely on skaffold and/or chartpress?

• To enable jupyterhub/chartpress to build images, while retaining support for using skaffold for local development experiences I recall skaffold brings some local development experiences to the table, while chartpress brings some publishing experiences to the table, and that in the past we have relied on chartpress for publishing the Helm chart - but not the images.

  The Helm chart should always be updated if a new image is to be published, so instead of having images build separate from the Helm chart in an automated CI system, I suggest the strategy to let chartpress do both. I want to be clear that this would be to make chartpress take on so far never automated responsibility, it wouldn’t imply removing any local development use of skaffold etc.

  I have not kept up to date with skaffolds possibilities, but I would prefer to stick with what I know will work at this point rather than re-evaluating again if it would be possible for skaffold to take on the full responsibility of the tasks I want us to get done. I know that chartpress has evolved to meet all the requirements I see it needs, such as: a) building multi-architecture images, b) packaging and publishing to a Helm chart repo in a gh-pages branch, updating the chart’s files Chart.yaml and values.yaml with the latest chart version and image versions.

  • Decision: ok to expand responsibility of chartpress to also build and publish images, along with the Helm chart they couple to?

[consideRatio]

Thank you @jacobtomlinson! I don’t think they need to be public, i didnt consider that aspect - it would be good if they arent public as that may mislead someone to use them for something.

[consideRatio]

Thank you @jacobtomlinson!!

Compiling the wheel-bundled Go code should absolutely be done and I’ll learn more follow up with that, while doing it also ensuring that there is a test before pushing to PyPI that the built wheel includes what we want.