Host Airflow-managed Helm repo

Description

@mik-laj Hi 👋 I was just coming to this repo to see if you’re interested in help getting the Helm chart repo set up to replace the chart in stable.

Stable repo is nearing the end of it’s deprecation period, and I’m glad to see a version of the Airflow chart is already here. See https://github.com/helm/charts/issues/21103 for the meta issue tracking moving all the stable repo charts to new homes.

To-do

• Decide on hosting options below (self-host/leverage Bitnami)
• If self-host, set up CI/CD for chart-testing and chart-releasing
• if self-host, list in Helm Hub (http://hub.helm.sh/) and Artifact Hub (https://artifacthub.io/)

Use case / motivation

Set up Helm repo hosting for your chart

Set up a Helm repo, either as a separate git repo in your org, or keeping the same setup you have now. We have created Helm chart repo actions for chart testing (CI) and releasing chart packages as your own GitHub-hosted Helm repo (CD).

Self-hosted options:

1. If we either move the chart to a separate git repo in the artifacthub gh org, or even move the hub github pages setting to a branch other than the main one, we can use the @helm/chart-releaser-action GitHub Action to automate the helm repo.
2. If we keep structure as-is, we can still use the helm/chart-releaser project, just with a custom script.

For either option we can also use the @helm/chart-testing-action to wrap the chart-testing project @mattfarina mentioned above. Here’s an demo repo to see how they work together: https://github.com/helm/charts-repo-actions-demo

Whichever option you decide I’m make a PR if it helps.

If you do decide to host your own Helm repo, you will also want to list it in

Alternatively leverage existing Bitnami Helm repo

There is also a version of the chart maintained by Bitnami, who have been very involved in the stable repo for years, but : https://github.com/bitnami/charts/tree/master/bitnami/airflow. You could instead decide to leverage that chart as the canonical source, and not host your own. It is also fine to have multiple instances of a chart to install the same app.

Related Issues

• https://github.com/helm/charts/issues/21103
• https://github.com/apache/airflow/issues/10486
• https://github.com/apache/airflow/issues/10379

[potiuk]

I do not have problems anymore with the licenses or sources 😃. This has been solved.

However, I am afraid it’s not as easy as ‘just create repo’ and as PMCs, we have formal obligations that we have to fulfill to release the Helm Chart. We discussed it several times in the past with the original authors of the Helm chart

There are a couple of things here:

Taking over the chart

I think we already decided we are not “taking over” the stable/helm chart. Those charts are being deprecated (Nov 13 is officially when the project ends). The project was developed outside of the community and “taking over” means also taking responsibility towards the users of the “helm/stable” chart, taking care of migration from the old chart, and being generally responsible for it - including troubleshooting and support. While this is possible and maybe even advisable - this is something that we as a community have to approve, vote on making sure that we “receive” the software formally - IMHO basically the software (including its git history) will have to be donated to Apache Airflow, by whoever is the current owner. Software is not only an asset.  It might also be a liability and we have to make sure that we know what we agree to.

I honestly do not think moving history is at all possible and I think it is a bad idea. Those projects do not share the common history and we even cannot legally do that before the code is officially donated to Apache Software Foundation. What I suggest is to write a migration guide from the old Helm chart to the new one, maybe (if there are missing features in the “future one”) it would be great if the original authors or users open issues in the Airflow project and we could consider adding them (@gsemet - as the original author, I would love to hear what you have to say here).

IMHO the “deprecation” notice should be “You have to migrate to the new chart from the old one but they are different” - because essentially they are. Now the question is - who should write and take care about such a migration guide, I think as a community we should help with that, but I consider that the work of the authors of the original chart to lead that effort. I am super happy to help (going for vacations for a week) and discuss/add missing features in the new chart or help to review and collaborate on the migration guide, but since the community was not involved at all in the old chart development  I think we should be very careful in stating that we are “taking it over”. I think we should care about our community and people who are using the old chart and help them to migrate to the new one (and as a community to take on the task of maintaining it) so I am happy to help with that, but we simply cannot take it over. I am also super happy to refer to the past contributors of the old helm (and give them the credit) and point people to the migration guide (that might be initially part of the helm/stable repo), and once it is well tried by some people we might add it is a “contributed” migration guide from the “stable” chart

We discussed it several times for example here or here.

Of course, we could go the whole process - the original authors (and only them) might propose at the devlist that the community take over also the history and the community might vote on it. But this is something the authors have to propose officially at the devlist, not in a GitHub issue, make a formal vote on it and it has to pass the vote. None of us here individually can make that decision. This is how the Airflow community works.

Releasing our current chart

Since we are going to release the Helm Chart as sources to our users, we cannot “just create a repository” and release it. This is not a “convenience binary” as in the case of Docker image (which is built from the officially released sources) - those are “sources”. We are obliged by  ASF policy ASF Release Policy to release it officially. This means that we have to do it formally - including preparing packages (storing, checksumming, signing on SVN), voting, releasing the software formally in Apache Software Foundation Subversion, etc. This can be connected with releasing the whole Airflow project or separately. I believe we agreed that we are going to release it separately, thus should follow a separate release process. Of course, the repository with master code synced from Apache Airflow can be done before. We can easily use GitHub Actions and tools like https://github.com/google/copybara to automatically move the Chart - related history from  https://github.com/apache/airflow (charts folder) to (for example) https://github.com/apache/airflow-chart. I am happy to help to set it up after my vacation. But we cannot advertise it to anyone else than members of the devlist. This is strictly forbidden by the ASF release policy. Relevant quote here:

Do not include any links on the project website that might encourage non-developers to download and use nightly builds, snapshots, release candidates, or any other similar package. The only people who are supposed to know about such packages are the people following the dev list (or searching its archives) and thus aware of the conditions placed on the package. If you find that the general public are downloading such test packages, then remove them.

So announcing it as “replacement” of the helm/stable chart without formally releasing it is against the rules of ASF. In order to announce it to the users, we have to follow the formal release process. There is no way around it.

As a PMC member, I simply cannot agree on a different approach as this is something we as PMCs are obliged to make sure happens and that we follow the right process. ASF indemnifies us - PMC members - from any legal obligations resulting from the release process - but only as long as we follow the rules of ASF. @dimberman - same with you, I am afraid 😃.  That’s why any approach not following the ASF rules is a very strong -1 (veto) from my side.

@scottrigby -> sorry for being an obstacle here, I am going to be a bit more involved with the helm chart testing in the coming weeks (after my holidays) and I am happy to help in people having a smooth transition to the new chart, but it simply cannot be done in the form that one day we just take over your obligations towards the “stable helm” users. I am happy to help to make it easier, but Apache Airflow is a mature project already with a mature ASF organization and we have to follow the rules.

For the reference - here is the relevant chapter of the ASF policy:

Every ASF release must contain a source package, which must be sufficient for a user to build and test the release provided they have access to the appropriate platform and tools. The source package must be cryptographically signed by the Release Manager with a detached signature; and that package together with its signature must be tested prior to voting +1 for release. Folks who vote +1 for release may offer their own cryptographic signature to be concatenated with the detached signature file (at the Release Manager’s discretion) prior to release.

Note that the PMC is responsible for all artifacts in their distribution directory, which is a subdirectory of downloads.apache.org ; and all artifacts placed in their directory must be signed by a committer, preferably by a PMC member. It is also necessary for the PMC to ensure that the source package is sufficient to build any binary artifacts associated with the release.

Every ASF release must comply with ASF licensing policy. This requirement is of utmost importance and an audit should be performed before any full release is created. In particular, every artifact distributed must contain only appropriately licensed code. More information can be found in the foundation website and in the release licensing FAQ.

[sryabkov]

@kaxil What’s the best way to track progress on the chart work? Maybe contribute too… We spent a lot of time with the airflow stable chart and are looking to switch soon, so we have both experience and motivation to help out…