[SIP-12] Proposal for Branch Management and Release Process

Motivation

Historically organizations partaking in the release process have managed their own fork on incubator-superset which is deployed in production via a Git submodule. Both companies have been actively developing Superset and deploy internally (features and bug fixes) at a higher cadence than the public PyPI releases. These forks came about because of the potential instability of the master branch (see SIP-13) and provided a mechanism to cherry-pick specific commits thus ensuring an increased stability.

These organizations have discussed ways of trying to fold these forked branches back into the incubator-superset repo to better ensure consistency and to serve as a mechanism for formalizing the release process which is somewhat sporadic.

Proposed Change

The proposed solution is based primarily on having release branches where certain organizations are the primary beta testers for stabilizing the branch.

Master branch

There will be no changes to how the master branch currently works. We will do our best to keep the master branch healthy in part by the initiatives outlined in SIP-13. If you create a pull request, please do it against the master branch. We will maintain stable branches for releases separately but we don’t accept pull requests to them directly. Instead, we will cherry-pick non-breaking changes from master to the two latest releases.

Release Cadence

We propose a regular release cadence of every two weeks. This period is a tradeoff between stability and flexibility ensuring that new features can be deployed within a reasonable time frame. Furthermore the longer the release cadence the greater the potential conflict when cherry-picking commits. A release will be deployed two weeks after it was cut.

We propose that each release branch is fully managed by a single organization.

Branch Naming

We propose that all Git branches follow the <prefix>/<suffix> naming convention:

• Personal: <username>/<issue>, i.e., jane-doe/issue-123
• Feature: feature/<name>, i.e., feature/dashboard-v2
• Release: release/<version>, i.e., release/1.1

Note if / is troublesome we suggest using a two hyphens (--) instead.

Semantic Versioning

Superset follows semantic versioning (<major>.<minor>.<patch>) although the major version has remained fixed at zero. We should release patch versions for bug fixes, minor versions for new features, and the major version for breaking changes. When we make breaking changes, we should also introduce deprecation warnings in a minor version so that our users learn about the upcoming changes and migrate their code in advance.

Git release branches represent the <major>.<minor> level whereas Git tags (associated with a specific commit) represent the fully defined <major>.<minor>.<patch> version. Note tags may contain an optional release candidate suffix (-rc).

The Git Ecosystem

The following diagram describes the apache/incubator-superset Git ecosystem and how we plan to couple this with several organization’s internal deployment (superset-internal which uses a Git submodule) which will help serve as a mechanism for stabilizing a release.

Day 0

On day 0 the next release branch (release/1.1) will be cut from master which references the latest commit. The commit will be tagged with v1.1.0-rc (the v1.1.0 release candidate) and deployed to PyPI.

Day 0 - 13

Only commits associated which resolve bugs present in the release candidate will be cherry-picked onto the release branch. Fixes to features added after the cut will not be cherry picked (†). Note there may be potential merge conflicts (which will have to be resolved) when cherry-picking a fix due to an upstream commit which is not in the release (‡). If this is too complex the cherry-pick should be skipped (—) as it will be included in the next release. New features can be committed on master if and only if they’re defined in their entirety within the two week cycle. Larger features should be developed on a feature specific branch (which is then merged into master) or using feature flags.

The following diagram shows the use of a feature branch which spans multiple releases.

Day 7

Organizations partaking in the release process will update the Git submodule reference to point to the release (release/1.1). This seven day lag provides a soak period of ensuring that critical bugs are not deployed to our production environments. The update is made to the .gitmodules file,

[submodule "incubator-superset"]
    path = incubator-superset
    url = https://github.com/apache/incubator-superset
    branch = release/1.1

which ensures it tracks the correct release branch. Like the two week release process the submodule branch will be updated every two weeks.

Day 7 - 20

Periodically organizations partaking in the release process will ensure that the submodule in their internal deployment will reference the latest commit from the remote branch (which contains all the cherries) via,

git submodule update --remote --recursive --merge 

which will fetch the latest changes for each submodule, merge them in, and check out the latest revision of the submodule. This step allows us to further test and stabilize the branch in a production environment.

Day 14

The first (non release candidate) version will be tagged (v1.1.0) and deployed to PyPI which references the latest commit on the branch (*). Simultaneously the next release branch (release/1.2) will be cut from master which will mimic v1.1.0 except it will also include any feature commits which were merged in the past 14 days. Note a new release branch is cut if and only if feature commits were added, to prevent duplication.

The following diagram shows an example where on day 14 a new release branch is not cut as there were no new features added since day 0. The next viable release cut date will then be day 28 to preserve the release cadence.

Day 14 - 27

Additional bug fixes to commits in the release candidate will be cherry-picked onto the n most recent releases (where n = 2, i.e., release/1.1 and release/1.2). Throughout this period the tag can be updated (incrementing the patch) as desired based on the severity of the fix.

Day 28 -

Assuming there is a new release branch (release/1.3) we will no longer be picking cherries onto release/1.1 since we only support two releases (release/1.2 and release/1.3). In summary most release branches will have a four week active lifespan.

Branches should exist for infinitum to ensure critical security fixes can be patched to the necessary branch.

Database Migrations

Database migrations (which are handled via Alembic) have an additional lineage component (used to determine upgrades and downgrades) which further complicates matters. We need to ensure the playback of versions is correct when upgrading to a newer release and that a downgrade is never necessary.

The logic for cherry-picking migrations is similar to above whereas migrations can be cherry-picked onto a release branch if and only if all of its ancestors are present on the branch. Given that there no notion of resolving merge conflicts in Alembic, any migration (and thus future migrations) related to a feature will be skipped (--) and included in the next release.

Action Items

Currently we plan on manually cherry-picking the commits from master to the desired release branch. This could be automated in the future based on PR tags and/or the assistance from a bot.

We also should check with Apache whether there are any concerns/feasibility issues with releasing on a two week cadence.

@beto @mistercrunch

[mistercrunch]

Reminder that part of the SIP process should be to notify the Apache Superset dev@ mailing list.

[rusackas]

I’m going to go ahead and close this due to age, but @john-bodley please feel free to re-open and kick start the DISCUSS and VOTE threads if you want to rekindle it.