Announcement: Create release 5.0 which is compatible with 3.20.0 but incompatible with 4.0

Background

Kartothek 4.0 is a great release with many extremely valuable simplifications. But during internal migrations from 3.20.0 to 4.0, it turned out that for many of those well thought changes an easy migration could not be taken with reasonable effort. Also some of the simplifications caused customer facing incidents. Thus we are currently stuck with 3.20.0.

To give this important library the deserved attention and speed we are working with a bunch of colleagues on getting out a stable version of kartothek again, where we can simply migrate to and accept community feedback and pull requetst.

To move on, we need to bring a current stable version (3.20.0) to a version which has the simplification of 4.0. We could have continued with development of 3.x branch and slowly converging against 4.0. We have decided against this for the following reasons:

1. some of the changes are on syntax level breaking and would require a new major version itself (between 3.x and 4.0)
2. pull requests and issues from the community are usually reported against the latest release. By that, we would on the one hand create a moving migration target and on the other hand slow down accepting those PR, since the main effort would go into 3.x first
3. this would also include dependency updates which would usually only be accepted for master, opening another dimension where 3.x and 4.0 are diverging.

Planned next steps

To overcome this we decided on the following way forward.

1. We will create a release 5.0 based on 3.20.0. All changes which happened after 3.20.0 will be discarded. There will be no migration path from 4.0.1 to 5.0
2. After having that release in place, we will again accept PRs, issues and dependency updates.
3. From the learning of the 3.20.0 to 4.0 migration, we will gradually add those incompatible changes back (like removal of the multi-table-feature). Where applicable, we will offer a straight migration path to a 6.0 version. Where this is not applicable (like when return types changed from Dict[str,List[T]] to List[T]) we might go a step in between. Communication on that will follow.
4. Finally there will be a release 6.0 which will have most of the simplification.

We know that especially the incompatibility between 4.0 and 5.0 will be an issue for some consumers. We that’s why encourage you to use the very stable kartothek version 3.20.0 and not version 4.x

[steffen-schroeder-by]

We merged @ilia-zaitcev-by’s PR. Next steps are releasing a 5.0.0rc1, extensive testing and then preparing the 5.0. We are currently internally building a plan for 6.0 with migration plans from 4.0 and from 5.0. With the learnings from the 3.20 to 4.0 upgrade we wanna make sure that this is as smooth as possible for all stakeholders. Furthermore, I created https://github.com/JDASoftwareGroup/kartothek/tree/maintainance_3.20 and https://github.com/JDASoftwareGroup/kartothek/tree/maintainance_4.0 so we are able to continue maintenance on those releases of needed. Thanks again to @ilia-zaitcev-by for preparing the PR and to @xhochy for the constructive feedback and attitude of finding a way forward. Closing this issues for now.

[steffen-schroeder-by]

Just want to give a small update on this. I had a very constructive call with @xhochy . I explained the reasoning about the intended actions. He helped me to understand the community and kartothek user’s point of view. I admit that we should have communicated earlier and more transparent the challenges the adoption of kartothek 4.0 gave us. Also the message as I have written it did not express enough a way for 4.0 onwards. It was never intended to pretend that version has never existed. Personally I’m still convinced that its simplification are needed. So let me explain the target state. There should be a version 6.0 which has most (if not all) simplifications version 4.0 introduced. There should be migrations paths for each incompatible change from 4.0 AND 5.0 to 6.0. So the investment which was made for migration from 3.20 to 4.0 is till valid. 4.0 is not supposed to be a dead end.

The next steps are (aligned with @xhochy):

• A BlueYonder associate will work on a migration path from 4.0 to 6.0. Once the results are ready, we will publish that as a separate Github Issue.
• I will create maintainance branches based on 3.20 and 4.0.1 (HEAD) so critical fixes can be applied on those branches. Even dependabot updates can be targeted on those branches
• On Monday 2021-06-07 I will merge PR https://github.com/JDASoftwareGroup/kartothek/pull/472
• We will create a 5.0rc from that and test that internally
• If those tests are successful we will publish a 5.0
• When we begin the work on deciding on a migration path from 5.0 to 6.0 with the learnings which we had with the 3.20 to 4.0 migration
• Edit: there are unreleased changes on master branch which should go into a 4.0.2. Will create a release for this as well.

Let me know your thoughts.