The way Keystone does semver is not meaningful to package consumers

Bug report

We had a heated debate about this yesterday on Slack. Changes to the way we version packages are going to effect most contributors, so I wanted to ensure we could have this conversation openly and involve everyone that wants to be involved, hence the GitHub issue instead of just leaving it at the Slack conversation.

Describe the bug

Currently this is happening to users:

1. I have a working Keystone site.
2. I bump a package with a patch update, e.g. 5.0.1 -> 5.0.2
3. Keystone no longer starts or breaks in strange ways.

Expected behaviour

From semver.org:

Given a version number MAJOR.MINOR.PATCH, increment the:

1. MAJOR version when you make incompatible API changes,
2. MINOR version when you add functionality in a backwards compatible manner, and
3. PATCH version when you make backwards compatible bug fixes.

This patch change is not backwards compatible, so should be a major bump.

What?

But Kevin, we do extensive amounts of work to comply with semver!

Explanation

I’m going to create an oversimplified example of why this happens as I think that would be easier to follow, and I’m going to go painfully slow so every step is clear. I have not researched exactly what combination of packages caused this for Chris on Slack yesterday, but please be assured this is biting people regularly, who are reaching out on Slack and in other GitHub issues. This has also happened on the Demo project, and install, run, broken out of the box is not the best introduction to any development tool.

So, for my made up example, let’s simplify and say we only have two packages involved in Keystone, @keystonejs/keystone, and @keystonejs/adapter-knex. We publish as follows:

@keystonejs/keystone@5.0.0

// index.js
class Keystone {
    add(a, b) { return a + b }
}

module.exports = { Keystone };

Dependencies: None

@keystonejs/adapter-knex@5.0.0

// index.js
class AdapterKnex {
    constructor(keystone) {
        this.keystone = keystone;
    }

    go() {
        console.log(this.keystone.add(1, 2));
    }
}

module.exports = { AdapterKnex };

Dependencies: @keystonejs/keystone: ^5.0.0

So I follow the (again, simplified, not real) docs and do the following in my project:

My Project

// index.js
const { Keystone } = require('@keystonejs/keystone');
const { AdapterKnex } = require('@keystonejs/adapter-knex');

const keystone = new Keystone();
const adapter = new AdapterKnex(keystone);
adapter.go();

Dependencies: @keystonejs/keystone: ^5.0.0, @keystonejs/adapter-knex: ^5.0.0

Success

Everything works at this point.

Output

3

Next Version

We decide that we hate the way keystone.add works, and refactor it to destructure the arguments as follows:

@keystonejs/keystone@6.0.0

// index.js
class Keystone {
    add({a, b}) { return a + b }
}

module.exports = { Keystone };

Dependencies: None

Now we’ve made a breaking change, so we dutifully bumped the keystone package to the next major version. However, adapter-knex’s API hasn’t changed, so we just bump its dependency and call keystone.add the new way, publishing as a patch:

@keystonejs/adapter-knex@5.0.1

// index.js
class AdapterKnex {
    constructor(keystone) {
        this.keystone = keystone;
    }

    go() {
        console.log(this.keystone.add({a: 1, b: 2}));
    }
}

module.exports = { AdapterKnex };

Dependencies: @keystonejs/keystone: ^6.0.0

What now?

I’ve been hard at work on my Keystone site, and it’s almost ready to release. Since I’m all about consuming the latest versions, I use Greenkeeper, Dependabot, Snyk or even just npm-check-updates to see if there are any patches I should pull in before I go live. Using npm-check-updates I see this:

@keystonejs/keystone           ^5.0.0  →   ^6.0.0  // This is red
@keystonejs/adapter-knex       ^5.0.0  →   ^5.0.1  // This is green

Since the project has repeatedly assured me its version numbers comply with semver, I know I can pull in just the patch version and not the major. I’m close to live so I don’t want to break anything. I update just adapter-knex.

Or even easier

I don’t change package.json in My Project at all, and I just npm i or yarn without a lockfile. Because I have ^5.0.0 for adapter-knex, npm dutifully installs the patch update and leaves v6 of keystone out of it.

In either of these cases

Now we have the following situation when running npm ls:

├── @keystonejs/keystone@5.0.0
├─┬ @keystonejs/adapter-knex@5.0.1
│ ├── @keystonejs/keystone@6.0.0

Which, if adapter-knex did its own requires would be fine, but remember My Project is the one that requires both and hooks them up together.

When I start Keystone, I get:

NaN

The keystone@6.0.0 dependency is not actually used, as adapter-knex does not require('@keystonejs/keystone') itself. I do the require in My Project. When I do that, I get keystone@5.0.0, and adapter-knex@5.0.1, then string them together, as per the docs.

I now have an incompatible combo of packages, all from bumping a single package with a patch version. Even worse is that unless I have integration tests, I may not even notice this is broken if it’s deep within Keystone and results in data loss on the way to the DB instead of preventing startup. I may deploy this with CI assuming we’re all good and only notice that the patch broke Keystone in production.

Required resolution

Either:

• I should be able to bump any single package in isolation with a patch or minor and not break my site,

or

• We could consider using peer dependencies to declare these, as that’s actually what they are, and tooling recognises that. The main issue with peer deps though is npm i without a lock file would dutifully bring in the patch and not the major. Then there’d be a console warning that there’s an unsatisfied peer dependency of @keystonejs/keystone@6.0.0. This is far from perfect, but at least it’s something. The current setup just silently breaks because I pulled in a patch.

or

• We could consider doing version checking at runtime that’d make this more apparent. The problem with only doing this and nothing else is it’s still the same experience of install patch, then project breaks when you start it. But, in the positives, at least the break is broadcasted for you and isn’t subtle and potentially data-destroying.

or

• We need to stop saying we do semver. Because while it may be true that we technically complied with semver from the perspective of each individual package, we’re doing all of this extra work so users can confidently pull in updates. If pulling in a patch update can break a user’s site, then why are we doing all of this effort to maintain it?

Desired resolution

It’d be nice if it was more clear what was compatible with what, e.g. https://github.com/keystonejs/keystone/issues/2606. I still feel we can use the version number to do that.

Existing opinions

Here’s a summation of each person’s viewpoint from the Slack chat yesterday. If you feel this is not a fair representation of what was said, please let me know, I’m doing this to save you having to reiterate, not to put words into your mouth:

@Noviny: This is a bug, users should be able to pull in patch and minor changes being confident that their Keystone won’t break. Being able to consume a minor or patch update without having to upgrade anything else should be the outcome of any solution, definitely.

@jesstelford: This is working as expected. What you’re experiencing is a combination of a configuration issue and the fact that NPM will install two versions side by side. The fact that one installation ended up with multiple versions is, unfortunately, entirely out of our hands. We are following semver. If you feel you can’t trust it, then you can’t trust it anywhere in the npm/yarn ecosystem, not just in Keystone.

If you want to be sure you can pull in that patch version, either read through the dependencies in package.json of that package and make sure none have changed majors, and if they have, read through all of those package.json files too, then make sure everything is compatible yourself, or install it, then npm ls and look for duplicates. The semver being a patch does not imply that it’s safe to pull in the update by itself, it means that that particular package’s API has not changed, and it has not.

@MadeByMike: I see the points on both sides, this is a nuanced discussion, let’s talk about it more.

Let’s keep the conversation going. I’m particularly interested in what @molomby, @timleslie, @Vultraz, @gautamsi and @JedWatson think about this.

[JedWatson]

Hey @thekevinbrown thanks for the comprehensive write-up! I flagged the conversation in Slack yesterday to come back to and haven’t read through all the details yet but I think I’ve got the jist, and this issue is really helpful 👍

Firstly, an acknowledgement that I think this is a real problem for Keystone’s users; and unfortunately I think this is the sort of edge-case issue where the current state of dependency management in the node ecosystem fails us. In theory, peerDependencies are designed to somewhat address this, but (afaik) we don’t really have a good constraints system for related optional dependencies. So being right technically is not necessarily good for the community, and it’s up to us here to work out how to strike that balance.

Note: in my experience peer dependencies are… basically broken for our use case. If anyone thinks they would help, we can have that conversation.

Also a side note, we can maybe mitigate this a bit by reviewing how the packages in Keystone are actually split up - bearing in mind that a common complaint about versions 1-4 was the “single package” approach and the sheer number of dependencies that were installed but given use case {x} didn’t need to be. Add in multiple native modules for database adapters, etc. and the problem gets worse so I expect while we can maybe minimise the number of core keystone packages and therefore reduce the chance for incompatibilities in upgrades to crop up, we can’t not deal with it.

Onto how I think we can improve things:

Starting with the example of:

@keystonejs/keystone@5.0.0
@keystonejs/adapter-knex@5.0.0

I do not think either package should have a dependency on the other one (this is currently not technically correct, see my note at the bottom). keystone should be provided an instance of the class adapter-knex provides, but this means keystone has a dependency on a version of adapter-knex which is unmanaged.

Looks like this:

const { Keystone } = require('@keystonejs/keystone');
const { KnexAdapter } = require('@keystonejs/adapter-knex');

const keystone = new Keystone({
  adapter: new KnexAdapter(),
});

Now, adapter-knex effectively has two APIs:

• one, “user-facing” is the API that users are intended to use (e.g the schemaName option)
• two, “user-facing” is the API that keystone internally uses when provided an adapter instance (e.g the connect method, ref the adapter framework)

Really, they’re all part of the same “Public API” of the adapter-knex module, but I think it’s helpful to think about this distinction.

A side note about how I think about semver

Semver is, more than anything, the way that package authors have to communicate to users how to interpret new versions. Use major to say “the API or some important internal implementation has changed; you probably need to update, or at least carefully verify, your usage of this package when you upgrade”. Use minor to say “we added a feature but it’s safe to upgrade”. Use patch to say “we fixed something, you should upgrade and expect no changes”

So versions are a bit of an art, because what if “correct behaviour” in a project was depending on a bug in a dependency that gets fixed in a patch release? Often, it’s up to package authors to make calls like that, and guess at the probable impact of changes. Which encourages more major versions.

But more major (and worse, really frequent major) versions are a liability for users because of the amount of work required to upgrade safely. Version churn can really hurt a project’s community.

Ultimately we have to strike a balance and imo it’s better to be cautious with changes, but not at the cost of progress. There’s no perfect solution, which comes back to why I think of it as a communication mechanism.

Back to the point

Let’s say we (breaking) change the “user-facing” API of adapter-knex, but not the “keystone-facing” API. Easy; major release to adapter-knex and we’re done.

Outcome:

@keystonejs/keystone@5.0.0
@keystonejs/adapter-knex@6.0.0

Now, what if we (breaking) change the “keystone-facing” API and not the “user-facing” API of adapter-knex? well, we’re still changing some public API so major release to adapter-knex but… we’re not done.

We also need (in theory) a patch update to keystone to correct its usage of the new adapter API. Why patch? well, we’re fixing an internal; not changing the public API at all. Users can upgrade to the new version without changing their usage.

… except they can’t. They can only safely upgrade if they also upgrade the new major version of the adapter-knex package.

So what do we want to communicate here? I think it’s this:

@keystonejs/keystone@6.0.0
@keystonejs/adapter-knex@7.0.0

But how do users know that these two package upgrades should be linked?

One option is to “lock” major version numbers across multiple packages. But I’m disinclined to do that because I think it’ll get out of control and is very likely to introduce unnecessary upgrade churn for users. (unless we moved to monthly releases for major changes, but I don’t think we’re there yet in terms of maturity as a project. Maybe in the future)

To explain how this can go badly:

We do a major mongoose upgrade, without otherwise changing any of the public API of adapter-mongoose. This should absolutely still be a major version. If versions are locked, that also means a new major of keystone. Which means a new major of adapter-knex. If you’re depending on our original packages:

@keystonejs/keystone@5.0.0
@keystonejs/adapter-knex@5.0.0

Pushing a major out to both of them because we upgraded mongoose (which this project is not even using) communicates a lie, and escalates upgrade fatigue by wasting people’s time.

So I believe that lock-stepping versions is bad.

What else we can do

Because I don’t believe node / npm / The Platform has a solution for this, I think we should solve it ourselves. I thought about adding this really early on, and now kind of regret not doing it.

Let’s imagine that packages, alongside their package version, declare an API Compatibility version.

So in the two packages:

@keystonejs/keystone
  package@5.0.0
  adapter-compat@1.0.0
@keystonejs/adapter-knex
  package@5.0.0
  adapter-compat@1.0.0
@keystonejs/adapter-mongoose
  package@5.0.0
  adapter-compat@1.0.0

If we change the “user-facing” API of adapter-knex we simply increment the package version:

@keystonejs/adapter-knex
  package@6.0.0
  adapter-compat@1.0.0

But if we change the “keystone-facing” API (and corresponding usage) we’d increment the api compatibility version as well:

@keystonejs/keystone
  package@6.0.0
  adapter-compat@2.0.0
@keystonejs/adapter-knex
  package@7.0.0
  adapter-compat@2.0.0
@keystonejs/adapter-mongoose
  package@6.0.0
  adapter-compat@2.0.0

(note that since we’re changing the adapter API, we need a new major of the mongoose adapter as well)

The API compatibility range would be validated by keystone when you start it, and an incompatible package would throw with a link to the docs and changelog.

We’d have a handful of API compatibility keys, others I can think of include admin, session, app, field, etc.

This is a bit more work for Keystone, but I like a few things about it:

• means we explicitly throw when incompatible versions of
• we have an abstraction between major package versions and which of those versions are otherwise compatible with each other
• we have a clear way of recognising when we change how keystone interacts with any of its dependencies
• changing a compatibility version immediately means a new major package version
• it should be one way, so it remains manageable; keystone says “I work with this version” and other packages say “I am this version”
• the problem (and this solution) is applicable to multiple parts of keystone where packages have implicit dependencies on the API of other packages, e.g, the File field and the @keystonejs/file-adapters package

I haven’t thought through exactly how we’d code the validation yet but it’s probably a reasonably straight-forward thing to implement cleanly and I think will prevent future pain related to this issue.

---

Regarding technical correctness of the “these packages should not be dependent on each other at all” statement: today, adapter-knex is dependent on keystone because we need the base adapter classes:

// @keystonejs/adapter-knex/lib/adapter-knex.js
const { BaseKeystoneAdapter, BaseListAdapter, BaseFieldAdapter } = require('@keystonejs/keystone');

I think this needs some more thought, maybe the base adapters belong in another package and not keystone core, to clear up the relationship between packages. But I don’t think any outcome here changes the rest of my points above so I’m going to call that discussion out of scope for now.

---

Keen to hear thoughts on this, and if there’s any enthusiasm to implement an api compatibility interface.

[emmatown]

Given that most things are now in the @keystone-next/keystone package, the other packages have a peer dependency on @keystone-next/keystone making it clear what version of @keystone-next/keystone it needs and we more consistently do majors when things have a chance of breaking for consumers, I think this is now addressed.