RFC: Packages extensibility

Problem

The tagling for changesets is A way to manage your versioning and changelogs with a focus on monorepos, currently it is A way to manage your versioning and changelogs for NPM packages with a focus on monorepos

The goal of this RFC is to suggest how we could realise the dream while not only supporting NPM.

There are also a bunch of issues talking about changing behavior around NPM publish.

Related discussions / PRs

https://github.com/changesets/changesets/issues/171 https://github.com/changesets/changesets/issues/218 https://github.com/changesets/changesets/issues/399 https://github.com/changesets/changesets/issues/425 https://github.com/changesets/changesets/issues/580 https://github.com/changesets/changesets/issues/654 https://github.com/changesets/changesets/issues/778 https://github.com/changesets/changesets/issues/800 https://github.com/changesets/changesets/pull/801

Proposed solution

Packages have a few basic components:

1. A name
2. A version
3. Dependencies to other packages in the repo
4. Publish status with version
5. Ability to be published

If changesets allows the concept of ‘packages’ to be extensible then a repository could have multiple types of packages registered. For example in an NX repo you could expose NX projects via the plugin. It would also allow non-node projects to be tracked by creating a plugin to support Ruby Gems, or .net NuGet packages etc.

Package interface

type PublishedState = "never" | "published" | "only-pre";

interface ChangesetPackage {
  name: string
  version: string
  kind: string
  directory: string
  dependencies: string[]

  getPublishInfo(): Promise<{
   publishedState: PublishedState
   publishedVersions: string[]
  }>
  /** assumes updateVersion has run */
  publish(): Promise<{}>
  updateVersion(version: string): Promise<{}>
}

Config

  "packages": [
    "@changesets/packages-nx",
    [
      "@changesets/packages-npm",
      { "repository": "npm", ignore: ['some-package'] }
    ],
    [
      "@changesets/packages-nuget",
      { }
    ]
  ]

Core workflows

Some of these diagrams could be improved, but is a first pass at visualising the workflows. Also have left out things like pre-release to keep things simple.

Get Packages

sequenceDiagram
    cli ->>+ getPackages: 
    getPackages ->> config: getPackagesPlugins
    config ->> getPackages: PackagePlugin[]
    loop map each plugin
        getPackages ->> plugin: getPackages
        plugin ->> getPackages: ChangesetPackage[]
    end

    note over getPackages: Input ChangesetPackage[][]
    loop flatMap packages
        alt !exists in map
            getPackages ->> getPackages: add to map
        end
    end

    getPackages ->>- cli: ChangesetPackage[]

Version

sequenceDiagram
    note over getPackages: *Changed
    version ->> getPackages: 
    getPackages ->> version: ChangesetPackage[]
    version ->> readChangeset: 
    readChangeset ->> version: NewChangeset[]

    version ->> assembleReleasePlan: changesets, packages
    note over assembleReleasePlan: Filters changesets from ignored packages
    note over assembleReleasePlan: Flatten changesets into package releases
    note over assembleReleasePlan: Add changes due to dependencies into releases
    assembleReleasePlan ->> version: { changesets: NewChangeset[], releases: ComprehensiveRelease[] }

    version ->> applyReleasePlan: 
    
    loop for each release
        note over package: *Changed from directly updating package.json
        applyReleasePlan ->> package: updateVersion()
        applyReleasePlan ->> config: Get changelog plugin
        note over applyReleasePlan: Gets commit for each change
        note over applyReleasePlan: Uses plugin to format lines

        applyReleasePlan ->> changeset file: Updates CHANGELOG.md for project
    end

Publish

This function gets rejigged a fair bit. Packages marked as private never call the publish() function on the package, they are just tagged (assuming the config allows that).

sequenceDiagram
    publish ->> getPackages: 
    getPackages ->> version: ChangesetPackage[]
    
    loop parallel map for each package
        alt is private package
            publish ->> git: get tag for current version
            alt tag exists
                publish -x publish: do nothing
                note over publish: return { published: false }
            else
                publish ->> git: tag release
                note over publish: return { published: true }
            end
        else
            publish ->> package: publish()
            alt success
                publish ->> git: tag release
                note over publish: return { published: true }
            end
        end
    end

Challenges

Source of truth?

While building https://github.com/changesets/changesets/pull/662 one of the challenges of tracking private packages is that Changeset uses the published NPM info as the source of truth but this isn’t possible for the private packages. Those instead use the git tags in the repo.

I think we should use tags in the git repo as our state, then we can put checks in place to ‘refresh’ if things get out of sync, ie publish succeeds but then tagging fails. We can detect and fix this state.

This approach will not work if you can publish without tagging though, or not tag per repo. Either that or only private packages rely on git tags being the source of truth?

Duplicate discovery

Package discovery will be done by multiple plugins, if multiple plugins discover a project I think we simply take the one from the plugin registered first (so ordering of the plugins matters).

[Andarist]

@tom-sherman note that changesets/action is already using Changeset for its publishing process and needs. There are some additional scripts there to plumb things together but they are pretty minimal. I’m not sure how much sense it makes to develop github actions in a monorepo setup - but ofc Changesets also work in normal repositories too (and changesets/action is an example of that). But either way… I imagine that this RFC could make it a little bit easier to manage github actions too.

---

As to the RFC itself - I think I’m on board with some form of this. There is a clear demand for a feature like this. I think that we should keep package.json as the base “manifest” for the package. It’s npm convention, but a harmless one for other ecosystems and it makes sense for polyrepos that include javascript packages to have a single type of a manifest for all packages. This would make the package discovery work out of the box at the expense of adding private: true to non-npm packages. Thoughts on that?

I’m open to discussing the exact implementation more. We could also start with a PoC PR and bikeshed things there. Note that nothing is set in stone when it comes to this feature - we need to try to get it right so I wouldn’t be surprised if this will require a couple of iterations before landing.

[brianespinosa]

I am not entirely sure where we might have landed on this RFC as I am coming back to read through the conversation to this point, but I will try to summarize:

It seems like perhaps @mitchellhamilton has the greatest concerns that we might need to address, but past that, it seems like this proposal would be sound and we need to figure out where to do the work. Is this accurate?

I am coming back here to look at this RFC and #848 because my teams will be in need some some of this support, and we might be able to underwrite some of the efforts to get the work done if there is consensus.

cc @Andarist @JakeGinnivan