Publish packages for each individual API.

This is going to be done in several steps:

• Re-organize the directory structure to map to individual modules (#1167)
• Setup and publish a googleapis-common npm module that just contains shared bits that will be used by all packages
• Generate a package.json for all of the individual APIs
• Script the npm publish process so that each individual package is published along with the larger meta-package

These steps are of course subject to change as we discover things through the process 😃

— UPDATE — We’re down to the last step of actually publishing the individual modules. @alexander-fenster, @jkwlui, and @JustinBeckwith last got together to chat about this, and came to a few conclusions:

State of the world today

• We decide when to run the generator in a somewhat ad-hoc fashion. As someone asks for a release, we cut it.
• We run the generator by having a developer clone the repo, run npm run generate, and then submitting a PR with like 1344247584 changes.
• After checking in that PR, someone has to submit a new release with releasetool. This will create a new tag. It is almost always semver major (see the fact that we’re on v38).
• The generator is somewhat monolithic. It works by blasting away the src/apis directory, and re-creating every API from scratch every time. This makes it easy to detect new APIs, and easy to detect removed APIs.

Where we need to be

• We want a bot that runs the generator(s) nightly. We should use synthtool for this.
• The bot should submit many PRs - one for each API removed, deleted, or modified.
• The individual commits could use the conventional commit scope as a way to signal which API/package this change applies to
• Some combination of releasetool and semantic-release could be used to cut individual releases. A scoped tag would be used for each package release from the mono-repo.
• The risk of this approach is that the number of tags could get out of control, fast.
• Alternatively, we could approach this similar to the way the gRPC generated libraries work. We create a new GitHub repository for each API, and break the output out of this repo and into the individual package repos.

There’s still a lot to figure out, but I suspect @bcoe is gonna love this problem.

[bcoe]

👋 an update on this long standing issue. There’s work that’s been happening this quarter that will facilitate publishing submodules, using the same release automation we use elsewhere – this project has been bigger than expected, due to some of the constraints:

• releasing each API individual would be too noisy, in terms of the bulk of PRs (so we wish to combine the release in one large PR).
• we don’t want to store secrets in GitHub actions, so we’ve been developing an approach for publishing multiple submodules from our kokoro CI/CD system.

In terms of time lines, I’m hopeful we’ll be able to start testing individual package publishes within ~2 weeks.

[nermaljcat]

ok @JustinBeckwith - I’ve deleted my comments. I reject censorship and prefer not to participate in a censored forum.