upgrading the kernel itself: controller.setKernelBundleID()

What is the Problem Being Solved?

We’ll need an inline way to upgrade the kernel itself.

Currently, the kernel source code is bundled once during initializeSwingSet and stored in the kvStore under the kernelBundle key. Each time the kernel is launched, this bundle is given to importBundle to form the “kernel compartment”.

I decided to keep this bundle around, rather than re-bundling the kernel source on each application restart, to 1: speed up restart (bundling can take a few seconds), and 2: reduce surprises when you update your source tree without resetting your chain or other application. During debugging sessions where we’re replaying recorded chain state under modified kernels, we’ve needed to overcome this stickiness with tools like packages/SwingSet/misc-tools/rekernelize.js, to re-bundle and overwrite the kvStore entry. As a result, I was considering removing this feature, and have the controller re-bundle the kernel source code each time the application launches.

But, after working on #4372 bundlecaps, I realized that this stickiness is actually a feature, which would play nicely into a mechanism to cleanly upgrade the kernel itself. The idea is that kvStore['kernelBundle'] becomes kvStore['kernelBundleID'], and initializeSwingSet is responsible for bundling and installing the initial version. Later, when the application is told to upgrade the kernel, it needs to:

• controller.installBundle(newKernelBundle) and get back newKernelBundleID
• controller.shutdown()
• controller.setKernelBundleID(newKernelBundleID)
• (maybe build a new controller)
• controller.start()

setKernelBundleID just checks that the bundleID is valid, and writes it into the kvStore. controller.start() reads the bundleID out of kvStore, loads the bundle itself, then does importBundle() as before.

Of course, it is critical that the new kernel can handle the persistent state in which it wakes up. It must look for kvStore flags that indicate whether particular features have been initialized or not. But the kernel is not obligated to mimic the behavior of some earlier version. The host application is responsible for triggering the upgrade at a consensus-managed moment, between blocks, so the new kernel version only has to be consistent with itself.

A separate issue is how e.g. cosmic-swingset should decide when an upgrade is appropriate. One option is to require an application upgrade, and have the new version pay attention to the block height. When the height reaches a pre-decided point, cosmic-swingset can shut down the kernel, call bundleSource() on the usual path packages/SwingSet/src/kernel/kernel.js, install the resulting bundle, then instruct the controller to use the new bundleID. This approach requires all validators to install the new application before the appointed cutover time, which is also what they would do to replace the Go code in cosmic-swingset, or other low-level non-JS code.

An alternate approach would be to use an in-band transaction to trigger the upgrade. Some external client could use signed txns to perform the controller.installBundle() ahead of time, just as they would install contract code. Then maybe a governance vote triggers the execution of some SwingSet-module code that performs the shutdown/setKernelBundleID/start. This would be driven by governance vote, and would not require validators to install any new software. The governing committee should be equivalent to getting all validators to replace their software, however, because the new kernel code gets nearly complete control over the chain. But the execution of the vote might be easier if it can be handled entirely within the governance module.

Description of the Design

Security Considerations

Replacing the kernel code is the most security-critical thing we can imagine, so both the implementation and the code that triggers it must be audited carefully.

Test Plan

unit tests

[michaelfig]

We’d like to confirm with @michaelfig that this plan will work

IMHO, it looks like it would work just fine.

and we’d like to understand how cosmic-swingset currently implement the “can you handle?” check.

The cosmos-sdk “can you handle check” defaults to “no”, and requires additional application wiring to change that to “yes” for a given upgrade named “XXX”. The chief strength of the Cosmos upgrade system is that it is lazy, and the responsibility of our future selves to design as needed.

But as long as the check currently says “no”, we think we don’t need any upgrade-helping code to go into version-1.

That’s right. The governance proposal would vote for a software upgrade at block 1000 to version-2, with human- and possibly machine-readable instructions for how to install the SDK that understands version-2. When block 1000 rolls around, the version-1 chain halts (“I don’t know version-2”). It doesn’t matter how many times you restart version-1, it just keeps halting.

But, if you start version-2 with version-1’s chain home directory, that would trigger the golang/cosmos/app code that says “Aha, version-2, I got this”, and does any cosmos-level data migrations needed. Then that migration would dispatch a message to cosmic-swingset’s version-2 handler (something like { type: 'UPGRADE_SWINGSET', upgradeName: 'version-2' }), which runs the SwingSet version-2 data migration (reinitializeSwingset as you specified). After that, the chain continues startup, continuing past the version-2 upgrade, and begins block 1000.

[warner]

After today’s kernel meeting, @kriskowal and I figured that we might not need to make any code changes for MN-1, and we’ve sketched out some small code changes needed for the subsequent version

What we need to add in time for version-2 is something like import { reinitializeSwingset } from '@agoric/swingset-vat'. This function will re-bundle the kernel code (as well as lockdown and the xsnap supervisor) and update the DB with the new bundles. That’s all.

Now the timeline of upgrade will be:

• validators create their DB with version-1 code that runs the original initializeSwingset (and doesn’t have reinitializeSwingset)
• validators then launch their nodes with code that runs buildSwingsetController each time they reboot
• at some point, version-1 will observe a governance action that schedules upgrade-to-2 at e.g. block height 1000
• at some point, validators build version-2 and prepare their supervisors to use it after the current version-1 exits with an error
• the last thing version-1 sees is the transition to block 1000, something inside cosmos or cosmic-swingset calls a function that asks “can you handle upgrade-to-2?”, the answer is “no”, and the process terminates
• the supervisor sees version-1 exits, and launches version-2
• the first thing version-2 sees is the same upgrade-to-2 query
  • in response, cosmic-swingset calls reinitializeSwingset before calling buildSwingsetController
  • reinitializeSwingset gets kernel source code from the new version-2 installation, and replaces the bundles in the DB
  • buildSwingsetController loads the bundles from the DB as usual, so it gets version-2
• validators are now using the version-2 kernel code (and lockdown and supervisor)

We’d like to confirm with @michaelfig that this plan will work, and we’d like to understand how cosmic-swingset currently implement the “can you handle?” check. But as long as the check currently says “no”, we think we don’t need any upgrade-helping code to go into version-1.

If so, we can defer this ticket indefintely, and/or close it entirely. If non-chain environments would use a similar “replace the whole process” approach for upgrade, then they wouldn’t benefit from an in-place “live” kernel upgrade either.