how to expose the Virtual/Durable Collection API? global? module import?

What is the Problem Being Solved?

How should we make the new Collections API (#2004) available to user-level code?

Imagine a library that uses our new VirtualMap or DurableSet or whatever. Should this library start with import { VirtualMap } from '@agoric/collections' ? Or should it look for globalThis.VirtualMap? Or should it require VirtualMap to be passed in as an argument?

There are two drivers here. The first is whether we’re comfortable having all pieces of this API be ambiently available. @erights has expressed concern that the Durable subset represents sufficiently more authority than the merely Virtual or Ephemeral portions, since Durable data can (in fact must) survive an upgrade, whereas Ephemeral is entirely in RAM and Virtual is just like RAM but bigger. I’m still of the opinion that basically everything should be Durable, or at least that any program which uses Virtual instead of Durable is running the risk of making the wrong decision and being in trouble when it comes to upgrade. So I’m less worried about withholding Durable from user code.

If we decide Durable should not be ambient, then it must be passed as an argument. That would still leave the question of how Virtual should be delivered.

The other driver is that we have two environments where these features might be used. The main one is inside a swingset vat, where both Virtual and Durable collections will be wired to DB access managed by liveslots and the virtual object manager. Code that runs here is loaded into a Compartment, which gives us some measure of control over how it imports things.

But an equally important one is in unit tests, run under Node.js, outside of any Compartment (to be precise it runs in the start compartment). It is critical that libraries should be unit testable without too much fuss, without building an entire swingset kernel to host it.

We have three rough approaches we could take.

globals

The simplest is to provide the collections as a global, perhaps globalThis.collections or globalThis.swingsetCollectionsAPI. Under swingset, liveslots prepares the collections globals at the same time it prepares the #2724 modified WeakMap/WeakSet. It would add these to the globalThis of the new vat Compartment before freezing it. Then we either modify ZCF to copy the globals into the contract’s Compartment (importBundle(contractBundle, { globals: { ...globalThis } })), or we modify importBundle to add a feature that automatically copies some globals into new child compartments (similar to how we currently force-inherit inescapableGlobalLexicals). Or we put the collections API objects on inescapableGlobalLexicals, although that seems unsatisfying.

Under unit tests, library developers are obligated to import a @agoric/fake-collection-shim from their test program, before their library gets a chance to look at the global. This shim would modify the global to add the necessary API objects. This looks just like what we’re already doing with SES/lockdown and eventual-send/HandledPromise. So we’d publish just one module to NPM: @agoric/fake-collection-shim.

module import (plus globals)

The second approach is to provide the collections on a global, but ask users to import { VirtualMap } from '@agoric/collections' anyways. This package (which we would publish to NPM) would contain a trivial re-export of the globals. Unit tests would import @agoric/fake-collection-shim in their test driver file, and import @agoric/collections in their library file. When a vat or contract is bundled for the chain, the archive would include a copy of @agoric/collections. Swingset would load that copy as expected, but would provide the necessary globals first.

The upside (in my mind) is discoverability. Library code that uses collections would have an obvious import statement, users/code-readers could look up @agoric/collections on NPM and find documentation (even if the implementation is elsewhere).

The (well, one) downside is that the global would still be present, and users might come to rely upon it rather than using the imported form. A library could use c = new swingsetCollections.VirtualSet() without any import, and it would work by-accident.

module import (without globals)

To address the last problem, we could have swingset prepare two separate Compartments. In the first one, swingset provides the collections API as a global, imports @agoric/collections, and collects the result as a Module Namespace Object. Then, swingset evaluates the vat code in a second Compartment which lacks the globals, but provides that MNO as a compartment-map override, so when the vat code performs the import, it gets the swingset-provided MNO instead of evaluating the contents of @agoric/collections itself.

The result is that the vat code doesn’t see the globals, partially removing the hazard that people might use the API from the wrong place and come to rely upon it being there. However this doesn’t help the unit test case: the global is still present, and unit test code could come to rely upon it being on the global without error. And unit tests are really the one opportunity we have to educate users about what is supposed to work and what isn’t.

(note: the global name might be sufficiently different than the imported one to reduce this hazard)

Also, this approach is more complicated, because the compartment-map override is new territory (I think it’d be the first time this feature would be used in anger). In particular, the code that creates the bundle needs to be aware of the fact that @agoric/collections is a “hole” to be filled in later, by the swingset-time loading code. If the bundler doesn’t agree with the loader about where the holes are, bad things will happen. One possibility is that the archive contains a copy of @agoric/collections even though the loader is told to replace it, and we think the loader would honor the archive over the override. Another possibility is that the bundler omits a module and the loader doesn’t replace it, in which case the loader will fail outright.

What to do?

@michaelfig suggested the third approach to me (assuming I understood our conversation correctly). @kriskowal recommended the first approach (just use a global) and has talked me out of the third approach (import override). I’m still hoping for the documentation/discovery aspects of the second approach (import), but I’m coming around to using the first (global).

I’m inclined to allow ambient access to Durable, but if we really don’t want that, we might put Virtual on the global, and pass Durable through vatPowers. Since “Durable” is closely tied to upgrade planning, and since upgrade planning also involves the “baggage” index (to pass pointers to Durable collections through to the new version), it might make a lot of sense to pass the Durable constructors through the same object that provides access to the baggage.

Security Considerations

• should Durable be ambient?
• if we use module overrides, do auditors need to be worried about what other modules appeared in the original source but have been replaced by the runtime?
  • correspondingly, in the use-a-global approach, they need to be aware of what globalThis contains and where it comes from
• can these APIs provide a resource-exhaustion vector, or is that sufficiently covered by the Metering we do elsewhere?

Test Plan

[michaelfig]

I’d love to have a module that’s a thin wrapper around globalThis.VatData, and that provides typing information. That’s by far the easiest way to support the need for types and makes dependencies explicit in the source code.

Especially since globalThis.VatData won’t be standardised as part of SES (unlike harden and assert).

[michaelfig]

@agoric/virtualStore ?

Maybe @agoric/virtual-store? Uppercase is not allowed in package names.