Chain recovery startup allows economic recovery first

What this means to me is:

• when our chain halts (because of some bug, or maybe a planned upgrade), it may be offline for a while until we get a fix ready
• the rest of the world moves on anyways, so things like asset prices can change considerably by the time our chain wakes back up
• from the chain’s point of view, no time passes, so it observes a sudden discontinuity in the asset price
  • this doesn’t happen until the first price-oracle message arrives after chain restart
• if I can submit a txn just as the chain is restarting, I might be able to take advantage of the chain’s temporary ignorance of the new state of the world
• everyone outside the chain knows when it is about to restart, and everybody wants to take that advantage, so we’re likely to get a flood of messages all fighting with each other to claim the “prize”
• those messages would compete with the actual price-oracle signal, making the problem worse

The fix we talked about was to do a “soft restart”, in which the chain is told that it is restarting, and for the first minute or so, it does not accept any messages other than economy-critical price-oracle signals.

We’d implement this with the #5334 backpressure mechanism which controls ingress at the mempool/txn level to exclude non-oracle-signed transactions from blocks during the restart window, plus some code in the new version that knows when this window starts and ends. If the chain halted just after block 100, such that the next block executed will be 101, then our replacement/upgraded software should have something in cosmic-swingset that does:

if (blockHeight === 101) {
  disableNonEconomicTxs();
} else if (blockHeight === 111) {
  enableNonEconomicTxs();
}

to give roughly 60 seconds for the economic engine to get prepared for user requests. We’d also need to ensure that the oracle price signals / etc can be delivered during that window, even if user requests are flooding the RPC servers/etc.

We might consider making this more explicit: let the vats that manage vaults give a signal when they believe they’re up to date, and disable non-economic messages until that point. That might mean control over the non-economic admissibility should be made available to userspace, which would be… exciting. It would also want a way for the cosmic-swingset layer to signal to those economy vats that we’d entered soft-start mode, and that the vats are responsible for exiting it when they’re ready.

if (blockHeight === 101) {
  disableNonEconomicTxs();
  controller.queueToVat(economy, 'economyPaused');
  // economy vats will re-enable the non-economic txs after getting a price update
}

@rowgraus points out that this feature could easily consume more effort than it warrants, and/or could expose more of an attack surface than it addresses, and I agree. I think we’ll need to invoke our economist friends for advice too.

[Tartuffo]

@mhofman I was able to convert it to a regular issue.

[Tartuffo]

@mhofman Can we remove the in-design label from this one?