Node crashes with `Error: nullifierToNote mappings must have a corresponding decryptedNote`

What happened?

I upgraded to 0.1.51 a few days ago.

Before I upgraded, my node Accounts was not fully synced and I was informed that should be fine. After I upgraded, the syncing went very quickly, which was great!

Then I issued the deposit-all command and that seemed to go ok except now my node is crashing.

This has happened twice so far now.

I installed and upgraded the node on Windows Subsystem for Linux using npm.

$ ironfish version
name       ironfish
version    0.1.51
git        67da183
runtime    node/16.17.1

Version

v0.1.51

Debug output

$ ironfish debug
Database in use, skipping output that requires database.
Stop the node and run the debug command again to show full output.

Iron Fish version               0.1.51 @ 67da183
Iron Fish library               0.0.28 @ 67da183
Operating system                Linux x64
CPU model(s)                    Intel(R) Core(TM) i5-4690 CPU @ 3.50GHz
CPU threads                     4
RAM total                       12.44 GiB
Heap total                      2.01 GiB
Node version                    v16.17.1
ironfish in PATH                true
Garbage Collector Exposed       false
Telemetry enabled               true
Node name                       null
Block graffiti                  null

Relevant log output

[00:57:23.613] [info] WebSocket server started at :::9033
[00:57:23.954] [info] Connected to the Iron Fish network
[00:57:54.144] [info] Starting sync from Rob3nxV (ddkissfmnode). work: +76431612140838651, ours: 258598, theirs: 258688
[00:57:54.145] [info] Finding ancestor using linear search on last 3 blocks starting at 00000...6396b (258598) from peer Rob3nxV (ddkissfmnode) at 258688
[00:57:54.331] [info] Found peer Rob3nxV (ddkissfmnode) ancestor 00000...6396b, syncing from 258598 -> 258688 (90) after 1 requests
[00:57:54.331] [info] Requesting 5 blocks starting at 00000...6396b (258598) from Rob3nxV (ddkissfmnode)
[00:57:58.631] [info] Added block seq: 258600, hash: 00000...3e0c9, txs: 101, progress: 99.96%, time: 593.1ms
[00:58:03.647] [info] Requesting 5 blocks starting at 00000...efd4f (258603) from Rob3nxV (ddkissfmnode)
[00:58:20.676] [info] Requesting 5 blocks starting at 00000...0a8bc (258608) from Rob3nxV (ddkissfmnode)
[00:58:50.677] [info] Peer Rob3nxV (ddkissfmnode) disconnected (DISCONNECTED) while syncing.
[00:58:54.149] [info] Starting sync from B5uAUqk (tetsugyo). work: +16225344079033059, ours: 258608, theirs: 258627
[00:58:54.150] [info] Finding ancestor using linear search on last 3 blocks starting at 00000...0a8bc (258608) from peer B5uAUqk (tetsugyo) at 258627
[00:58:54.379] [info] Found peer B5uAUqk (tetsugyo) ancestor 00000...0a8bc, syncing from 258608 -> 258627 (19) after 1 requests
[00:58:54.380] [info] Requesting 5 blocks starting at 00000...0a8bc (258608) from B5uAUqk (tetsugyo)
[00:59:24.383] [info] Peer B5uAUqk (tetsugyo) disconnected (DISCONNECTED) while syncing.
[00:59:34.151] [info] Starting sync from lKtliQY (tetsugyo). work: +88219358349890026, ours: 258608, theirs: 258712
[00:59:34.152] [info] Finding ancestor using linear search on last 3 blocks starting at 00000...0a8bc (258608) from peer lKtliQY (tetsugyo) at 258712
[00:59:34.386] [info] Found peer lKtliQY (tetsugyo) ancestor 00000...0a8bc, syncing from 258608 -> 258712 (104) after 1 requests
[00:59:34.386] [info] Requesting 5 blocks starting at 00000...0a8bc (258608) from lKtliQY (tetsugyo)
[01:00:06.582] [info] Requesting 5 blocks starting at 00000...49015 (258613) from lKtliQY (tetsugyo)
[01:00:36.583] [info] Peer lKtliQY (tetsugyo) disconnected (DISCONNECTED) while syncing.
[01:00:41.542] [info] Starting sync from G0hWVz3. work: +84771640496568829, ours: 258613, theirs: 258714
[01:00:41.542] [info] Finding ancestor using linear search on last 3 blocks starting at 00000...49015 (258613) from peer G0hWVz3 at 258714
[01:00:41.833] [info] Found peer G0hWVz3 ancestor 00000...49015, syncing from 258613 -> 258714 (101) after 1 requests
[01:00:41.833] [info] Requesting 5 blocks starting at 00000...49015 (258613) from G0hWVz3
[01:01:11.834] [info] Peer G0hWVz3 disconnected (DISCONNECTED) while syncing.
[01:01:14.152] [info] Starting sync from qBqWA2U (WWW.IRONFISH.HOST). work: +83074237406983153, ours: 258613, theirs: 258711
[01:01:14.152] [info] Finding ancestor using linear search on last 3 blocks starting at 00000...49015 (258613) from peer qBqWA2U (WWW.IRONFISH.HOST) at 258711
[01:01:14.354] [info] Found peer qBqWA2U (WWW.IRONFISH.HOST) ancestor 00000...49015, syncing from 258613 -> 258711 (98) after 1 requests
[01:01:14.354] [info] Requesting 5 blocks starting at 00000...49015 (258613) from qBqWA2U (WWW.IRONFISH.HOST)
[01:01:51.769] [info] Requesting 5 blocks starting at 00000...c0e7d (258618) from qBqWA2U (WWW.IRONFISH.HOST)
[01:02:13.386] [info] Added block seq: 258620, hash: 00000...2e675, txs: 137, progress: 99.96%, time: 621.0ms
[01:02:15.868] [info] Requesting 5 blocks starting at 00000...3197e (258623) from qBqWA2U (WWW.IRONFISH.HOST)
[01:02:34.809] [info] Requesting 5 blocks starting at 00000...3946e (258628) from qBqWA2U (WWW.IRONFISH.HOST)
[01:03:01.520] [info] Requesting 5 blocks starting at 00000...0b752 (258633) from qBqWA2U (WWW.IRONFISH.HOST)
[01:03:31.912] [info] Requesting 5 blocks starting at 00000...17717 (258638) from qBqWA2U (WWW.IRONFISH.HOST)
[01:04:00.135] [info] Added block seq: 258640, hash: 00000...b9a78, txs: 629, progress: 99.97%, time: 2858.5ms
[01:04:05.299] [info] Requesting 5 blocks starting at 00000...2892b (258643) from qBqWA2U (WWW.IRONFISH.HOST)
[01:04:14.123] [info] Requesting 5 blocks starting at 00000...5a3df (258648) from qBqWA2U (WWW.IRONFISH.HOST)
[01:04:31.066] [info] Requesting 5 blocks starting at 00000...b2764 (258653) from qBqWA2U (WWW.IRONFISH.HOST)
/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/assert.js:18
            throw new Error(message || `Expected value not to be undefined`);
                  ^

Error: nullifierToNote mappings must have a corresponding decryptedNote
    at Function.isNotUndefined (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/assert.js:18:19)
    at /usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/wallet/account.js:213:37
    at async LevelupDatabase.withTransaction (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/storage/database/database.js:41:28)
    at async Account.expireTransaction (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/wallet/account.js:198:9)
    at async Wallet.expireTransactions (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/wallet/wallet.js:601:17)
    at async Wallet.eventLoop (/usr/lib/node_modules/ironfish/node_modules/@ironfish/sdk/build/src/wallet/wallet.js:183:9)

Graffiti

null

[hughy]

Hold off on rescanning for now – it will be useful to be able to debug your database in its current state.

In the meantime, if you’d still like to get points for hosting a node I’d suggest doing either of the following:

1. Start a new node with a new datadir, but copy your chain database to the new datadir
2. Move your accounts database out of your current datadir and restart your node (it will create a new, empty account)

[hughy]

This will, unfortunately, keep happening until the database issue is resolved: expireTransactions is running in a loop and will keep trying and failing to process the expired transaction that causes the error.

I’m not sure yet what might have caused this, but two of your account datastores are out of sync, so there’s a broken data reference.

I’ll seed if I can develop a workaround on a branch. The only built-in way to resolve this is to rescan --reset 😱 , but I know you just went through that (long) process.