NBXplorer synchronization hanging

Describe the bug With a fully synced Bitcoin Core node, NBXplorer nondeterministically hangs at 99% at various block heights when synchronizing for the first time, or when BTCPayServer has been offline for an extended period.

To Reproduce the bug Steps to reproduce the reported bug:

1. Install Bitcoin Core on an Embassy and let it fully sync.
2. Install BTCPayServer on and Embassy, as well as necessary dependencies.
3. Configure and Start the BTCPayServer service.
4. Launch the service and log in - let the nodes sync.
5. Nondeterministically witness NBXplorer block height hanging at 99% synced.

Expected behavior

With a fully synced Bitcoin Core node, NBXplorer should fully sync to the latest block during initial synchronization and not “get stuck” at 99%.

Screenshots

Your BTCPay Environment:

• EmbassyOS v0.2.14 on RPi4/armv7
• BTCPay Server Version: 1.1.2
• NBXplorer Version: 2.1.52
• Bitcoin Core Version: 0.21.1
• Deployment Method: Custom Docker deployment on Start9 Embassy
• Browser: Firefox

Logs (if applicable)

2021-07-28 22:16:09.703 +00:00 [INF] BTC: Explorer url is http://127.0.0.1:24444/

2021-07-28 22:16:09.818 +00:00 [INF] BTC: Cookie file is /datadir/nbxplorer/Main/.cookie

2021-07-28 22:16:10.487 +00:00 [INF] Starting listening NBXplorer (BTC)

2021-07-28 22:16:10.680 +00:00 [INF] Start watching invoices

2021-07-28 22:16:10.786 +00:00 [INF] Starting payment request expiration watcher

2021-07-28 22:16:11.232 +00:00 [INF] 0 pending payment requests being checked since last run

2021-07-28 22:16:11.318 +00:00 [INF] Importing 0 invoices into the search table in 0 pages

2021-07-28 22:16:11.319 +00:00 [INF] Full invoice search import successful

2021-07-28 22:16:11.505 +00:00 [INF] NBXplorer BTC: NotConnected => Synching

2021-07-28 22:17:18.468 +00:00 [INF] User 'cf8994e7-e435-411a-b0ad-221dbabea4b0' logged in.

Additional context

• Bitcoin Core is fully synced
• NBXplorer is properly whitelisted in bitcoind.conf
• In settings-nbx.config, btc.startheight=-1
• NBX cookie auth properly configured
• The block height value does not change with a page refresh
• I’ve read through the synchronization docs and ruled out all scenarios here. Running ps aux --sort -rss yields:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      3035  8.9 16.8 735128 661312 ?       SLl  Jul27 268:09 bitcoind -onion=<redacted>:9050 -externalip=<redacted> -datadir=/root/.bitcoin -conf=/root/
root     26350 18.2  8.2 1223052 325108 ?      Sl   Jul28 237:27 lnd
root      1394  2.1  5.3 677752 211564 ?       SLl  Jul28  28:28 dotnet /nbxplorer/NBXplorer.dll
root     32167  0.0  2.8 236724 111308 ?       Sl   Jul28   0:58 node /app/node_modules/.bin/next start
root     30536  0.7  2.5 211804 99804 ?        Ssl  Jul28   9:48 node /relay/dist/app.js
root      1395  0.6  2.4 711140 96960 ?        SLl  Jul28   9:02 dotnet ./BTCPayServer.dll

• Issues I’ve explored and ruled out their solutions as fixes: #1215 #2134 #143

I’m wondering if some value is getting cached as the current blockchain height, and NBXplorer keeps trying to sync from that point and hanging.

Not using FastSync but wondering if a solution like this is necessary to clear out corrupted data.

Please let me know if you need additional logs or information. Thank you to the team for everything you have created thus far. CC: @ProofOfKeags

[kn0wmad]

Is this still an ongoing issue? @NicolasDorier anything we need to identify this?

Yes. Have a user that just reported this same issue. NBXplorer hanging on block 704140.

Logs:

[pavlenex]

Is this still an ongoing issue? @NicolasDorier anything we need to identify this?