Allow to resume indexing using blk files after interruption

Currently, restarting the indexer before it finishes indexing completely will cause it to continue fetching blocks using the RPC instead of the .blk files, which is significantly slower.

[bjdweck]

Success! It caught up and is continuing indexing with the block files. Thanks!

^[0m^[0m^[34m2019-06-21T14:19:13.977+00:00 - TRACE - skipping block 00000000000000000040598fcaae5907a2e35b95494dcaefc046934bc7f0d32d
^[0m^[0m^[34m2019-06-21T14:19:13.978+00:00 - TRACE - skipping block 000000000000000001190eb117434d9d2cc40a0a01bcb6ec954f16b565dbfd4e
^[0m^[0m^[34m2019-06-21T14:19:13.978+00:00 - TRACE - skipping block 00000000000000000132152884395e13993c0f227db2d1bfd92e20ffa7492001
^[0m^[0m^[34m2019-06-21T14:19:13.979+00:00 - TRACE - skipping block 000000000000000000b1b5a79a9a679b34b108a255ccaed424d7911443696f18
^[0m^[0m^[34m2019-06-21T14:19:13.979+00:00 - TRACE - fetched 99 blocks
^[0m^[0m^[34m^[0m^[34m2019-06-21T14:19:29.873+00:00 - TRACE - parsing 133588918 bytes
^[0m2019-06-21T14:19:29.873+00:00 - TRACE - reading "/data/bitcoin/blocks/blk00943.dat"
^[0m^[0m^[34m2019-06-21T14:19:30.098+00:00 - TRACE - fetched 139 blocks
^[0m^[0m^[34m^[0m^[34m2019-06-21T14:19:42.202+00:00 - TRACE - reading "/data/bitcoin/blocks/blk00944.dat"
^[0m2019-06-21T14:19:42.202+00:00 - TRACE - parsing 134064939 bytes
^[0m^[0m^[34m^[0m^[34m2019-06-21T14:19:56.908+00:00 - TRACE - reading "/data/bitcoin/blocks/blk00945.dat"
^[0m2019-06-21T14:19:56.908+00:00 - TRACE - parsing 133234079 bytes
^[0m^[0m^[34m2019-06-21T14:20:12.360+00:00 - TRACE - reading "/data/bitcoin/blocks/blk00946.dat"
^[0m^[0m^[36m2019-06-21T14:24:32.846+00:00 - DEBUG - writing 660077 rows to RocksDB { path: "/data/electrs_bitcoin_db/mainnet/mainnet/newindex/history" }, flush=Disable
^[0m^[0m^[34m^[0m^[34m^[0m^[34m2019-06-21T14:24:33.616+00:00 - TRACE - fetched 178 blocks
^[0m2019-06-21T14:24:33.615+00:00 - TRACE - parsing 134209943 bytes
^[0m2019-06-21T14:24:33.616+00:00 - TRACE - reading "/data/bitcoin/blocks/blk00947.dat"

[bjdweck]

It’s much faster pulling from the BlkFiles – but I was left with the curious issue that electrs continued to crash and restart, which was the root of my issue above…

^^^@^^^^^:/tmp/esplora$ sudo docker run --network bridge --volume 
 /raid/fileserver/BitcoinFullNode/esplora:/data --rm -i -t esplora:c34a449 bash -c "/srv/explorer/run.sh bitcoin-mainnet explorer"
Enabled mode explorer
/usr/lib/python2.7/dist-packages/supervisor/options.py:298: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
  'Supervisord is running as root and it is searching '
2019-06-22 18:40:14,930 CRIT Supervisor running as root (no user in config file)
2019-06-22 18:40:14,930 INFO Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2019-06-22 18:40:14,941 INFO RPC interface 'supervisor' initialized
2019-06-22 18:40:14,941 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-06-22 18:40:14,941 INFO supervisord started with pid 25
2019-06-22 18:40:15,945 INFO spawned: 'electrs' with pid 28
2019-06-22 18:40:15,946 INFO spawned: 'nginx' with pid 29
2019-06-22 18:40:15,948 INFO spawned: 'bitcoind' with pid 30
2019-06-22 18:40:15,950 INFO spawned: 'prerenderer' with pid 31
2019-06-22 18:40:17,089 INFO success: electrs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-22 18:40:17,089 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-22 18:40:17,089 INFO success: bitcoind entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-22 18:40:17,089 INFO success: prerenderer entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-22 21:09:46,163 INFO exited: electrs (terminated by SIGKILL; not expected)
2019-06-22 21:09:47,264 INFO spawned: 'electrs' with pid 4457
2019-06-22 21:09:48,582 INFO success: electrs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-23 00:10:07,381 INFO exited: electrs (terminated by SIGKILL; not expected)
2019-06-23 00:10:08,416 INFO spawned: 'electrs' with pid 8852
2019-06-23 00:10:09,763 INFO success: electrs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-23 01:45:53,222 INFO exited: electrs (terminated by SIGKILL; not expected)
2019-06-23 01:45:54,977 INFO spawned: 'electrs' with pid 13239
2019-06-23 01:45:55,966 INFO success: electrs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-06-23 03:40:43,641 INFO exited: electrs (terminated by SIGABRT (core dumped); not expected)
2019-06-23 03:40:44,674 INFO spawned: 'electrs' with pid 17670
2019-06-23 03:40:46,506 INFO success: electrs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

…usually with no indication as to why – except one time where I saw a cut-off line in the error log similar to:

terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc

…which would seem to indicate that my machine is out of memory… and a quick check of top confirms that I am at 97% of both physical and swap… I have 8GB of physical and 8GB of swap and I have shut down almost every process except for Docker and it’s still crashing… but this issue is beyond the scope of this project, so…

THANKS AGAIN!!!