electrs: Indexing fails with OversizedVectorAllocation

I confirmed that this is an issue with https://github.com/Blockstream/electrs. I can index with no issues using https://github.com/romanz/electrs. Here’s the full stacktrace:

DEBUG - writing 1257558 rows to RocksDB { path: "./db/mainnet/newindex/txstore" }, flush=Disable
TRACE - parsing 133409003 bytes
TRACE - fetched 136 blocks
TRACE - reading "/home/user/.bitcoin/blocks/blk00978.dat"
thread 'parse-blocks-9' panicked at 'failed to parse Block: OversizedVectorAllocation { requested: 90466760178240, max: 33554432 }', src/libcore/result.rs:997:5
stack backtrace:
   0:     0x5577243017a3 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h50ebfb8734a81144
                               at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1:     0x5577242ffc0b - std::panicking::default_hook::{{closure}}::hc55d0892611a29ff
                               at src/libstd/sys_common/backtrace.rs:71
                               at src/libstd/sys_common/backtrace.rs:59
                               at src/libstd/panicking.rs:197
   2:     0x5577242fec30 - std::panicking::rust_panic_with_hook::h24c9a1c35b1f49cc
                               at src/libstd/panicking.rs:211
                               at src/libstd/panicking.rs:474
   3:     0x5577242fe81d - std::panicking::continue_panic_fmt::h8ed9632bdd4b9299
                               at src/libstd/panicking.rs:381
   4:     0x55772430f145 - rust_begin_unwind
                               at src/libstd/panicking.rs:308
   5:     0x557724313d7b - core::panicking::panic_fmt::h0d6d5c8b201e3246
                               at src/libcore/panicking.rs:85
   6:     0x55772404d026 - core::result::unwrap_failed::h3ebf7163694dfeff
   7:     0x55772414de2f - rayon::iter::plumbing::bridge_producer_consumer::helper::h29134a41eb48cd14
   8:     0x55772414e0e6 - rayon_core::join::join_context::{{closure}}::h7c2357ec41061411
   9:     0x55772414dc2c - rayon::iter::plumbing::bridge_producer_consumer::helper::h29134a41eb48cd14
  10:     0x55772414df6c - rayon_core::join::join_context::{{closure}}::h7c2357ec41061411
  11:     0x55772414dc2c - rayon::iter::plumbing::bridge_producer_consumer::helper::h29134a41eb48cd14
  12:     0x55772414e5bb - <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute::hd88c5e5586054d01
  13:     0x55772407cce0 - rayon_core::registry::WorkerThread::wait_until_cold::hc51d52410ab9f6a5
  14:     0x55772414e01f - rayon_core::join::join_context::{{closure}}::h7c2357ec41061411
  15:     0x55772414dc2c - rayon::iter::plumbing::bridge_producer_consumer::helper::h29134a41eb48cd14
  16:     0x55772414e0e6 - rayon_core::join::join_context::{{closure}}::h7c2357ec41061411
  17:     0x55772414dc2c - rayon::iter::plumbing::bridge_producer_consumer::helper::h29134a41eb48cd14
  18:     0x55772414df6c - rayon_core::join::join_context::{{closure}}::h7c2357ec41061411
  19:     0x55772414dc2c - rayon::iter::plumbing::bridge_producer_consumer::helper::h29134a41eb48cd14
  20:     0x55772414e5bb - <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute::hd88c5e5586054d01
  21:     0x55772407cce0 - rayon_core::registry::WorkerThread::wait_until_cold::hc51d52410ab9f6a5
  22:     0x55772414e01f - rayon_core::join::join_context::{{closure}}::h7c2357ec41061411
  23:     0x55772414dc2c - rayon::iter::plumbing::bridge_producer_consumer::helper::h29134a41eb48cd14
  24:     0x55772414e0e6 - rayon_core::join::join_context::{{closure}}::h7c2357ec41061411
  25:     0x55772414dc2c - rayon::iter::plumbing::bridge_producer_consumer::helper::h29134a41eb48cd14
  26:     0x55772414df6c - rayon_core::join::join_context::{{closure}}::h7c2357ec41061411
  27:     0x55772414dc2c - rayon::iter::plumbing::bridge_producer_consumer::helper::h29134a41eb48cd14
  28:     0x55772414e5bb - <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute::hd88c5e5586054d01
  29:     0x5577241d6395 - rayon_core::registry::WorkerThread::wait_until_cold::h5e2de67a7cd9b2c3
  30:     0x5577241d6007 - std::sys_common::backtrace::__rust_begin_short_backtrace::h4dafa15514cc26c0
  31:     0x5577241d5b00 - std::thread::Builder::spawn_unchecked::{{closure}}::he0b91ba59dc3f7b1
  32:     0x55772430ea7d - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h4cac16ae2114a837
                               at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/liballoc/boxed.rs:702
  33:     0x557724310377 - std::sys::unix::thread::Thread::new::thread_start::h2adc1b80820f790e
                               at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/liballoc/boxed.rs:702
                               at src/libstd/sys_common/thread.rs:14
                               at src/libstd/sys/unix/thread.rs:80
  34:     0x7f7d7a9f16da - start_thread
  35:     0x7f7d7a50288e - __clone
  36:                0x0 - <unknown>
Aborted (core dumped)

[romanz]

lol, why the sad face? It was refactored like that by you iirc 😉

I should have refactored the original electrs to support the new indexing schema, but didn’t had the time to do it…

[shesek]

Have you tried increasing the limit with e.g. ulimit -n 2048?

Might be useful to do this automatically, as was done in electrs: https://github.com/romanz/electrs/commit/4c7413db9d751a0a6b13d35292534f4d35ef22a5

Will PR the fix.

Awesome! Waiting for this 😃

Yes, it got refactored into src/new_index/fetch.rs 😦

lol, why the sad face? It was refactored like that by you iirc 😉