[BUG] Plotter segfault for some plots

Describe the bug Plotter segfault in phase 1

[121795.335173] chia[78466]: segfault at 40 ip 00007f2f7fb16a6c sp 00007f2f7b248670 error 6 in chiapos.cpython-38-x86_64-linux-gnu.so[7f2f7fadf000+71000]
[121795.335201] Code: c0 02 00 00 48 8d 1c 80 48 c1 e3 04 48 03 9c 24 a0 02 00 00 80 7c 24 2f 01 48 83 9c 24 b0 00 00 00 ff 80 bc 24 2e 01 00 00 80 <c6> 43 40 01 0f 86 3a fc ff ff 48 8b 73 30 48 8b 53 38 8b 8c 24 b8

To Reproduce Start plotting process with following:

ID: faf41e4e666e5b5a932b665fb032049b74d35ae4a18eba30f06f89dbab66e0b2
Plot size is: 32
Buffer size is: 5000MiB
Using 128 buckets
Using 1 threads of stripe size 65536

Expected behavior Completed plot

Relevant log file lines Crash occurred at following point:

Starting phase 1/4: Forward Propagation into tmp files... Mon Nov  2 05:58:17 2020
Computing table 1
F1 complete, time: 189.863 seconds. CPU (99.39%) Mon Nov  2 06:01:27 2020
...
Forward propagation table time: 4354.224 seconds. CPU (98.820%) Mon Nov  2 08:15:36 2020
Computing table 4
        Bucket 0 uniform sort. Ram: 3.620GiB, u_sort min: 3.250GiB, qs min: 0.813GiB.
        Bucket 1 uniform sort. Ram: 3.620GiB, u_sort min: 1.625GiB, qs min: 0.812GiB.
...
        Bucket 76 uniform sort. Ram: 3.620GiB, u_sort min: 1.625GiB, qs min: 0.812GiB.
        Bucket 77 uniform sort. Ram: 3.620GiB, u_sort min: 3.250GiB, qs min: 0.813GiB.

Segmentation fault      (core dumped)

Desktop (please complete the following information):

• OS: Linux
• OS Version/Flavor: 20.04.1 LTS (GNU/Linux 5.4.0-52-generic x86_64)
• CPU: Intel 8Cx3.2GHz (Silver 4215R)

Additional context Contact @eFishCent on keybase for more info.

[wjblanke]

OK I am optimistic we figured this one out. It’s not due to bad RAM. Flakiness may be due to our edge case bug causing STL “undefined results”.

[wjblanke]

OK I got further by finding similar assembly in ProofOfSpace

_ZNSt6vectorI9PlotEntrySaIS0_EEixEm(): /usr/include/c++/9/bits/stl_vector.h:1043 (discriminator 2) return (this->_M_impl._M_start + __n); 14a8e: 48 8d 2c 80 lea (%rax,%rax,4),%rbp _Z13phase1_threadP10THREADDATA(): /home/wjblanke/covid/chiapos/src/phase1.hpp:354 (discriminator 2) PlotEntry& R_entry = bucket_R[idx_R]; 14a92: 41 0f b7 46 02 movzwl 0x2(%r14),%eax _ZNSt6vectorI9PlotEntrySaIS0_EEixEm(): /usr/include/c++/9/bits/stl_vector.h:1043 (discriminator 2) 14a97: 48 c1 e5 04 shl $0x4,%rbp 14a9b: 48 03 ac 24 d0 01 00 add 0x1d0(%rsp),%rbp 14aa2: 00 14aa3: 48 8d 1c 80 lea (%rax,%rax,4),%rbx 14aa7: 48 c1 e3 04 shl $0x4,%rbx 14aab: 48 03 9c 24 f0 01 00 add 0x1f0(%rsp),%rbx 14ab2: 00 _Z13phase1_threadP10THREADDATA(): /home/wjblanke/covid/chiapos/src/phase1.hpp:357 (discriminator 2) matches++; 14ab3: 80 bc 24 15 01 00 00 cmpb $0x1,0x115(%rsp) 14aba: 01 14abb: 48 83 9c 24 80 00 00 sbbq $0xffffffffffffffff,0x80(%rsp) 14ac2: 00 ff /home/wjblanke/covid/chiapos/src/phase1.hpp:362 (discriminator 2) if (metadata_size <= 128) { 14ac4: 80 bc 24 16 01 00 00 cmpb $0x80,0x116(%rsp) 14acb: 80 /home/wjblanke/covid/chiapos/src/phase1.hpp:360 (discriminator 2) R_entry.used = true; 14acc: c6 43 40 01 movb $0x1,0x40(%rbx) /home/wjblanke/covid/chiapos/src/phase1.hpp:362 (discriminator 2) if (metadata_size <= 128) { 14ad0: 0f 86 0a ff ff ff jbe 149e0 <phase1_thread(THREADDATA)+0x1000> /home/wjblanke/covid/chiapos/src/phase1.hpp:375 Bits(R_entry.right_metadata, metadata_size - 128));

Apparently R_entry.used = true is crashing because I assume R_entry is invalid (bucket_R[idx_R] is NULL?)

                    PlotEntry& L_entry = bucket_L[idx_L];
                    PlotEntry& R_entry = bucket_R[idx_R];

                    if (bStripeStartPair)
                        matches++;

                    // Sets the R entry to used so that we don't drop in next iteration
                    R_entry.used = true;
                    // Computes the output pair (fx, new_metadata)
                    if (metadata_size <= 128) {

Now maybe this could be due to bad RAM, or there is something wrong happening with in bucket_R but not in all runs with plotid.