Crash caused by SIGSEGV in unpack_byte_array

What happened: Reading a parquet file to a pandas dataframe, fastparquet crashes due to an error in unpack_byte_array. I cannot share the parquet file due to PII. It has been created by AWS DMS.

Dump from parquet-tools - column name is ‘request’:

request:         OPTIONAL BINARY L:STRING R:0 D:1
--
request:          BINARY UNCOMPRESSED DO:0 FPO:1238 SZ:358286/358286/1.00 VC:10 ENC:RLE,PLAIN ST:[no stats for this column]

Ultimately this ends up with a call to:

read_plain(io_obj.read(), type_=6, count=10, width=None, utf=True, stat=False)

With a 283180 length byte array.

What you expected to happen: No crashes on reading these parquet files.

Minimal Complete Verifiable Example:

from fastparquet.encoding import read_plain
from fastparquet.cencoding import NumpyIO

def test_read_plain():
    # raw_bytes = b'a' * 283180 # will also cause a SIGSEGV
    raw_bytes = b'iJwY'

    io_obj = NumpyIO(raw_bytes)
    
    read_plain(io_obj.read(), type_=6, count=1, width=None, utf=True, stat=False)

Run this and you will get a stack dump:

tests/unit/test_fastparquet.py::test_read_plain Fatal Python error: Segmentation fault

Current thread 0x000000011abb4e00 (most recent call first):
  File "/Users/nw/.virtualenvs/bp-lambdas-2/lib/python3.8/site-packages/fastparquet/encoding.py", line 41 in read_plain
  File "/Users/nw/dev/src/backup-pipeline/transformer-v2/tests/unit/test_fastparquet.py", line 11 in test_read_plain
  <snip>
fish: Job 1, 'pytest --pdb -vv tests/unit/tes…' terminated by signal SIGSEGV (Address boundary error)

Python version: Python 3.8.11 Operating System: Mac / AWS Lambda Lib Versions: fastparquet==0.7.1, thrift==0.13.0, numpy==1.21.2

[martindurant]

(also, your versions of OS, python, thrift and numpy, if you wouldn’t mind)

[martindurant]

Thanks for the extra information, it looks like it may be helpful, but will take a little time for me to digest.

Are you using fastparquet 0.7.1 ?