Unable to install/import fastparquet

Environment:

• Python version: 3.7.10
• Pip version: 19.3.1
• Operating System: Using Google Colab
• Install method (conda, pip, source): pip

Description:

Unable to import fastparquet library in a google colab session. This issue appeared today (4 days ago I was able to use fastparquet without any complications).

How to Reproduce:

Run the following install command: pip install fastparquet==0.6.0

Get the following error:

Collecting fastparquet==0.6.0
  Downloading https://files.pythonhosted.org/packages/26/99/bc42cc692008f16758272598eb11fc0be192ed608c379a5aa3c957706267/fastparquet-0.6.0.tar.gz (70kB)
     |████████████████████████████████| 71kB 4.0MB/s 
Requirement already satisfied: pandas>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from fastparquet==0.6.0) (1.1.5)
Requirement already satisfied: numba>=0.49 in /usr/local/lib/python3.7/dist-packages (from fastparquet==0.6.0) (0.51.2)
Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.7/dist-packages (from fastparquet==0.6.0) (1.19.5)
Collecting thrift>=0.11.0
  Downloading https://files.pythonhosted.org/packages/97/1e/3284d19d7be99305eda145b8aa46b0c33244e4a496ec66440dac19f8274d/thrift-0.13.0.tar.gz (59kB)
     |████████████████████████████████| 61kB 4.1MB/s 
Collecting cramjam>=2.3.0
  Downloading https://files.pythonhosted.org/packages/0b/c8/cd86a067cd48c479e8b8a83967fff046aa11dd734af5f83d96df6c45fcd4/cramjam-2.3.0-cp37-cp37m-manylinux2010_x86_64.whl (1.4MB)
     |████████████████████████████████| 1.4MB 8.7MB/s 
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.1.0->fastparquet==0.6.0) (2018.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.1.0->fastparquet==0.6.0) (2.8.1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba>=0.49->fastparquet==0.6.0) (56.1.0)
Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in /usr/local/lib/python3.7/dist-packages (from numba>=0.49->fastparquet==0.6.0) (0.34.0)
Requirement already satisfied: six>=1.7.2 in /usr/local/lib/python3.7/dist-packages (from thrift>=0.11.0->fastparquet==0.6.0) (1.15.0)
Building wheels for collected packages: fastparquet, thrift
  Building wheel for fastparquet (setup.py) ... error
  ERROR: Failed building wheel for fastparquet
  Running setup.py clean for fastparquet
  Building wheel for thrift (setup.py) ... done
  Created wheel for thrift: filename=thrift-0.13.0-cp37-cp37m-linux_x86_64.whl size=348136 sha256=c9711b3bba5bcf2b8e07c29bfb6e7b1fd8a4eac9ed1948845431119ab95e8061
  Stored in directory: /root/.cache/pip/wheels/02/a2/46/689ccfcf40155c23edc7cdbd9de488611c8fdf49ff34b1706e
Successfully built thrift
Failed to build fastparquet
Installing collected packages: thrift, cramjam, fastparquet
    Running setup.py install for fastparquet ... error
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-katprh_w/fastparquet/setup.py'"'"'; __file__='"'"'/tmp/pip-install-katprh_w/fastparquet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-629g406g/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

Attempt to install the 0.6.0.post1 version discussed in the issue #598 with the following command: : pip install fastparquet==0.6.0.post1

Fast-parquet seems to be installed successfully with the following terminal output:

Collecting fastparquet==0.6.0.post1
  Downloading https://files.pythonhosted.org/packages/ab/60/efd90686e2e4f6eebd4f1c807de03784e3e0e0fd945709aaab215fd650b7/fastparquet-0.6.0.post1-cp37-cp37m-manylinux2010_x86_64.whl (515kB)
     |████████████████████████████████| 522kB 4.2MB/s 
Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.7/dist-packages (from fastparquet==0.6.0.post1) (1.19.5)
Collecting thrift>=0.11.0
  Downloading https://files.pythonhosted.org/packages/97/1e/3284d19d7be99305eda145b8aa46b0c33244e4a496ec66440dac19f8274d/thrift-0.13.0.tar.gz (59kB)
     |████████████████████████████████| 61kB 4.7MB/s 
Requirement already satisfied: pandas>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from fastparquet==0.6.0.post1) (1.1.5)
Collecting cramjam>=2.3.0
  Downloading https://files.pythonhosted.org/packages/0b/c8/cd86a067cd48c479e8b8a83967fff046aa11dd734af5f83d96df6c45fcd4/cramjam-2.3.0-cp37-cp37m-manylinux2010_x86_64.whl (1.4MB)
     |████████████████████████████████| 1.4MB 6.1MB/s 
Requirement already satisfied: numba>=0.49 in /usr/local/lib/python3.7/dist-packages (from fastparquet==0.6.0.post1) (0.51.2)
Requirement already satisfied: six>=1.7.2 in /usr/local/lib/python3.7/dist-packages (from thrift>=0.11.0->fastparquet==0.6.0.post1) (1.15.0)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.1.0->fastparquet==0.6.0.post1) (2018.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=1.1.0->fastparquet==0.6.0.post1) (2.8.1)
Requirement already satisfied: llvmlite<0.35,>=0.34.0.dev0 in /usr/local/lib/python3.7/dist-packages (from numba>=0.49->fastparquet==0.6.0.post1) (0.34.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from numba>=0.49->fastparquet==0.6.0.post1) (56.1.0)
Building wheels for collected packages: thrift
  Building wheel for thrift (setup.py) ... done
  Created wheel for thrift: filename=thrift-0.13.0-cp37-cp37m-linux_x86_64.whl size=348124 sha256=cb235b7bf55eea56d7b5ba71bf28a7cdfce1726f7b9da5d9aa885d6032fe6d4b
  Stored in directory: /root/.cache/pip/wheels/02/a2/46/689ccfcf40155c23edc7cdbd9de488611c8fdf49ff34b1706e
Successfully built thrift
Installing collected packages: thrift, cramjam, fastparquet
Successfully installed cramjam-2.3.0 fastparquet-0.6.0.post1 thrift-0.13.0

Try to import the library with the following command: import fastparquet

Output:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-c8f9ade0b7f5> in <module>()
----> 1 import fastparquet

/usr/local/lib/python3.7/dist-packages/fastparquet/__init__.py in <module>()
      3 
      4 from .thrift_structures import parquet_thrift
----> 5 from .core import read_thrift
      6 from .writer import write
      7 from . import core, schema, converted_types, api

/usr/local/lib/python3.7/dist-packages/fastparquet/core.py in <module>()
      7     from thrift.protocol.TCompactProtocol import TCompactProtocol
      8 
----> 9 from . import encoding
     10 from .compression import decompress_data
     11 from .converted_types import convert, typemap

/usr/local/lib/python3.7/dist-packages/fastparquet/encoding.py in <module>()
      5 import numba
      6 from numba.experimental import jitclass
----> 7 from .speedups import unpack_byte_array
      8 from .thrift_structures import parquet_thrift
      9 

/usr/local/lib/python3.7/dist-packages/fastparquet/speedups.pyx in init fastparquet.speedups()

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

Library versions on device:

absl-py==0.12.0
alabaster==0.7.12
albumentations==0.1.12
altair==4.1.0
appdirs==1.4.4
argon2-cffi==20.1.0
astor==0.8.1
astropy==4.2.1
astunparse==1.6.3
async-generator==1.10
atari-py==0.2.6
atomicwrites==1.4.0
attrs==20.3.0
audioread==2.1.9
autograd==1.3
Babel==2.9.0
backcall==0.2.0
beautifulsoup4==4.6.3
bleach==3.3.0
blis==0.4.1
bokeh==2.3.1
Bottleneck==1.3.2
branca==0.4.2
bs4==0.0.1
CacheControl==0.12.6
cachetools==4.2.1
catalogue==1.0.0
certifi==2020.12.5
cffi==1.14.5
chainer==7.4.0
chardet==3.0.4
click==7.1.2
cloudpickle==1.3.0
cmake==3.12.0
cmdstanpy==0.9.5
colorcet==2.0.6
colorlover==0.3.0
community==1.0.0b1
contextlib2==0.5.5
convertdate==2.3.2
coverage==3.7.1
coveralls==0.5
crcmod==1.7
cufflinks==0.17.3
cvxopt==1.2.6
cvxpy==1.0.31
cycler==0.10.0
cymem==2.0.5
Cython==0.29.22
daft==0.0.4
dask==2.12.0
datascience==0.10.6
debugpy==1.0.0
decorator==4.4.2
defusedxml==0.7.1
descartes==1.1.0
dill==0.3.3
distributed==1.25.3
dlib==19.18.0
dm-tree==0.1.6
docopt==0.6.2
docutils==0.17
dopamine-rl==1.0.5
earthengine-api==0.1.260
easydict==1.9
ecos==2.0.7.post1
editdistance==0.5.3
en-core-web-sm==2.2.5
entrypoints==0.3
ephem==3.7.7.1
et-xmlfile==1.0.1
fa2==0.3.5
fancyimpute==0.4.3
fastai==1.0.61
fastdtw==0.3.4
fastprogress==1.0.0
fastrlock==0.6
fbprophet==0.7.1
feather-format==0.4.1
filelock==3.0.12
firebase-admin==4.4.0
fix-yahoo-finance==0.0.22
Flask==1.1.2
flatbuffers==1.12
folium==0.8.3
future==0.16.0
gast==0.3.3
GDAL==2.2.2
gdown==3.6.4
gensim==3.6.0
geographiclib==1.50
geopy==1.17.0
gin-config==0.4.0
glob2==0.7
google==2.0.3
google-api-core==1.26.3
google-api-python-client==1.12.8
google-auth==1.28.1
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.4
google-cloud-bigquery==1.21.0
google-cloud-bigquery-storage==1.1.0
google-cloud-core==1.0.3
google-cloud-datastore==1.8.0
google-cloud-firestore==1.7.0
google-cloud-language==1.2.0
google-cloud-storage==1.18.1
google-cloud-translate==1.5.0
google-colab==1.0.0
google-pasta==0.2.0
google-resumable-media==0.4.1
googleapis-common-protos==1.53.0
googledrivedownloader==0.4
graphviz==0.10.1
greenlet==1.0.0
grpcio==1.32.0
gspread==3.0.1
gspread-dataframe==3.0.8
gym==0.17.3
h5py==2.10.0
HeapDict==1.0.1
hijri-converter==2.1.1
holidays==0.10.5.2
holoviews==1.14.3
html5lib==1.0.1
httpimport==0.5.18
httplib2==0.17.4
httplib2shim==0.0.3
humanize==0.5.1
hyperopt==0.1.2
ideep4py==2.0.0.post3
idna==2.10
imageio==2.4.1
imagesize==1.2.0
imbalanced-learn==0.4.3
imblearn==0.0
imgaug==0.2.9
importlib-metadata==3.10.1
importlib-resources==5.1.2
imutils==0.5.4
inflect==2.1.0
iniconfig==1.1.1
intel-openmp==2021.2.0
intervaltree==2.1.0
ipykernel==4.10.1
ipython==5.5.0
ipython-genutils==0.2.0
ipython-sql==0.3.9
ipywidgets==7.6.3
itsdangerous==1.1.0
jax==0.2.12
jaxlib==0.1.65+cuda110
jdcal==1.4.1
jedi==0.18.0
jieba==0.42.1
Jinja2==2.11.3
joblib==1.0.1
jpeg4py==0.1.4
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.3.5
jupyter-console==5.2.0
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
kaggle==1.5.12
kapre==0.1.3.1
Keras==2.4.3
Keras-Preprocessing==1.1.2
keras-vis==0.4.1
kiwisolver==1.3.1
knnimpute==0.1.0
korean-lunar-calendar==0.2.1
librosa==0.8.0
lightgbm==2.2.3
llvmlite==0.34.0
lmdb==0.99
LunarCalendar==0.0.9
lxml==4.2.6
Markdown==3.3.4
MarkupSafe==1.1.1
matplotlib==3.2.2
matplotlib-inline==0.1.2
matplotlib-venn==0.11.6
missingno==0.4.2
mistune==0.8.4
mizani==0.6.0
mkl==2019.0
mlxtend==0.14.0
more-itertools==8.7.0
moviepy==0.2.3.5
mpmath==1.2.1
msgpack==1.0.2
multiprocess==0.70.11.1
multitasking==0.0.9
murmurhash==1.0.5
music21==5.5.0
natsort==5.5.0
nbclient==0.5.3
nbconvert==5.6.1
nbformat==5.1.3
nest-asyncio==1.5.1
networkx==2.5.1
nibabel==3.0.2
nltk==3.2.5
notebook==5.3.1
np-utils==0.5.12.1
numba==0.51.2
numexpr==2.7.3
numpy==1.19.5
nvidia-ml-py3==7.352.0
oauth2client==4.1.3
oauthlib==3.1.0
okgrade==0.4.3
opencv-contrib-python==4.1.2.30
opencv-python==4.1.2.30
openpyxl==2.5.9
opt-einsum==3.3.0
osqp==0.6.2.post0
packaging==20.9
palettable==3.3.0
pandas==1.1.5
pandas-datareader==0.9.0
pandas-gbq==0.13.3
pandas-profiling==1.4.1
pandocfilters==1.4.3
panel==0.11.2
param==1.10.1
parso==0.8.2
pathlib==1.0.1
patsy==0.5.1
pexpect==4.8.0
pickleshare==0.7.5
Pillow==7.1.2
pip-tools==4.5.1
plac==1.1.3
plotly==4.4.1
plotnine==0.6.0
pluggy==0.7.1
pooch==1.3.0
portpicker==1.3.1
prefetch-generator==1.0.1
preshed==3.0.5
prettytable==2.1.0
progressbar2==3.38.0
prometheus-client==0.10.1
promise==2.3
prompt-toolkit==1.0.18
protobuf==3.12.4
psutil==5.4.8
psycopg2==2.7.6.1
ptyprocess==0.7.0
py==1.10.0
pyarrow==3.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.2
pycparser==2.20
pyct==0.4.8
pydata-google-auth==1.1.0
pydot==1.3.0
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
pyemd==0.5.1
pyerfa==1.7.2
pyglet==1.5.0
Pygments==2.6.1
pygobject==3.26.1
pymc3==3.7
PyMeeus==0.5.11
pymongo==3.11.3
pymystem3==0.2.0
PyOpenGL==3.1.5
pyparsing==2.4.7
pyrsistent==0.17.3
pysndfile==1.3.8
PySocks==1.7.1
pystan==2.19.1.1
pytest==3.6.4
python-apt==0.0.0
python-chess==0.23.11
python-dateutil==2.8.1
python-louvain==0.15
python-slugify==4.0.1
python-utils==2.5.6
pytz==2018.9
pyviz-comms==2.0.1
PyWavelets==1.1.1
PyYAML==3.13
pyzmq==22.0.3
qdldl==0.1.5.post0
qtconsole==5.1.0
QtPy==1.9.0
regex==2019.12.20
requests==2.23.0
requests-oauthlib==1.3.0
resampy==0.2.2
retrying==1.3.3
rpy2==3.4.3
rsa==4.7.2
scikit-image==0.16.2
scikit-learn==0.22.2.post1
scipy==1.4.1
screen-resolution-extra==0.0.0
scs==2.1.3
seaborn==0.11.1
Send2Trash==1.5.0
setuptools-git==1.2
Shapely==1.7.1
simplegeneric==0.8.1
six==1.15.0
sklearn==0.0
sklearn-pandas==1.8.0
smart-open==5.0.0
snowballstemmer==2.1.0
sortedcontainers==2.3.0
SoundFile==0.10.3.post1
spacy==2.2.4
Sphinx==1.8.5
sphinxcontrib-serializinghtml==1.1.4
sphinxcontrib-websupport==1.2.4
SQLAlchemy==1.4.7
sqlparse==0.4.1
srsly==1.0.5
statsmodels==0.10.2
sympy==1.7.1
tables==3.4.4
tabulate==0.8.9
tblib==1.7.0
tensorboard==2.4.1
tensorboard-plugin-wit==1.8.0
tensorflow==2.4.1
tensorflow-datasets==4.0.1
tensorflow-estimator==2.4.0
tensorflow-gcs-config==2.4.0
tensorflow-hub==0.12.0
tensorflow-metadata==0.29.0
tensorflow-probability==0.12.1
termcolor==1.1.0
terminado==0.9.4
testpath==0.4.4
text-unidecode==1.3
textblob==0.15.3
textgenrnn==1.4.1
Theano==1.0.5
thinc==7.4.0
tifffile==2021.4.8
toml==0.10.2
toolz==0.11.1
torch==1.8.1+cu101
torchsummary==1.5.1
torchtext==0.9.1
torchvision==0.9.1+cu101
tornado==5.1.1
tqdm==4.41.1
traitlets==5.0.5
tweepy==3.10.0
typeguard==2.7.1
typing-extensions==3.7.4.3
tzlocal==1.5.1
uritemplate==3.0.1
urllib3==1.24.3
vega-datasets==0.9.0
wasabi==0.8.2
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
wordcloud==1.5.0
wrapt==1.12.1
xarray==0.15.1
xgboost==0.90
xkit==0.0.0
xlrd==1.1.0
xlwt==1.3.0
yellowbrick==0.9.1
zict==2.0.0
zipp==3.4.1

[aiqc]

@JacobMaciejewski I ran into this issue in my dependency soup.

The tricky thing is that upgrading numpy 1.20+ vs 1.19.5 breaks compatibility with different versions of tensorflow and pytorch. So I was chasing those in circles.

Thankfully, when I upgraded fastparquet from 0.6.3 to fastparquet==0.7.1 the error does not appear with numpy==1.19.5!

thank you fastparquet team! this unblocked my conference presentation.

[edwintorok]

Updating numpy: yes that works (tried with numpy 1.20.0 and fastparquet.1.6.0), so perhaps another way to solve this bug is to update the dependency on fastparquet to say that it requires numpy >= 1.20 (if its wheel was built with numpy 1.20).