Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Pickle incompatibility between 0.25 and 1.0 when saving a MultiIndex dataframe

See original GitHub issue

This seems to have caused the problem described here:

https://stackoverflow.com/questions/61641738/pandas-1-0-cannot-pickle-load-dict-containing-dataframe-with-multiindex

which I’m now also experiencing. I dumped a MultiIndex dataframe containing ndarrays to a pickle on disk under pandas 0.25.x in Python 3.6, and now I’m getting:

AttributeError: Can't get attribute 'FrozenNDArray' on <module 'pandas.core.indexes.frozen'

when trying to load it in pandas 1.0.3 (still on Python 3.6). Any suggestions/workarounds? Should I instead open up a new issue?

This solved issue seems related, but is for Python 2.7:

https://github.com/pandas-dev/pandas/issues/31988

This comment in the original rationale for getting rid of FrozenNDArray mentions pandas.compat.pickle_compat.py, which seems relevant:

https://github.com/pandas-dev/pandas/issues/9031#issuecomment-306659873

_Originally posted by @mspacek in https://github.com/pandas-dev/pandas/pull/29335#issuecomment-628138718_

Issue Analytics

State:
Created 3 years ago
Comments:9 (3 by maintainers)

Top GitHub Comments

4reactions

jrebackcommented, Aug 5, 2020

exactly what i said pickle.load does not handle backward compatibility never has never will

pd.read_pickle does

3reactions

jrebackcommented, Aug 5, 2020

you should simply use pd.read_pickle

Top Results From Across the Web

Pandas 1.0+ cannot pickle.load dict containing DataFrame ...

I store related DataFrames, each with a MultiIndex, in one dict and pickle it. In Pandas 0.22-0.25 I could pickle.load the dict.

What's new in 0.25.0 (July 18, 2019)

Backwards incompatible API changes# · Indexing with date strings with UTC offsets# · MultiIndex constructed from levels and codes# · GroupBy.

What's New - Xarray

Fix an issue that datasets from being saved when time variables with units that cftime can parse but pandas can not were present...

pandas2.py

If it is a MultiIndex, the number of keys in the other DataFrame (either the ... 'epsilon'] >>> float_values = [0.0, 0.25, 0.5,...

Base pandas Dataset API — Modin 0.13.3+0.gbac4031.dirty ...

level (int or level name, default None) – If the axis is a MultiIndex (hierarchical), ... from pandas.api.types import CategoricalDtype >>> cat_dtype ...