question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Weird interaction between Astropy and the Pickle module

See original GitHub issue

Description

Using the astropy si function on an astropy quantity object returns the wrong units after having used the pickle module to read in a dictionary with an astropy quantity object.

Expected behavior

b = 50*u.GHz

print(b.si)
>>>> <Quantity 5.e+10 1 / s>

Actual behavior

b = 50*u.GHz

print(b.si)
>>>> <Quantity 1.35135135 Ci>

Steps to Reproduce

  1. Dump a dictionary containing an astropy object to a pickle file
import pickle
import astropy.units as u

a = {'test': 1*u.GHz }
with open('dictionary.pkl', 'wb') as f:
    pickle.dump(a, f)

  1. In a new file use the si function on an astropy object after having loaded the pickled dictionary
import pickle
import astropy.units as u

with open('dictionary.pkl', 'rb') as f:
    a =  pickle.load(f)

b = 50*u.GHz

print(b.si)
>>>> <Quantity 1.35135135 Ci>

This only seems to be an issue when the pickled object contains an astropy quantity. Additionally, if you dump and load in the same file, the si function works as intended. It appears as if the loading of an astropy object through pickle alters the state of astropy in some way?

I have tried to use the astropy.io.misc functions fnpickle and fnunpickle but to no help (I believe these are simply wrappers of the pickle dump and load functions?).

System Details

macOS-10.16-x86_64-i386-64bit Python 3.8.6 | packaged by conda-forge | (default, Oct 7 2020, 18:42:56) [Clang 10.0.1 ] Numpy 1.19.4 astropy 4.2.1 Scipy 1.5.3 Matplotlib 3.3.3

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:8 (6 by maintainers)

github_iconTop GitHub Comments

2reactions
mhvkcommented, Jun 23, 2021

OK, fixed! Thanks @MetinSa for reporting (must have been a strange one to find!) and thanks @pllim for the good sleuthing that really helped to narrow it down - another nice collaboration!

2reactions
mhvkcommented, Jun 22, 2021

Yes, confirmed! What a weird error. The sorting was a very good clue, and it turns out that after the pickle load, the “score” is 0 for all options, because, somehow, the following is no longer true:

u.s in u.si.bases

i.e., the “s” unit stored in u.si.bases is now different from that on the module – which should never happen…

EDIT: Narrowing it down to the loading

In [1]: import pickle

In [2]: up = pickle.Unpickler(open('dictionary.pkl', 'rb'))

In [3]: import astropy.units as u

In [4]: u.s in u.si.bases
Out[4]: True

In [5]: up.load()
Out[5]: {'test': <Quantity 1. GHz>}

In [6]: u.s in u.si.bases
Out[6]: False
Read more comments on GitHub >

github_iconTop Results From Across the Web

Weird interaction between Astropy and the Pickle module issue ...
This only seems to be an issue when the pickled object contains an astropy quantity. Additionally, if you dump and load in the...
Read more >
HDF5, YAML, ASDF, Parquet, pickle (astropy.io.misc)
Unpickle pickled objects from a specified file and return the contents. astropy.io.misc.hdf5 Module¶. This package contains functions for reading and writing ...
Read more >
Astropy issue comments - Github-Gist
I think we are going with Python 3.x compatibility *now* as much as possible (see http://astropy.org/development/codeguide.html#interface-and-dependencies).
Read more >
Issues With Pickle Module — mod_wsgi 4.9.4 documentation
This article describes various limitations on what data can be stored using the “pickle” module from a WSGI application script file. This arises...
Read more >
astropy.utils.decorators — gammapy v0.18.2
Subclassing the class and return the subclass can lead to problems with pickle and will look weird in the Sphinx docs. """ cls....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found