question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

autoreload magic doesn't reload objects

See original GitHub issue

update: the problem appears to be with any object, unrelated to pickle. pickle was just exposing it in a forceful way. Skip to the latest diagnostic https://github.com/ipython/ipython/issues/11588#issuecomment-464949800

when autoreload reloads a python module, it appears to not be fully reloading the already instantiated objects, and pickle fails to pickle them with:

Can't pickle <class 'mytestclass.Test'>: it's not the same object as mytestclass.Test

Here is a notebook to reproduce, must be run in 3 parts (or 3 cells):

# cell1
%reload_ext autoreload
%autoreload 2

import pickle

file = "mytestclass.py"

text = """
class Test():
    def __init__(self):
        self.ok = 1
"""
with open("mytestclass.py","w") as f: f.write(text)

from mytestclass import *
test = Test()

# works
p = pickle.dumps(test)
# cell2
# touch forces autoreloader to reload the library file
!echo touch $file
!touch $file
# cell3
# now it fails
p = pickle.dumps(test)

The error is:

In [6]: p = pickle.dumps(test)
---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
<ipython-input-6-740c326b650e> in <module>()
----> 1 p = pickle.dumps(test)

PicklingError: Can't pickle <class 'mytestclass.Test'>: it's not the same object as mytestclass.Test

Please note that it should be run in 3 parts - if it’s run in one go the reload effect won’t be activated. Can be run in jupyter notebook or directly in ipython shell.

I traced why pickle fails and expanded on it in this issue https://github.com/fastai/fastai/issues/1493 - tldr; it compares memory addresses of the object’s class and the class in sys.modules - if their memory addresses don’t match it refuses to pickle.

I think there are other issues with the incomplete reload, (i.e. object methods disappearing after reload), but I’m yet to make reproducible cases for those. I believe pickle is just exposing a deeper problem.

I tried the git master ipython and went all the way back to ipython-6.0.0 - the problem is the same.

And yes, I read the caveats section, that does say that reload isn’t 100% perfect.

Thank you.

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Reactions:5
  • Comments:22 (9 by maintainers)

github_iconTop GitHub Comments

3reactions
daharncommented, Mar 6, 2019

I have looked into this a little. From what I see, only classes and functions within a reloaded module are updated, but instances of these classes are not touched.

Updating these would be relatively straight forward setting instance.__class__ = new_class with new_class being the updated class reference.

Finding these instances is more complicated. The only way I can think of right now is passing globals() to the update function and iterate through all objects in the workspace (and recursively through all nested objects like lists, dicts etc.), checking whether isinstance(obj, old_class) and then doing the above update.

Although I believe this would be the most logical behavior for %autoreload, I am not sure of the performance costs.

Any other ideas?

2reactions
stas00commented, Feb 19, 2020

So happy you noticed that change, @jdanbrown!

Indeed, both test cases that were failing: https://github.com/ipython/ipython/issues/11588#issuecomment-464949800 https://github.com/ipython/ipython/issues/11588#issue-405929887 no longer fail with ipython-7.9.0.

I’m not sure whether more tests are needed. I will start using the %autoreload feature again and see if I notice any more glitches.

Read more comments on GitHub >

github_iconTop Results From Across the Web

ipython autoreload doesn't work - Stack Overflow
I tried everything. I added the autoreload magic inside code, inside config file, everywhere. I also added the folder of the module to...
Read more >
autoreload — IPython 8.7.0 documentation
Reload all modules AND autoload newly added objects every time before executing the Python code typed. %aimport. List modules which are to be...
Read more >
Automatically Reload Modules with %autoreload
Another one of the magic methods in IPython is related to reloading modules. It's called %autoreload. It's not enabled by default, ...
Read more >
Jupyter / IPython: After editing a module, changes are not ...
1) The simplest and most certain is to restart the ipython kernel after changing an imported module. But this has downsides as well,...
Read more >
#23621 (Module autoreloading doesn't work due to some ...
The general problem with module reloading in Python is that other modules may (and probably do) still have references to classes or functions...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found