Opening pickled files saved in Google Drive causes Colab notebook to hang

Describe the current behavior Occasionally, opening pickled files saved in Google Drive causes the jupyter notebook to hang. The issue is sporadic. Sometimes the issue happens, other times it does not. Opening 10 pickled files, 1/10 may cause the loading process to hang (i.e., it’s not always the first pickled file that causes the issue). The pickled files were saved at different times but were all saved from within the Colab environment and all saved to Google Drive. None of the files were saved when using GPU. The issue happens both when running with and without the GPU. The pickled files are typically less than 2MB. When the issue occurs, the RAM usage always spikes up to nearly 100% as depicted by the Colab RAM usage chart.

Example of code used to open pickled files:

from google.colab import drive
import pickle
drive.mount('/content/gdrive')
%cd /content/gdrive/MyDrive/
pickle.load(open('test.pkl', 'rb'))

Interrupting the notebook usually does not work. Or if it does work, then after interrupting, the notebook is unusable where running a cell just results in the cell “loading” action to occur with no output.

In this case, the notebook is saved as a gist on Github. But the issue has also occurred when using a notebook that is saved in Google drive.

Occasionally, the cell will die with the following message:

ConnectionAbortedError: [Errno 103] Software caused connection abort

After which the kernel needs to be reloaded, otherwise the following message occurs:

ERROR:root:Internal Python error in the inspect module.
Below is the traceback from this internal error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-8817ba280272>", line 117, in <module>
    table(superv=True)
  File "<ipython-input-3-8817ba280272>", line 77, in table
    b4 = pickle.load(open('[file].pkl', 'rb'))
OSError: [Errno 107] Transport endpoint is not connected: '[file].pkl'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py", line 2040, in showtraceback
    stb = value._render_traceback_()
AttributeError: 'OSError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/IPython/core/ultratb.py", line 1101, in get_records
    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
  File "/usr/local/lib/python3.7/dist-packages/IPython/core/ultratb.py", line 319, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/IPython/core/ultratb.py", line 353, in _fixed_getinnerframes
    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
  File "/usr/lib/python3.7/inspect.py", line 1502, in getinnerframes
    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)
  File "/usr/lib/python3.7/inspect.py", line 1460, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)
  File "/usr/lib/python3.7/inspect.py", line 696, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:
  File "/usr/lib/python3.7/inspect.py", line 725, in getmodule
    file = getabsfile(object, _filename)
  File "/usr/lib/python3.7/inspect.py", line 709, in getabsfile
    return os.path.normcase(os.path.abspath(_filename))
  File "/usr/lib/python3.7/posixpath.py", line 383, in abspath
    cwd = os.getcwd()
OSError: [Errno 107] Transport endpoint is not connected

Output of the javascript console when the notebook has frozen:

initResin external_polymer_binary.js:194:269
Loading... external_polymer_binary.js:2725:55
Loading notebook. https://gist.github.com/[username]/[id]#[notebook title] external_polymer_binary.js:3479:416
Service workers are disabled SecurityError: The operation is insecure. 2 output_binary.js:149:314
Using //@ to indicate sourceURL pragmas is deprecated. Use //# instead outputframe.html:169:14
Failed to register/update a ServiceWorker for scope ‘https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/’: Storage access is restricted in this context due to user settings or private browsing mode. outputframe.html:149:156
Service workers are disabled SecurityError: The operation is insecure. output_binary.js:149:314
Using //@ to indicate sourceURL pragmas is deprecated. Use //# instead outputframe.html:169:14
Failed to register/update a ServiceWorker for scope ‘https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/’: Storage access is restricted in this context due to user settings or private browsing mode. outputframe.html:149:156
Service workers are disabled SecurityError: The operation is insecure. output_binary.js:149:314
Using //@ to indicate sourceURL pragmas is deprecated. Use //# instead outputframe.html:169:14
Failed to register/update a ServiceWorker for scope ‘https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/’: Storage access is restricted in this context due to user settings or private browsing mode. outputframe.html:149:156
Service workers are disabled SecurityError: The operation is insecure. output_binary.js:149:314
Using //@ to indicate sourceURL pragmas is deprecated. Use //# instead outputframe.html:169:14
Failed to register/update a ServiceWorker for scope ‘https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/’: Storage access is restricted in this context due to user settings or private browsing mode. outputframe.html:149:156
Service workers are disabled SecurityError: The operation is insecure. output_binary.js:149:314
Using //@ to indicate sourceURL pragmas is deprecated. Use //# instead outputframe.html:169:14
Failed to register/update a ServiceWorker for scope ‘https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/’: Storage access is restricted in this context due to user settings or private browsing mode. outputframe.html:149:156
Service workers are disabled SecurityError: The operation is insecure. output_binary.js:149:314
Using //@ to indicate sourceURL pragmas is deprecated. Use //# instead outputframe.html:169:14
Failed to register/update a ServiceWorker for scope ‘https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/’: Storage access is restricted in this context due to user settings or private browsing mode. outputframe.html:149:156
Service workers are disabled SecurityError: The operation is insecure. output_binary.js:149:314
Partitioned cookie or storage access was provided to “https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20221006-060055-RC01_479328028” because it is loaded in the third-party context and dynamic state partitioning is enabled.
Partitioned cookie or storage access was provided to “https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20221006-060055-RC01_479328028” because it is loaded in the third-party context and dynamic state partitioning is enabled.
Partitioned cookie or storage access was provided to “https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20221006-060055-RC01_479328028” because it is loaded in the third-party context and dynamic state partitioning is enabled.
Partitioned cookie or storage access was provided to “https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20221006-060055-RC01_479328028” because it is loaded in the third-party context and dynamic state partitioning is enabled.
Partitioned cookie or storage access was provided to “https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20221006-060055-RC01_479328028” because it is loaded in the third-party context and dynamic state partitioning is enabled.
Partitioned cookie or storage access was provided to “https://23eeyef71bm-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20221006-060055-RC01_479328028” because it is loaded in the third-party context and dynamic state partitioning is enabled.
Connected to https://colab.research.google.com/tun/m/m-s-1i9j7zfypj8ds external_polymer_binary.js:6411:1111
LSP: Pyright language server 1.1.267 starting external_polymer_binary.js:6081:223
LSP: Server root directory: /datalab/web/pyright external_polymer_binary.js:6081:223
LSP: Starting service instance "<default>" external_polymer_binary.js:6081:223
LSP: Background analysis(1) root directory: /datalab/web/pyright external_polymer_binary.js:6081:223
LSP: Background analysis(1) started external_polymer_binary.js:6081:223
LSP: Un-registering request handler for 70be23cc-dcba-4b86-8684-c44b478d53e0 failed. external_polymer_binary.js:6081:223
LSP: stubPath typings is not a valid directory. external_polymer_binary.js:6081:223
result was "accepted" errorReason: external_polymer_binary.js:6822:69
Using //@ to indicate sourceURL pragmas is deprecated. Use //# instead outputframe.html:169:14
Failed to register/update a ServiceWorker for scope ‘https://v6r528x2kx-496ff2e9c6d22116-0-colab.googleusercontent.com/’: Storage access is restricted in this context due to user settings or private browsing mode. outputframe.html:149:156
Service workers are disabled SecurityError: The operation is insecure. output_binary.js:149:314
Partitioned cookie or storage access was provided to “https://v6r528x2kx-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20221006-060055-RC01_479328028” because it is loaded in the third-party context and dynamic state partitioning is enabled.
LSP: Pyright language server 1.1.267 starting external_polymer_binary.js:6081:223
LSP: Server root directory: /datalab/web/pyright external_polymer_binary.js:6081:223
LSP: Starting service instance "<default>" external_polymer_binary.js:6081:223
LSP: Background analysis(1) root directory: /datalab/web/pyright external_polymer_binary.js:6081:223
LSP: Background analysis(1) started external_polymer_binary.js:6081:223
LSP: Un-registering request handler for 5ba07ed8-3852-468b-b6a7-3a10c0f17b16 failed. external_polymer_binary.js:6081:223
LSP: stubPath typings is not a valid directory. external_polymer_binary.js:6081:223
Some cookies are misusing the recommended “SameSite“ attribute 1055
Empty string passed to getElementById(). 15 external_polymer_binary.js:515:168
Empty string passed to getElementById(). 7 external_polymer_binary.js:515:168
Empty string passed to getElementById(). external_polymer_binary.js:515:168
Empty string passed to getElementById(). 2 external_polymer_binary.js:515:168
Empty string passed to getElementById(). 3 external_polymer_binary.js:515:168
Feature Policy: Skipping unsupported feature name “accelerometer”. external_polymer_binary.js:3912:330
Feature Policy: Skipping unsupported feature name “autoplay”. external_polymer_binary.js:3912:330
Feature Policy: Skipping unsupported feature name “gyroscope”. external_polymer_binary.js:3912:330
Feature Policy: Skipping unsupported feature name “magnetometer”. external_polymer_binary.js:3912:330
Feature Policy: Skipping unsupported feature name “serial”. external_polymer_binary.js:3912:330
Feature Policy: Skipping unsupported feature name “usb”. external_polymer_binary.js:3912:330
Feature Policy: Skipping unsupported feature name “xr-spatial-tracking”. external_polymer_binary.js:3912:330
Feature Policy: Skipping unsupported feature name “clipboard-write”. external_polymer_binary.js:3912:330
Feature Policy: Skipping unsupported feature name “accelerometer”. external_polymer_binary.js:3913:413
Feature Policy: Skipping unsupported feature name “autoplay”. external_polymer_binary.js:3913:413
Feature Policy: Skipping unsupported feature name “gyroscope”. external_polymer_binary.js:3913:413
Feature Policy: Skipping unsupported feature name “magnetometer”. external_polymer_binary.js:3913:413
Feature Policy: Skipping unsupported feature name “serial”. external_polymer_binary.js:3913:413
Feature Policy: Skipping unsupported feature name “usb”. external_polymer_binary.js:3913:413
Feature Policy: Skipping unsupported feature name “xr-spatial-tracking”. external_polymer_binary.js:3913:413
Feature Policy: Skipping unsupported feature name “clipboard-write”. external_polymer_binary.js:3913:413
Feature Policy: Skipping unsupported feature name “accelerometer”. external_polymer_binary.js:3914:337
Feature Policy: Skipping unsupported feature name “autoplay”. external_polymer_binary.js:3914:337
Feature Policy: Skipping unsupported feature name “gyroscope”. external_polymer_binary.js:3914:337
Feature Policy: Skipping unsupported feature name “magnetometer”. external_polymer_binary.js:3914:337
Feature Policy: Skipping unsupported feature name “serial”. external_polymer_binary.js:3914:337
Feature Policy: Skipping unsupported feature name “usb”. external_polymer_binary.js:3914:337
Feature Policy: Skipping unsupported feature name “xr-spatial-tracking”. external_polymer_binary.js:3914:337
Feature Policy: Skipping unsupported feature name “clipboard-write”. external_polymer_binary.js:3914:337
Content Security Policy: Ignoring “'report-sample'” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https:” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “http:” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “'unsafe-inline'” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/js/bg/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://support.google.com/inapp/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/tools/feedback/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/inproduct_help/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/support/content/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/recaptcha/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/recaptcha/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “'report-sample'” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https:” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “http:” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “'unsafe-inline'” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/js/bg/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://support.google.com/inapp/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/tools/feedback/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/inproduct_help/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/support/content/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/recaptcha/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/recaptcha/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “'report-sample'” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https:” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “http:” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “'unsafe-inline'” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/js/bg/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://support.google.com/inapp/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/tools/feedback/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/inproduct_help/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/support/content/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/recaptcha/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/recaptcha/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “'report-sample'” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https:” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “http:” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “'unsafe-inline'” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/js/bg/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://support.google.com/inapp/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/tools/feedback/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/inproduct_help/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/support/content/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.google.com/recaptcha/” within script-src: ‘strict-dynamic’ specified
Content Security Policy: Ignoring “https://www.gstatic.com/recaptcha/” within script-src: ‘strict-dynamic’ specified
No output window available external_polymer_binary.js:3932:263
Using //@ to indicate sourceURL pragmas is deprecated. Use //# instead outputframe.html:169:14
Failed to register/update a ServiceWorker for scope ‘https://v6r528x2kx-496ff2e9c6d22116-0-colab.googleusercontent.com/’: Storage access is restricted in this context due to user settings or private browsing mode. outputframe.html:149:156
Service workers are disabled SecurityError: The operation is insecure. output_binary.js:149:314
Partitioned cookie or storage access was provided to “https://v6r528x2kx-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20221006-060055-RC01_479328028” because it is loaded in the third-party context and dynamic state partitioning is enabled.
Element.setCapture() is deprecated. Use Element.setPointerCapture() instead. For more help https://developer.mozilla.org/docs/Web/API/Element/setPointerCapture external_polymer_binary.js:521:144
Document.releaseCapture() is deprecated. Use Element.releasePointerCapture() instead. For more help https://developer.mozilla.org/docs/Web/API/Element/releasePointerCapture external_polymer_binary.js:523:102
Empty string passed to getElementById(). external_polymer_binary.js:515:168
Empty string passed to getElementById(). 4 external_polymer_binary.js:515:168
Empty string passed to getElementById(). external_polymer_binary.js:515:168
Empty string passed to getElementById(). 6 external_polymer_binary.js:515:168
Empty string passed to getElementById(). 7 external_polymer_binary.js:515:168

Describe the expected behavior

What web browser you are using Tried in both Chrome and Firefox

Additional context

[cperry-goog]

@RyuuOujiXS please be respectful of other commenters. We do block users for negative behavior on here.

@davidshumway one of the eng is taking a look at this, no timeline as of now.

[davidshumway]

Are you hitting our Drive limits? I just noticed you mentioning an extreme amount of data stored in Drive https://research.google.com/colaboratory/faq.html#drive-timeout

Yes that’s probably the issue. Thanks for pointing out the faq note.