question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Bug?] `RuntimeError: File size unexpectedly exceeded ZIP64 limit` and switching from np.save to np.savez

See original GitHub issue

So I’m using python 3.7, numpy 1.19.5 (the latest version of numpy that supports 3.7). I was training with roughly 4GB MLP models and automatically save them after training, and the runs crashed with RuntimeError: File size unexpectedly exceeded ZIP64 limit. Upon Googling this error, this is just something that happens in NumPy with matrices larger than 2GB (see https://github.com/numpy/numpy/issues/13153), which is a reasonable thing to support in SB3. While this sure feels like something numpy should fix after looking into the mechanisms of the bug, the official numpy maintainer recommendation was to switch from .save to .savez, and I’m aware of no new changes to this.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:9 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
Miffylicommented, Apr 14, 2022

Quite likely the same issue. Try installing stable-baselines3 from the repository with pip3 install git+https://github.com/DLR-RM/stable-baselines3 (the change is not on pip yet, I think).

0reactions
ChipHennigcommented, Apr 14, 2022

Worked! Thanks

Read more comments on GitHub >

github_iconTop Results From Across the Web

runtime error: file size unexpectedly exceeded zip64 limit
np.savez(os.path.join(data_path,'file.npz'),Arrays). I am getting 'RuntimeError file size unexpectedly exceeded zip64 limit '.
Read more >
Numpy Savez, Explained
This tutorial shows how to save multiple Numpy arrays into a single file with Numpy savez. It explains the syntax of np.savez and...
Read more >
Release Notes — NumPy v1.17 Manual
This release contains fixes for bugs reported against NumPy 1.17.4 along with ... savez was not using the force_zip64 flag, which limited the...
Read more >
[SciPy-User] Maximum file size for .npz format?
Hi! I need to save a fairly large set of arrays to disk. I have saved it using numpy.savez, and the resulting file...
Read more >
Release Notes — NumPy v1.17 Manual
In the future it will not be possible to switch the writeable flag to True from python. This deprecation should not affect many...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found