[Bug?] `RuntimeError: File size unexpectedly exceeded ZIP64 limit` and switching from np.save to np.savez
See original GitHub issueSo I’m using python 3.7, numpy 1.19.5 (the latest version of numpy that supports 3.7). I was training with roughly 4GB MLP models and automatically save them after training, and the runs crashed with RuntimeError: File size unexpectedly exceeded ZIP64 limit
. Upon Googling this error, this is just something that happens in NumPy with matrices larger than 2GB (see https://github.com/numpy/numpy/issues/13153), which is a reasonable thing to support in SB3. While this sure feels like something numpy should fix after looking into the mechanisms of the bug, the official numpy maintainer recommendation was to switch from .save
to .savez
, and I’m aware of no new changes to this.
Issue Analytics
- State:
- Created 2 years ago
- Comments:9 (4 by maintainers)
Top Results From Across the Web
runtime error: file size unexpectedly exceeded zip64 limit
np.savez(os.path.join(data_path,'file.npz'),Arrays). I am getting 'RuntimeError file size unexpectedly exceeded zip64 limit '.
Read more >Numpy Savez, Explained
This tutorial shows how to save multiple Numpy arrays into a single file with Numpy savez. It explains the syntax of np.savez and...
Read more >Release Notes — NumPy v1.17 Manual
This release contains fixes for bugs reported against NumPy 1.17.4 along with ... savez was not using the force_zip64 flag, which limited the...
Read more >[SciPy-User] Maximum file size for .npz format?
Hi! I need to save a fairly large set of arrays to disk. I have saved it using numpy.savez, and the resulting file...
Read more >Release Notes — NumPy v1.17 Manual
In the future it will not be possible to switch the writeable flag to True from python. This deprecation should not affect many...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Quite likely the same issue. Try installing stable-baselines3 from the repository with
pip3 install git+https://github.com/DLR-RM/stable-baselines3
(the change is not on pip yet, I think).Worked! Thanks