Saving large array fails with savez_compressed() but works with save()
See original GitHub issueI got this strange problem on a CentOS box, Python 2.6.6, Numpy 1.9.1:
[minhle@node069 ~]$ python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.version.version
'1.9.1'
>>> a = np.ones((775890380,))
>>> np.savez_compressed('/home/minhle/scratch/test.npz', a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/npyio.py", line 560, in savez_compressed
_savez(file, args, kwds, True)
File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/npyio.py", line 597, in _savez
format.write_array(fid, np.asanyarray(val))
File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/format.py", line 562, in write_array
array.tofile(fp)
IOError: 775890380 requested and 233691638 written
>>> np.save('/home/minhle/scratch/test.npy', a)
>>> b = np.load('/home/minhle/scratch/test.npy')
>>> b[:10]
array([ 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
>>> quit()
[minhle@node069 ~]$ cat /etc/*-release
CentOS release 6.5 (Final)
Cluster Manager v5.2
slave
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)
Issue Analytics
- State:
- Created 9 years ago
- Comments:9 (3 by maintainers)
Top Results From Across the Web
what reliable method to save huge numpy arrays
I saved some arrays using numpy.savez_compressed(). One of the arrays is gigantic, it has the shape (120000,7680), type float32.
Read more >How to Save a NumPy Array to File for Machine Learning
You can save your NumPy arrays to CSV files using the savetxt() function. This function takes a filename and array as arguments and...
Read more >Why You Should Save NumPy Arrays with Zarr
This post tells you why and how to use the Zarr format to save a numpy array. It walks you through the code...
Read more >NumPy: the absolute basics for beginners
An array consumes less memory and is convenient to use. NumPy uses much less memory to store data and it provides a mechanism...
Read more >Saving and loading data — deepdish 0.3.5.git documentation
... store lists and tuples, but it's not as natural, so prefer numpy arrays whenever possible. Here's an example saving and HDF5 using...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
I just got this error too but with
np.savez
as opposed tonp.savez_compressed
.savez
needs to store files temporarily on disk, and perhaps you run out of space on/tmp
. Try settingTMPDIR=$HOME/tmp
or so.