question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Saving large array fails with savez_compressed() but works with save()

See original GitHub issue

I got this strange problem on a CentOS box, Python 2.6.6, Numpy 1.9.1:

[minhle@node069 ~]$ python
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.version.version
'1.9.1'
>>> a = np.ones((775890380,))
>>> np.savez_compressed('/home/minhle/scratch/test.npz', a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/npyio.py", line 560, in savez_compressed
    _savez(file, args, kwds, True)
  File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/npyio.py", line 597, in _savez
    format.write_array(fid, np.asanyarray(val))
  File "/home/minhle/.local/lib/python2.6/site-packages/numpy/lib/format.py", line 562, in write_array
    array.tofile(fp)
IOError: 775890380 requested and 233691638 written
>>> np.save('/home/minhle/scratch/test.npy', a)
>>> b = np.load('/home/minhle/scratch/test.npy')
>>> b[:10]
array([ 1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.,  1.])
>>> quit()
[minhle@node069 ~]$ cat /etc/*-release
CentOS release 6.5 (Final)
Cluster Manager v5.2
slave
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
CentOS release 6.5 (Final)
CentOS release 6.5 (Final)

Issue Analytics

  • State:closed
  • Created 9 years ago
  • Comments:9 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
samuelacommented, Jul 10, 2015

I just got this error too but with np.savez as opposed to np.savez_compressed.

1reaction
pvcommented, Jul 21, 2015

savez needs to store files temporarily on disk, and perhaps you run out of space on /tmp. Try setting TMPDIR=$HOME/tmp or so.

Read more comments on GitHub >

github_iconTop Results From Across the Web

what reliable method to save huge numpy arrays
I saved some arrays using numpy.savez_compressed(). One of the arrays is gigantic, it has the shape (120000,7680), type float32.
Read more >
How to Save a NumPy Array to File for Machine Learning
You can save your NumPy arrays to CSV files using the savetxt() function. This function takes a filename and array as arguments and...
Read more >
Why You Should Save NumPy Arrays with Zarr
This post tells you why and how to use the Zarr format to save a numpy array. It walks you through the code...
Read more >
NumPy: the absolute basics for beginners
An array consumes less memory and is convenient to use. NumPy uses much less memory to store data and it provides a mechanism...
Read more >
Saving and loading data — deepdish 0.3.5.git documentation
... store lists and tuples, but it's not as natural, so prefer numpy arrays whenever possible. Here's an example saving and HDF5 using...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found