Memory allocation for buffers

With the current implementation of buffers.py one can request a buffersize which doesn’t fit in the memory provided but because of numpys implementation of np.zeros() the memory is not allocated before it is actually used. But because the buffer is meant to be filled completely (otherwise one could just use a smaller buffer) the computer will finally run out of memory and start to swap heavily. Because there are only smaller parts of the buffer that are accessed at once (minibatches) the system will just swap the necessary pages in and out of memory. At that moment the progress of the run is most likely lost and one has to start a new run with a smaller buffer.

I would recommend using np.ones instead, as it will allocate the buffer at the beginning and fail if there is not enough memory provided by the system. The only issue is that there is no clear error description in the case where the system memory is exceeded but python gets simply killed by the OS with a SIGKILL. Maybe one could catch that command?

[ndormann]

oh too slow

[PartiallyTyped]

This should work:

import psutil

import psutil
mem = psutil.virtual_memory()
# svmem(total=17179869184, available=7359184896, percent=57.2, used=9309233152, free=2441043968, active=4924092416, inactive=4230881280, wired=4385140736)
mem.available
# 7359184896

import numpy as np
x = np.empty(1000000000)
x.nbytes
# 8000000000
assert x.nbytes < mem.available
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# AssertionError