librosa MemoryError

I keep getting a MemoryError from librosa

$ python train.py --checkpoint_every=1 --data_dir=data
Using default logdir: ./logdir/train/2016-11-05T21-43-34
Trying to restore saved checkpoints from ./logdir/train/2016-11-05T21-43-34 ... No checkpoint found.
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/neural-network/tensorflow-wavenet/wavenet/audio_reader.py", line 91, in thread_main
    for audio, filename in iterator:
  File "/home/neural-network/tensorflow-wavenet/wavenet/audio_reader.py", line 24, in load_generic_audio
    audio, _ = librosa.load(filename, sr=sample_rate, mono=True)
  File "/usr/local/lib/python2.7/dist-packages/librosa/core/audio.py", line 149, in load
    y = np.concatenate(y)
MemoryError

[jyegerlehner]

Looks like you need to either reduce memory consumption, or run on a machine with more memory (either host memory, GPU memory or both).

You can set sample_size (which defaults to 100000). If you set sample_size to, say 64000, then instead of having a [1,100000,512] tensor you’d get a [1,64000,512].

[bmcfee]

[threadjacking thanks to @Cortexelus]

If your input data is in a format supported by libsndfile (eg, flac/wav/ogg but not mp3), I’d recommend using soundfile instead of librosa.load (which sits on top of audioread).

librosa’s loader is really intended for short signals, and is not at all optimized for memory usage. I’ve used soundfile on long signals (~3hr duration) without a hitch though, and since it provides numpy array views of the data, it should play nicely with everything else.

(In case you’re wondering, the main reason we don’t switch librosa over to soundfile instead of audioread is mp3 support.)