Excessive memory consumption

The network currently runs into out of memory issues at a low number of layers. This seems to be a problem with TensorFlow’s atrous_conv2d operation. If I set the dilation factor to 1, which means atrous_conv2d simply calls conv2d, I can easily run with 10s of layers. It could just be the additional batch_to_space and space_to_batch operations, in which case I can write a single C++ op for atrous_conv2d.

[ibab]

I’ve fixed the problem in 8add545787140187bbce93f7e63b0bb70150a843. Like atrous_conv2d, I swap the width dimension out into the batch dimension and perform a regular convolution, but without padding in the height dimension. It’s now possible to train large stacks of dilation layers without running out of memory.

Judging from occasional garbage collection log messages, I think the issue mentioned by @jyegerlehner is also valid. It would probably make sense to cut inputs to a fixed size.

[ibab]

I’m still in the process of finding good hyperparameters, and finding the cause of the generation issue in #13. After that, I’ll generate audio samples and provide statistics on how long it takes to train the model.