Add dropout and recurrent_dropout to CuDNNLSTM and CuDNNGRU

Native Keras GRU and LSTM layers support dropout and recurrent_dropout, but their CuDNN-accelerated counterparts, CuDNNLSTM and CuDNNGRU, do not. It might be good to add these features. Although CuDNN RNNs do not support dropout natively, it seems to be possible to implement it outside of CuDNN. At least TensorFlow is capable of that. In Keras dropout can be applied either on inputs (dropout), which should be straightforward, or on previous hidden state (recurrent_dropout). I’m not sure if the latter might be possible, tough.

The reason is using CuDNN RNN implementation for fast training and allow dropout regularization at the same time.

Please comment if this makes sense or it is wanted. I’d be happy to try implementing that. Thanks.

[fchollet]

Recurrent dropout is not implemented in cuDNN RNN ops. At the cuDNN level. So we can’t have it in Keras.

The dropout option in the cuDNN API is not recurrent dropout (unlike what is in Keras), so it is basically useless (regular dropout doesn’t work with RNNs).

Actually using such dropout in a stacked RNN will wreck training.

[ml-pickle]

This was would be tremendously helpful to many, many people. Not being able to use dropout often renders CuDNN layers virtually useless for training smaller datasets.