Speech Recognition Seems To Overfit

Hi, I don’t know if this is an issue with the framework, but did not know where else to ask. I have been training the speech recognition example (speech.yml) for about 80 epochs on a Titan X within a tensorflow-gpu-py3 based docker image. For some reason, the training loss has gone way down, but my validation loss is still very high and the sample predictions it spits out are gibberish.

Example output:

Epoch 80/inf, loss=5.845: 100%|##########| 2432/2432 [07:04<00:00,  7.80samples/s]
Validating, loss=766.122:  94%|#########4| 256/271 [00:21<00:01, 13.70samples/s]
Prediction: "thg asi bw a hnta e tb incotnetk rndegibnrtrlan ty bmna ekftett trelaob"
Truth: "and what inquired missus macpherson has mary ann given you her love"

1. Is this sort of behavior expected this early in training?
2. How long would you expect to have to train this model to start getting reasonable results.

[scottstephenson]

@bharris47 We just put up some bigger data sets. Can you point your train url and checksum to one of these?

URLS:
10 hour(0.9GB):  http://kur.deepgram.com/data/lsc100-10p-train.tar.gz
20 hour(1.8GB):  http://kur.deepgram.com/data/lsc100-20p-train.tar.gz
50 hour (4.5GB):  http://kur.deepgram.com/data/lsc100-50p-train.tar.gz
100 hour (8.9GB): http://kur.deepgram.com/data/lsc100-100p-train.tar.gz

CHECKSUMS:
10 hour:  46354f284588fec8facd2fc6abee6ba9d020e21bdcb26081d3990e96e720d8a6
20 hour:  e8075b10d3750e6532d10cfb2c00aa8047518b1046708fdcab14e4d1f56c499d
50 hour:  14a740c0ea1097b60e3d0df924f2e0175009a997cb9c9b55e27b28fe280afdc0
100 hour: cad3d2aa735d50d4ddb051fd8455f2dd7625ba0bb1c7dd1528da171a10f4fe86

The 10 hour set is double the size of the default dataset for the stock speech example. You can keep going up in scale and by the 50 hour mark, you’re bound to start seeing pretty good output.

Note: The above datasets are fractions of the 100 hour dataset (100%=100p, 50%=50p, …) from librispeech. They should not be concatenated because: 100p contains 50p which contains 20p which contains 10p.

[ajsyp]

Glad to hear! Feel free to use our data format as a template for adding even more data! It’s just a simple tarball.