Missing tensor in pre-trained model

I’m trying to use the pre-trained model provided in the readme. When I run recognise.py it throws the following error

Traceback (most recent call last):
  File "recognize.py", line 103, in <module>
    saver.restore(sess, tf.train.latest_checkpoint('asset/train/ckpt'))
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/training/saver.py", line 1388, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/client/session.py", line 766, in run
    run_metadata_ptr)
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/client/session.py", line 964, in _run
    feed_dict_string, options, run_metadata)
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
    target_list, options, run_metadata)
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
    raise type(e)(node_def, op, message)
NotFoundError: Tensor name "lyr-conv1d_5/mean" not found in checkpoint files asset/train/ckpt/model-020-45480
	 [[Node: save/RestoreV2_217 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_217/tensor_names, save/RestoreV2_217/shape_and_slices)]]

Caused by op u'save/RestoreV2_217', defined at:
  File "recognize.py", line 102, in <module>
    saver = tf.train.Saver()
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/training/saver.py", line 1000, in __init__
    self.build()
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/training/saver.py", line 1030, in build
    restore_sequentially=self._restore_sequentially)
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/training/saver.py", line 624, in build
    restore_sequentially, reshape)
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/training/saver.py", line 361, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/training/saver.py", line 200, in restore_op
    [spec.tensor.dtype])[0])
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/ops/gen_io_ops.py", line 441, in restore_v2
    dtypes=dtypes, name=name)
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Library/Python/2.7/lib/python/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): Tensor name "lyr-conv1d_5/mean" not found in checkpoint files asset/train/ckpt/model-020-45480
	 [[Node: save/RestoreV2_217 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_217/tensor_names, save/RestoreV2_217/shape_and_slices)]]

I’m using tensorflow-0.12.1 Any help will be much appreciated

[ryanfb]

I ran into this issue as well (though I am running on a GPU with 4GB of RAM). I found that by re-running the training myself under my setup, I was able to produce a training checkpoint that I could use to run recognize.py successfully. I reduced the batch size to 4 as suggested, and after 20 epochs training terminated with a loss of 8.72. Running tensorflow 0.12.1 and sugartensor 0.0.2.3.

I’ve uploaded the resulting checkpoint to Figshare in case it’s usable to anyone else, as training can take quite a long time on a less-powerful GPU: https://figshare.com/articles/speech-to-text-wavenet_VCTK_training_checkpoint/4555483

With it, I get the following for asset/data/wav48/p225/p225_003.wav:

six spoons of fresh snow peas five thick slabs of blue ceese and maybe a snack for he brother bob

[buriburisuri]

I’m completely confused with tf version and sugartensor version because tf updating is so fast and google change library function name without fast version compatibility.

I’ll make docker image include VCTK corpus and pre-trained weights and share.