pianoroll_rnn: Key decoder/multi_rnn_cell/cell_1/basic_lstm_cell/bias not found in checkpoint

I’m using python 3.6.5 on windows, with tensorflow1.7.0 magenta 0.3.7. When I tried to restore from a checkpoint and create a bundle file, it gives me an error message:

WARNING:tensorflow:From C:\Program Files\Python36\lib\site-packages\tensorflow\contrib\learn\python\learn\datasets\base.py:198: retry (from tensorflow.contrib.le
n.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
WARNING:tensorflow:No bundle description provided.
INFO:tensorflow:Saving generator bundle to /tmp/pianoroll_rnn_nade.mag
WARNING:tensorflow:Writing bundle file with no bundle description.
INFO:tensorflow:hparams = {'batch_size': 1, 'rnn_layer_sizes': [128, 128, 128], 'nade_hidden_units': 128, 'dropout_keep_prob': 0.5, 'clip_norm': 5, 'learning_rat
: 0.001}
2018-04-20 17:51:03.498882: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow bin
y was not compiled to use: AVX2
INFO:tensorflow:Checkpoint used: /tmp/pianoroll_rnn_nade/logdir/run1\train\model.ckpt-111
INFO:tensorflow:Restoring parameters from /tmp/pianoroll_rnn_nade/logdir/run1\train\model.ckpt-111
2018-04-20 17:51:03.599090: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1273] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found:
ey decoder/multi_rnn_cell/cell_1/basic_lstm_cell/bias not found in checkpoint
Traceback (most recent call last):
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1327, in _do_call
    return fn(*args)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1312, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1420, in _call_tf_sessionrun
    status, run_metadata)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: Key decoder/multi_rnn_cell/cell_1/basic_lstm_cell/bias not found in checkpoint
         [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device=
job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\pianoroll\pianoroll_rnn_nade_generate.py", line 255, in <module>
    console_entry_point()
  File "D:\pianoroll\pianoroll_rnn_nade_generate.py", line 251, in console_entry_point
    tf.app.run(main)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
    _sys.exit(main(argv))
  File "D:\pianoroll\pianoroll_rnn_nade_generate.py", line 245, in main
    generator.create_bundle_file(bundle_filename, FLAGS.bundle_description)
  File "C:\Program Files\Python36\lib\site-packages\magenta\music\sequence_generator.py", line 223, in create_bundle_file
    self.initialize()
  File "C:\Program Files\Python36\lib\site-packages\magenta\music\sequence_generator.py", line 141, in initialize
    self._model.initialize_with_checkpoint(checkpoint_file)
  File "C:\Program Files\Python36\lib\site-packages\magenta\music\model.py", line 60, in initialize_with_checkpoint
    saver.restore(self._session, checkpoint_file)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1775, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 905, in run
    run_metadata_ptr)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1140, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1321, in _do_run
    run_metadata)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\client\session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key decoder/multi_rnn_cell/cell_1/basic_lstm_cell/bias not found in checkpoint
         [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device=
job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
  File "D:\pianoroll\pianoroll_rnn_nade_generate.py", line 255, in <module>
    console_entry_point()
  File "D:\pianoroll\pianoroll_rnn_nade_generate.py", line 251, in console_entry_point
    tf.app.run(main)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
    _sys.exit(main(argv))
  File "D:\pianoroll\pianoroll_rnn_nade_generate.py", line 245, in main
    generator.create_bundle_file(bundle_filename, FLAGS.bundle_description)
  File "C:\Program Files\Python36\lib\site-packages\magenta\music\sequence_generator.py", line 223, in create_bundle_file
    self.initialize()
  File "C:\Program Files\Python36\lib\site-packages\magenta\music\sequence_generator.py", line 141, in initialize
    self._model.initialize_with_checkpoint(checkpoint_file)
  File "C:\Program Files\Python36\lib\site-packages\magenta\music\model.py", line 57, in initialize_with_checkpoint
    saver = tf.train.Saver()
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1311, in __init__
    self.build()
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1320, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1357, in _build
    build_save=build_save, build_restore=build_restore)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 809, in _build_internal
    restore_sequentially, reshape)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 448, in _AddRestoreOps
    restore_sequentially)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 860, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 1541, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 3290, in create_op
    op_def=op_def)
  File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1654, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key decoder/multi_rnn_cell/cell_1/basic_lstm_cell/bias not found in checkpoint
         [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device=
job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

generate command:

pianoroll_rnn_nade_generate --run_dir=/tmp/pianoroll_rnn_nade/logdir/run1 --bundle_file=/tmp/pianoroll_rnn_nade.mag --save_generator_bundle

train command:

pianoroll_rnn_nade_train --run_dir=/tmp/pianoroll_rnn_nade/logdir/run1 --sequence_example_file=/tmp/pianoroll_rnn_nade/sequence_examples/training_pianoroll_tracks.tfrecord --hparams="batch_size=48,rnn_layer_sizes=[128]" --num_training_steps=20000

[joaopfonseca]

Sure thing, sorry for the delay:

I ran these commands: from os import system

Number of iterations.

TRAIN_STEPS=10 #for testing

Folder containing MIDI and/or MusicXML files. can have child folders.

INPUT_DIRECTORY=‘/home/jfonseca/side_projs/midi_proj/midi_collection’

TFRecord file that will contain NoteSequence protocol buffers.

SEQUENCES_TFRECORD=‘/home/jfonseca/side_projs/midi_proj/note_sequences/notesequences.tfrecord’

SequenceExamples are fed into the model during training and evaluation

SEQUENCES_EXAMPLES=‘/home/jfonseca/side_projs/midi_proj/note_sequences/sequence_examples/’

Run dir

RUN_DIR= ‘/home/jfonseca/side_projs/midi_proj/logdir/run1’

Create dataset

system( “”" convert_dir_to_note_sequences
–input_dir=%s
–output_file=%s
–recursive “”" % (INPUT_DIRECTORY, SEQUENCES_TFRECORD) )

Create SequenceExamples

system( “”" pianoroll_rnn_nade_create_dataset
–input=%s
–output_dir=%s
–eval_ratio=0.10 “”" % (SEQUENCES_TFRECORD, SEQUENCES_EXAMPLES) )

Train and Evaluate the Model

system( “”" pianoroll_rnn_nade_train
–run_dir=%s
–sequence_example_file=%seval_pianoroll_tracks.tfrecord
–hparams=“batch_size=48,rnn_layer_sizes=[128]”
–num_training_steps=%s
“”" % (RUN_DIR, SEQUENCES_EXAMPLES, TRAIN_STEPS) )

system( “”" pianoroll_rnn_nade_generate
–run_dir=%s
–output_dir=/home/jfonseca/midi_proj/generated
–num_outputs=20
–num_steps=128
–primer_pitches=“[67,64,60]”
–hparams=“batch_size=48,rnn_layer_sizes=[128]” “”" % RUN_DIR ) `

I was running each command one by one, the error occurred in the pianoroll_rnn_nade_generate:

Traceback (most recent call last): File "/home/jfonseca/miniconda3/envs/magenta/bin/pianoroll_rnn_nade_generate", line 11, in <module> sys.exit(console_entry_point()) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/magenta/models/pianoroll_rnn_nade/pianoroll_rnn_nade_generate.py", line 251, in console_entry_point tf.app.run(main) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/magenta/models/pianoroll_rnn_nade/pianoroll_rnn_nade_generate.py", line 247, in main run_with_flags(generator) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/magenta/models/pianoroll_rnn_nade/pianoroll_rnn_nade_generate.py", line 210, in run_with_flags generated_sequence = generator.generate(primer_sequence, generator_options) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/magenta/music/sequence_generator.py", line 196, in generate return self._generate(input_sequence, generator_options) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/magenta/models/pianoroll_rnn_nade/pianoroll_rnn_nade_sequence_generator.py", line 121, in _generate total_steps, pianoroll_seq, **args) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/magenta/models/pianoroll_rnn_nade/pianoroll_rnn_nade_model.py", line 98, in generate_pianoroll_sequence steps_per_iteration=steps_per_iteration) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/magenta/models/shared/events_rnn_model.py", line 361, in _generate_events steps_per_iteration=steps_per_iteration) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/magenta/common/beam_search.py", line 134, in beam_search beam_entries, generate_step_fn, branch_factor, first_iteration_num_steps) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/magenta/common/beam_search.py", line 67, in _generate_branches all_sequences, all_states, all_scores) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/magenta/models/shared/events_rnn_model.py", line 215, in _generate_step temperature) File "/home/jfonseca/miniconda3/envs/magenta/lib/python2.7/site-packages/magenta/models/pianoroll_rnn_nade/pianoroll_rnn_nade_model.py", line 75, in _generate_step_for_batch return final_state, loglik[:, 0] IndexError: too many indices for array

Again, thank you for your help!

[antoniomechas]

I have the same problem, creating dataset and training went fine, but same error when using pianoroll_rnn_nade_generate:

  File "/home/gizmow/magenta/magenta/magenta/models/pianoroll_rnn_nade/pianoroll_rnn_nade_model.py", line 75, in _generate_step_for_batch
    return final_state, loglik[:, 0]
IndexError: too many indices for array

Could someone solve this? Thanks!