InconsistencyError: Trying to reintroduce a removed node

After a fresh install training hangs at the beginning.

# commit e6f38c1d15b732f1a6c52c03f3ba30157732433d
python main.py

[27/07/2018 23:17:54] Starting training!
[27/07/2018 23:17:54] <<< Training model >>>
[27/07/2018 23:17:54] Training parameters: {'reload_epoch': 0, 'shuffle': True, 'verbose': 1, 'data_augmentation': False, 'class_weights': None, 'epochs_for_save': 1, 'lr_reducer_exp_base': -0.5, 'lr_gamma': 0.8, 'start_eval_on_epoch': 1, 'n_epochs': 500, 'patience': 10, 'tensorboard': True, 'epoch_offset': 0, 'mean_substraction': False, 'normalize': True, 'n_parallel_loaders': 1, 'each_n_epochs': 1, 'patience_check_split': 'val', 'reduce_each_epochs': False, 'lr_warmup_exp': -1.5, 'lr_reducer_type': 'exponential', 'tensorboard_params': {'batch_size': 50, 'write_grads': False, 'write_images': False, 'label_word_embeddings_with_vocab': True, 'embeddings_metadata': None, 'write_graph': True, 'embeddings_layer_names': ['source_word_embedding', 'target_word_embedding'], 'word_embeddings_labels': ['source_text', 'target_text'], 'embeddings_freq': 1, 'log_dir': 'tensorboard_logs', 'histogram_freq': 0}, 'metric_check': 'Bleu_4', 'start_reduction_on_epoch': 0, 'homogeneous_batches': False, 'batch_size': 50, 'num_iterations_val': None, 'normalization_type': '(-1)-1', 'initial_lr': 0.001, 'lr_half_life': 100, 'lr_decay': None, 'joint_batches': 4, 'maxlen': 50, 'eval_on_epochs': True, 'eval_on_sets': [], 'extra_callbacks': [<keras_wrapper.extra.callbacks.EvalPerformance object at 0x7f587f90acd0>, <keras_wrapper.extra.callbacks.Sample object at 0x7f5813395350>]}
[27/07/2018 23:17:57] SeqOptimizer apply <theano.scan_module.scan_opt.PushOutScanOutput object at 0x7f5857fe08d0>
[27/07/2018 23:17:57] Traceback:
[27/07/2018 23:17:57] Traceback (most recent call last):
  File "/home/bzamecnik/.virtualenvs/nmt/local/lib/python2.7/site-packages/theano/gof/opt.py", line 251, in apply
    sub_prof = optimizer.optimize(fgraph)
  File "/home/bzamecnik/.virtualenvs/nmt/local/lib/python2.7/site-packages/theano/gof/opt.py", line 97, in optimize
    ret = self.apply(fgraph, *args, **kwargs)
  File "/home/bzamecnik/.virtualenvs/nmt/local/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 685, in apply
    node = self.process_node(fgraph, node)
  File "/home/bzamecnik/.virtualenvs/nmt/local/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 745, in process_node
    node, args)
  File "/home/bzamecnik/.virtualenvs/nmt/local/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 854, in push_out_inner_vars
    add_as_nitsots)
  File "/home/bzamecnik/.virtualenvs/nmt/local/lib/python2.7/site-packages/theano/scan_module/scan_opt.py", line 906, in add_nitsot_outputs
    reason='scanOp_pushout_output')
  File "/home/bzamecnik/.virtualenvs/nmt/local/lib/python2.7/site-packages/theano/gof/toolbox.py", line 569, in replace_all_validate_remove
    chk = fgraph.replace_all_validate(replacements, reason)
  File "/home/bzamecnik/.virtualenvs/nmt/local/lib/python2.7/site-packages/theano/gof/toolbox.py", line 543, in replace_all_validate
    fgraph.validate()
  File "/home/bzamecnik/.virtualenvs/nmt/local/lib/python2.7/site-packages/theano/gof/toolbox.py", line 434, in validate_
    ret = fgraph.execute_callbacks('validate')
  File "/home/bzamecnik/.virtualenvs/nmt/local/lib/python2.7/site-packages/theano/gof/fg.py", line 594, in execute_callbacks
    fn(self, *args, **kwargs)
  File "/home/bzamecnik/.virtualenvs/nmt/local/lib/python2.7/site-packages/theano/gof/toolbox.py", line 600, in validate
    raise theano.gof.InconsistencyError("Trying to reintroduce a removed node")
InconsistencyError: Trying to reintroduce a removed node

[lvapeab]

Hi, this is a Theano error that appeared with the new backend (see https://github.com/Theano/Theano/issues/5994). Apparently, it appears when using the Theano scan function with masks in Keras.

It is not a critical error and the model keeps working and training. Nevertheless, it is cumbersome. If you use the Theano flag optimizer_excluding=scanOp_pushout_output, the messages are gone.

[chocolocked]

Great! Thanks so much!

On Fri, Jan 25, 2019 at 2:07 AM Álvaro Peris [e-mail] wrote:

Hi @chocolocked https://github.com/chocolocked ,

you need to modify the Theano config. This can be done in the ~/.theanorc file or passing the modifications as environment variables. You have more information about this in the Theano manual http://deeplearning.net/software/theano/library/config.html.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lvapeab/nmt-keras/issues/66#issuecomment-457489393, or mute the thread https://github.com/notifications/unsubscribe-auth/AV9TNDB4O-fCmZqjOOixksAVpPMsNDLxks5vGrtSgaJpZM4VksNb .