'You must feed a value for placeholder tensor' when using Tensorboard callback with submodels

Please make sure that the boxes below are checked before you submit your issue. If your issue is an implementation question, please ask your question on StackOverflow or join the Keras Slack channel and ask there instead of filing a GitHub issue.

Thank you!

• Check that you are up-to-date with the master branch of Keras. You can update with: pip install git+git://github.com/keras-team/keras.git --upgrade --no-deps

• If running on TensorFlow, check that you are up-to-date with the latest version. The installation instructions can be found here.

• If running on Theano, check that you are up-to-date with the master branch of Theano. You can update with: pip install git+git://github.com/Theano/Theano.git --upgrade --no-deps

• Provide a link to a GitHub Gist of a Python script that can reproduce your issue (or just copy the script here if it is short).

Trying to train a model that is composed of submodels, and using the Tesnorboard callback. If histogram_freq is set, there’s an error about missing input to the submodel

using tensorflow 1.8 (same error also in older versions)

sample code that reproduces the error:

import tensorflow
keras = tensorflow.keras
import numpy


INPUT_LEN = 10
N = 100


def get_sub_model(in_dim, out_dim):
    input_layer = keras.layers.Input(shape=(in_dim,))
    dense=keras.layers.Dense(out_dim)
    return keras.models.Model(inputs=input_layer, outputs=dense(input_layer))


input_layer = keras.layers.Input(shape=(INPUT_LEN,), name='model_input')
sub_model1 = get_sub_model(INPUT_LEN, 5)
sub_model2 = get_sub_model(5, 1)
m = keras.models.Model(inputs=[input_layer], outputs=sub_model2(sub_model1(input_layer)))
m.compile(loss='binary_crossentropy', optimizer='adam')

x = numpy.random.rand(N, INPUT_LEN)
y = numpy.random.randint(0, 2, N)

m.fit(x, y, callbacks=[keras.callbacks.TensorBoard(log_dir='/tmp', histogram_freq=1)], epochs=10, validation_data=(x,y))

error:

Traceback (most recent call last):
  File "/Users/ophir/dev/ophir/tensorboard_issue.py", line 25, in <module>
    m.fit(x, y, callbacks=[keras.callbacks.TensorBoard(log_dir='/tmp', histogram_freq=1)], epochs=10, validation_data=(x,y))
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/engine/training.py", line 1216, in fit
    validation_steps=validation_steps)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/engine/training_arrays.py", line 269, in fit_loop
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/callbacks.py", line 95, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/keras/_impl/keras/callbacks.py", line 799, in on_epoch_end
    result = self.sess.run([self.merged], feed_dict=feed_dict)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/Users/ophir/miniconda3/envs/p3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'input_1' with dtype float and shape [?,10]
	 [[Node: input_1 = Placeholder[dtype=DT_FLOAT, shape=[?,10], _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

The reason of using submodels is the need to also run them separately

[pdpino]

I had the same problem, that only appeared when using TensorBoard callback and histogram_freq other than 0. Clearing tensorflow session right before creating the model fixed it:

import keras.backend as K
K.clear_session()

From this issue

[igorgad]

I am having the same problem here. Calling K.clear_session() is not helping on my case. Any workaround?