DDPG example fails adding BatchNormalization with Tensorflow

If I take the DDPG example and add a BatchNormalization layer to either the actor or critic model, I get a TypeError with Tensorflow (but not Theano):

Using TensorFlow backend.
Traceback (most recent call last):
  File "ddpg_pendulum.py", line 27, in <module>
    actor.add(BatchNormalization())         # Fails with Tensorflow
  File "/python/lib/python3.4/site-packages/keras/models.py", line 332, in add
    output_tensor = layer(self.outputs[0])
  File "/python/lib/python3.4/site-packages/keras/engine/topology.py", line 572, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/python/lib/python3.4/site-packages/keras/engine/topology.py", line 635, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/python/lib/python3.4/site-packages/keras/engine/topology.py", line 166, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/python/lib/python3.4/site-packages/keras/layers/normalization.py", line 133, in call
    broadcast_running_mean = K.reshape(self.running_mean, broadcast_shape)
  File "/python/lib/python3.4/site-packages/keras/backend/tensorflow_backend.py", line 1463, in reshape
    return tf.reshape(x, shape)
  File "/python/lib/python3.4/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2448, in reshape
    name=name)
  File "/python/lib/python3.4/site-packages/tensorflow/python/framework/op_def_library.py", line 493, in apply_op
    raise err
  File "/python/lib/python3.4/site-packages/tensorflow/python/framework/op_def_library.py", line 490, in apply_op
    preferred_dtype=default_dtype)
  File "/python/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 669, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/python/lib/python3.4/site-packages/tensorflow/python/framework/constant_op.py", line 176, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/python/lib/python3.4/site-packages/tensorflow/python/framework/constant_op.py", line 165, in constant
    tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/python/lib/python3.4/site-packages/tensorflow/python/framework/tensor_util.py", line 441, in make_tensor_proto
    tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values])
  File "/python/lib/python3.4/site-packages/tensorflow/python/framework/tensor_util.py", line 441, in <listcomp>
    tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values])
  File "/python/lib/python3.4/site-packages/tensorflow/python/util/compat.py", line 65, in as_bytes
    (bytes_or_text,))
TypeError: Expected binary or unicode string, got 1

Code to reproduce it (only two lines have been added):

import numpy as np
import gym

from keras.models import Sequential, Model
from keras.layers import Dense, Activation, Flatten, Input, merge, BatchNormalization
from keras.optimizers import Adam

from rl.agents import DDPGAgent
from rl.memory import SequentialMemory
from rl.random import OrnsteinUhlenbeckProcess


ENV_NAME = 'Pendulum-v0'
gym.undo_logger_setup()


# Get the environment and extract the number of actions.
env = gym.make(ENV_NAME)
np.random.seed(123)
env.seed(123)
assert len(env.action_space.shape) == 1
nb_actions = env.action_space.shape[0]

# Next, we build a very simple model.
actor = Sequential()
actor.add(Flatten(input_shape=(1,) + env.observation_space.shape))
actor.add(BatchNormalization())         # Fails with Tensorflow
actor.add(Dense(16))
actor.add(Activation('relu'))
actor.add(Dense(16))
actor.add(Activation('relu'))
actor.add(Dense(16))
actor.add(Activation('relu'))
actor.add(Dense(nb_actions))
actor.add(Activation('linear'))
print(actor.summary())

action_input = Input(shape=(nb_actions,), name='action_input')
observation_input = Input(shape=(1,) + env.observation_space.shape, name='observation_input')
flattened_observation = Flatten()(observation_input)
x = merge([action_input, flattened_observation], mode='concat')
x = BatchNormalization()(x)            # Fails with Tensorflow
x = Dense(32)(x)
x = Activation('relu')(x)
x = Dense(32)(x)
x = Activation('relu')(x)
x = Dense(32)(x)
x = Activation('relu')(x)
x = Dense(1)(x)
x = Activation('linear')(x)
critic = Model(input=[action_input, observation_input], output=x)
print(critic.summary())

# Finally, we configure and compile our agent. You can use every built-in Keras optimizer and
# even the metrics!
memory = SequentialMemory(limit=100000, window_length=1)
random_process = OrnsteinUhlenbeckProcess(size=nb_actions, theta=.15, mu=0., sigma=.3)
agent = DDPGAgent(nb_actions=nb_actions, actor=actor, critic=critic, critic_action_input=action_input,
                  memory=memory, nb_steps_warmup_critic=100, nb_steps_warmup_actor=100,
                  random_process=random_process, gamma=.99, target_model_update=1e-3)
agent.compile(Adam(lr=.001, clipnorm=1.), metrics=['mae'])

# Okay, now it's time to learn something! We visualize the training here for show, but this
# slows down training quite a lot. You can always safely abort the training prematurely using
# Ctrl + C.
agent.fit(env, nb_steps=50000, visualize=True, verbose=1, nb_max_episode_steps=200)

# After training is done, we save the final weights.
agent.save_weights('ddpg_{}_weights.h5f'.format(ENV_NAME), overwrite=True)

# Finally, evaluate our algorithm for 5 episodes.
agent.test(env, nb_episodes=5, visualize=True, nb_max_episode_steps=200)

I’m on the latest master of everything.

$ python -V; pip freeze | egrep -i 'keras|theano'
Python 3.4.3+
Keras==1.2.2
keras-rl==0.2.0
Theano==0.8.2

[ParthaEth]

Hi, Since in Keras2.0 mode option in BatchNormalization has been removed it would be great to be able to support mode=0, which is the only mode supported in Keras 2.0. Is there any plan on that?

[chpohl]

I ran into a similar situation. I get a different error message, but I guess the underlying problem is the same:

tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'flatten_1_input' with dtype float and shape [?,1,17]
	 [[Node: flatten_1_input = Placeholder[dtype=DT_FLOAT, shape=[?,1,17], _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]
	 [[Node: sequential_1/activation_3/Tanh/_401 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_839_sequential_1/activation_3/Tanh", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

A temporary fix to use Keras > 2.0 is setting trainable=False for the BatchNormalization. I tried debugging this and found out that the if DDPGAgent.uses_learning_phase is true, something goes wrong once the warmup steps for the actor are completed. I followed the error to keras.backend.tensorflow_backend._call(), until where everything runs appropriately. But once tf_session.TF_SessionRunCallable() is called the error is thrown and I can’t figure out why. Hope that helps someone.