Batch norm update bug

https://github.com/carpedm20/DCGAN-tensorflow/blob/60aa97b6db5d3cc1b62aec8948fd4c78af2059cd/model.py#L223

In here that you’re intending to just update the discriminator, but you are updating the batch_norm parameters of the generator too.

same holds for here: https://github.com/carpedm20/DCGAN-tensorflow/blob/60aa97b6db5d3cc1b62aec8948fd4c78af2059cd/model.py#L232

You are updating self.g_loss where inside it uses the discriminator weights that are instantiated with train=True.

[youngleec]

Oh I see. It may be a small bug. Because updates_collections=None force the updates for the moving mean and the moving variance in place. We may need to set updates_collections and add the dependency explicitly. Thanks a lot.

[etienne-v]

Hi @po0ya @youngleec . Do you have any solution or advice on the correct way to implement batch norm in this case? What I currently do is the following: I use placeholders for the training flags for when d and g are training:

d_istraining_ph = tf.placeholder(...)
g_istraining_ph = tf.placeholder(...)

When I build the generator and discriminator graph I pass the correct placeholders to d and g:

gz = generator(z, is_training=g_istraining_ph)
d_real = discriminator(x, is_training=d_istraining_ph)
d_fake = discriminator(gz, is_training=d_is_training_ph, reuse=False)

… where I’ve implemented the tf.layers.batch_normalization(...., training=is_training, ...) function inside the generator and discriminator models and the reuse flag is set to True to reuse the same variables for the second call to discriminator.

When building the training operations I collect the associated trainable variables:

d_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope='discriminator')
g_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope='generator')

where I’ve used tf.variable_scope('discriminator', reuse=reuse) and tf.variable_scope('generator') to wrap the discriminator and generator in scopes when I defined them.

The optimizers, gradient calculations and update steps are as follow:

d_opt = tf.train.AdamOptimizer(.....)
g_opt = tf.train.AdamOptimizer(.....)

d_grads_vars = d_opt.compute_gradients(d_loss, d_vars)
g_grads_vars = g_opt.compute_gradients(g_loss, g_vars)

with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS, scope='discriminator')):
    d_train = d_opt.apply_gradients(d_grads_vars)
with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS, scope='generator')):
    g_train = g_opt.apply_gradients(g_grads_vars)

When calling d_train during training, I set d_istraining=True and g_istraining=False, which will force the generator to use the running statistics but not so for the discriminator. In this case I suppose the discriminator’s batch norm statistics will be updated with both real data and fake data from the generator. When calling g_train during training I do the opposite by setting d_istraining=False and g_istraining=True.

Is my reasoning correct?