Batch normalization

Hello, Chelsea.

The batch normalization documentation says that this ops is not attached to the TensorFlow graph by default. So, there’re two ways to force the updates during training:

• explicitly tell the graph to update ops in tf.GraphKeys.UPDATE_OPS
• or set updates_collections parameter of batch_norm to None.

I don’t see neither of those in the code. Maybe I’m missing something.

I haven’t been able to make the first way work due to while cycle in map_fn function. But the second modification is easy and seems to work. Although, I’m not sure I see any difference in performance.

[ghost]

@cbfinn, as you mentioned before,

I compute the test-time statistics using the test batch of data, instead of computing the average training statistics

This seems to be a bit of cheating especially on test-time. In general, we can assume evaluating only one sample at a time on test-time and then there is no way to get proper statistics for batch_norm. This means the test-set performance will partially dependent on the size of batch.

[cbfinn]

Yes

On Thu, Apr 19, 2018, 7:35 PM Jackie Loong [e-mail] wrote:

Hi, I see your approcach. If I use moving average of the statistics by adding update_op into train ops, Then Need I set train=FALSE when testing use batch_norm function?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cbfinn/maml/issues/9#issuecomment-382949995, or mute the thread https://github.com/notifications/unsubscribe-auth/ABMlAZJQHg-N16-ZxjUNiimpNGK70_XVks5tqUl3gaJpZM4OzLMX .