Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Batch normalization

See original GitHub issue

Hello, Chelsea.

The batch normalization documentation says that this ops is not attached to the TensorFlow graph by default. So, there’re two ways to force the updates during training:

  • explicitly tell the graph to update ops in tf.GraphKeys.UPDATE_OPS
  • or set updates_collections parameter of batch_norm to None.

I don’t see neither of those in the code. Maybe I’m missing something.

I haven’t been able to make the first way work due to while cycle in map_fn function. But the second modification is easy and seems to work. Although, I’m not sure I see any difference in performance.

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

ghostcommented, Jan 12, 2018

@cbfinn, as you mentioned before,

I compute the test-time statistics using the test batch of data, instead of computing the average training statistics

This seems to be a bit of cheating especially on test-time. In general, we can assume evaluating only one sample at a time on test-time and then there is no way to get proper statistics for batch_norm. This means the test-set performance will partially dependent on the size of batch.

cbfinncommented, Apr 20, 2018


On Thu, Apr 19, 2018, 7:35 PM Jackie Loong wrote:

Hi, I see your approcach. If I use moving average of the statistics by adding update_op into train ops, Then Need I set train=FALSE when testing use batch_norm function?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread .

Read more comments on GitHub >

github_iconTop Results From Across the Web

Batch normalization - Wikipedia
Batch normalization (also known as batch norm) is a method used to make training of artificial neural networks faster and more stable through...
Read more >
A Gentle Introduction to Batch Normalization for Deep Neural ...
Batch normalization is a technique for training very deep neural networks that standardizes the inputs to a layer for each mini-batch.
Read more >
Batch normalization in 3 levels of understanding
Batch -Normalization (BN) is an algorithmic method which makes the training of Deep Neural Networks (DNN) faster and more stable. It consists of ......
Read more >
Batch Normalization: Accelerating Deep Network Training by ...
Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, ...
Read more >
Introduction to Batch Normalization - Analytics Vidhya
Batch normalization is the process to make neural networks faster and more stable through adding extra layers in a deep neural network.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found