Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Global Step is not updating?

See original GitHub issue

As seen in https://github.com/Zehaos/MobileNet/issues/27 – the global step does not update after each training step has been taken. Is there a fix to this coming up soon? I have tried both the older version of yellowfin.py in that issue and also the latest one available. In both instances, the global step doesn’t update.

I believe the issue comes from the global variable existing only within the optimizer but not globally. As a quick fix, I moved the definition of the global step (at https://github.com/JianGoForIt/YellowFin/blob/master/tuner_utils/yellowfin.py#L60) out of the optimizer and directly in the graph, before feeding in this variable back to the optimizer.

Is there a cleaner solution to this?

Issue Analytics

State:
Created 6 years ago
Comments:7 (2 by maintainers)

Top GitHub Comments

1reaction

JianGoForItcommented, Aug 11, 2017

@kwotsin @ReDeiPirati , we have updated the code and the global_step issue is gone now. Please check it out.

0reactions

ReDeiPiraticommented, Jul 19, 2017

@kwotsin PR done!

Top Results From Across the Web

global step is not updating if the automatic optimisation is not ...

I turned off the automatic optimisation, because I am using SAM optimizer (https://github.com/davda54/sam). After that, the global_step of the trainer is not ......

python - TensorFlow 'global_step' variable not getting updated ...

I have a network where I am using exponential decay of the learning rate. For this I'm keeping track of a 'global_step' TF...

tf.compat.v1.train.create_global_step | TensorFlow v2.11.0

Create global step tensor in graph. ... With the deprecation of global graphs, TF no longer tracks variables in collections.

How Do I Enable Loss Scaling? - Huawei support

Updating the Global Step ... After the loss scaling function is enabled, the step where the loss scaling overflow occurs needs to be...

昇腾TensorFlow（20.1）-Loss Scaling:Updating the Global Step

Updating the Global Step. After the loss scaling function is enabled, the step where the loss scaling overflow occurs needs to be discarded....