question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Stop backprop when all gradients are None

See original GitHub issue

We need to stop backprop if all gradient variables are None.

(chainer.Variable(numpy.array([1, 2], 'f')) * 1).backward()

It causes TypeError:

TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:22 (15 by maintainers)

github_iconTop GitHub Comments

1reaction
niboshicommented, Aug 14, 2019

In order to avoid that, there could be another option:

  1. Flag on a variable whether .grad is manually set. If no grad is set, raise an error. If None is set, just do nothing.

Variable already has ._grad_valid flag. Currently it can be False only in Parameter. We could set it to False by default for variables created as outputs of FunctionNode.apply().

1reaction
toslunarcommented, Nov 26, 2018

We need to stop backprop if all gradient variables are None.

Be careful with the meaning of “all”. Recall:

  1. v.backward() may gather gradients from variables other than v;
  2. a function node may have multiple outputs.

The condition “all the output nodes of a function node f are None” does not imply we can ignore f (i.e. continue the loop) in Variable.backward().

In #5709, a related topic is discussed.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Why tape.gradient returns all none in my Sequential model?
Your model is not being recorded by the tape. You have to put the computations into the context of the tape if you...
Read more >
Introduction to gradients and automatic differentiation
In this guide, you will explore ways to compute gradients with TensorFlow, especially in ... Conversely, to disable the default behavior of watching...
Read more >
None gradients with nn.Parameter - autograd - PyTorch Forums
The warning seem to indicate the issue: if nothing requires gradients, then nothing will be computed. You need to find where the Tensors...
Read more >
Autograd tutorial - Pytorch中文手册
The autograd package provides automatic differentiation for all operations on Tensors. It is a define-by-run framework, which means that your backprop is ...
Read more >
Understanding Autograd: 5 Pytorch tensor functions - Medium
So how does during backpropagation, PyTorch(or any other DL library for that ... The signature for backward is backward(gradient=None, ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found