Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

KeyError in Stash

See original GitHub issue

Thank you so much for the awesome repository!

I use stash and find there is an error in PoPCat().isolate(ns)

I have checked the stash is fine and have run successfully.

Thanks again for your help!

Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torchgpipe/skip/skippable.py", line 173, in forward
    poppable_tensors[name] = skip_tracker.load(batch, ns, name)
  File "/opt/conda/lib/python3.6/site-packages/torchgpipe/skip/tracker.py", line 38, in load
    return self.tensors.pop((ns, name))
KeyError: (<Namespace 'bfb2fb79-66d2-4437-abf9-ef62cfb17836'>, 'k')

  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
    input = module(input)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torchgpipe/skip/skippable.py", line 175, in forward
    raise RuntimeError(f"'{name}' has not been stashed")
RuntimeError: 'k' has not been stashed

Issue Analytics

State:
Created 4 years ago
Comments:8 (4 by maintainers)

Top GitHub Comments

1reaction

wentaozhucommented, Dec 11, 2019

Yes. I forgot the []. Thank you so much!

0reactions

subleecommented, Dec 13, 2019

@wentaozhu Thank you for making it reproducible. I got it. If you use stash and pop without GPipe, I highly recommend verify_skippables() to check skip connection errors before training your module. You will get an exception like this before training:

TypeError: one or more pairs of stash and pop do not match:

* '1' declared 'p' as poppable but it was not stashed
* '1' declared 's' as poppable but it was not stashed
* '1' declared 'i' as poppable but it was not stashed
* '1' declared 'k' as poppable but it was not stashed
* no module declared 'skip' as poppable but stashed

I didn’t find a point to improve. I close this issue.