load_state_dict doesn't catch 'StopIteration' exception
See original GitHub issueHi everyone,
I am using icefall to train a RNNT model and after I start from a batch and load the sampler sate, the dynamicbucketsampler raise an StopIteration which resulted the program to abort. I think that “exception” should have been caught somewhere up the stack trace and processed.
-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/home/azureuser/anaconda3/envs/icefall-torch1.10cu102/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/home/azureuser/codebase/icefall-birch/egs/gigaspeech/ASR/pruned_transducer_stateless2/train.py", line 869, in run
train_dl = gigaspeech.train_dataloaders(
File "/home/azureuser/codebase/icefall-birch/egs/gigaspeech/ASR/pruned_transducer_stateless2/asr_datamodule.py", line 320, in train_dataloaders
train_sampler.load_state_dict(sampler_state_dict)
File "/home/azureuser/codebase/lhotse-birch/lhotse/dataset/sampling/dynamic_bucketing.py", line 174, in load_state_dict
self._fast_forward()
File "/home/azureuser/codebase/lhotse-birch/lhotse/dataset/sampling/dynamic_bucketing.py", line 190, in _fast_forward
next(self)
File "/home/azureuser/codebase/lhotse-birch/lhotse/dataset/sampling/base.py", line 261, in __next__
batch = self._next_batch()
File "/home/azureuser/codebase/lhotse-birch/lhotse/dataset/sampling/dynamic_bucketing.py", line 233, in _next_batch
batch = next(self.cuts_iter)
StopIteration
Issue Analytics
- State:
- Created a year ago
- Comments:21
Top Results From Across the Web
Why can't my except block catch a StopIteration exception?
The StopIteration exception is not raised in the yield call, so you won't be catching it with that try/except setup.
Read more >How to catch StopIteration Exception in Python? - Tutorialspoint
When an iterator is done, it's next method raises StopIteration. This exception is not considered an error.
Read more >StopIteration Exception in Python - Pylenin
Handle StopIteration error in Python using the try-except block. ... You can catch such errors using the StopIteration Exception class.
Read more >Python 3 Tutorial: How To Handle StopIteration Exception
In this Python 3 Tutorial we look at how to handle the StopIteration Exception in Python3. Be sure to like, share and comment...
Read more >PEP 479 – Change StopIteration handling inside generators
This PEP proposes a change to generators: when StopIteration is raised ... a StopIteration exception into a generator, if the generator doesn't catch...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Yeah, that was solved, I am using it for our training. And all epoch now contains the same correct number of batches! Thank you and I think we can close it now
On Sun, Jul 24, 2022 at 10:16 Piotr Żelasko @.***> wrote:
I was away for some time — can you clarify if the PR resolved the issue? Then I could close it.