Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ValueError: generator already executing

See original GitHub issue

I have received the following error message when it finished the second epoch of command python train.py

6515/6516 [============================>.] - ETA: 0s - loss: 0.7473 - Atelectasis_loss: 0.1070 - Cardiomegaly_loss: 0.0215 - Effusion_loss: 0.0999 - Infiltration_loss: 0.1867 - Mass_loss: 0.0552 - Nodule_loss: 0.0686 - Pneumonia_loss: 0.0160 - Pneumothorax_loss: 0.0484 - Consolidation_loss: 0.0452 - Edema_loss: 0.0185 - Emphysema_loss: 0.0246 - Fibrosis_loss: 0.0183 - Pleural_Thickening_loss: 0.0354 - Hernia_loss: 0.0021

*********************************
current learning rate: 0.0010000000474974513
Traceback (most recent call last):
  File "train.py", line 232, in <module>
    main()
  File "train.py", line 215, in main
    class_weight=class_weights,
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 2213, in fit_generator
    callbacks.on_epoch_end(epoch, epoch_logs)
  File "/home/paperspace/anaconda3/lib/python3.6/site-packages/keras/callbacks.py", line 76, in on_epoch_end
    callback.on_epoch_end(epoch, logs)
  File "/home/paperspace/CheXNet-Keras/callback.py", line 76, in on_epoch_end
    x, y = load_generator_data(self.generator, self.steps, len(self.class_names))
  File "/home/paperspace/CheXNet-Keras/callback.py", line 22, in load_generator_data
    batch_x, batch_y = next(generator)
ValueError: generator already executing

My workaround solution is saving weights.h5 and best_weights.h5 after each completed epoch during training. So, I can observe the increasing number of the mean auroc after epochs.

Let me guess, Is it something wrong with the callback.py file about thread safe?

Note: my machine is Quadro P6000 Cloud 24GB GPU on Paperspace

Issue Analytics

State:
Created 6 years ago
Comments:8 (2 by maintainers)

Top GitHub Comments

2reactions

brucechou1983commented, Mar 5, 2018

@sakares @sriniwaas @aguang1201 This is fixed. Please update to 0.3.0.

2reactions

brucechou1983commented, Feb 27, 2018

Hey guys,

Sorry for the inconvenient caused. I’m actually working on a new version of customized generator which is based on the thread safe keras.utils.Sequence. Hopefully I will release it after finishing all tests in this week.

Top Results From Across the Web

python - builtins.ValueError: generator already executing

are giving me a "builtins.ValueError: generator already executing" Does anyone know how to fix this? Also if there is a more pythonic way...

Python - generator already executing - Neo4j - 40421

ValueError : generator already executing is raised when multiple threads try to consume from one generator. Note that the driver object is thread ......

Value error generator already executing · Issue #14 - GitHub

ValueError : generator already executing. This is the code. def video_generator(topic): """Video streaming generator function.

10293 - repo sync says "ValueError: generator already ...

Issue 10293: repo sync says "ValueError: generator already executing" when running in parallel. Reported by schedule kcwu@chromium.org kcwu@chromium.org.

"generator already executing" when operating on a stream

I am writing a custom ner.make-gold recipe. I am replacing the stream = split_sentences(stream) call with my own segment_documents() ...