Incomplete error

@de-vri-es Thank you so much for your quick response!

Incomplete error popped up when I ran train_csv.py on my custom data. Train_frcnn.py from yhenon has no problem with my data.

---

(1) I tried to run train_csv.py on my custom data with Keras 2.0.9. It would stop at different step of the program if I restarted again. It could stop at Epoch 1/20 156/873[…]. But it never finish a epoch.

This is the error:

=============
Epoch 1/20
 31/873 [>.............................] - ETA: 10:59 - loss: 1.1267 - regression_loss: 0.0020 - classification_loss: 1.1246Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keras/utils/data_utils.py", line 579, in data_generator_task
    generator_output = next(self._generator)
  File "build/bdist.linux-x86_64/egg/keras_retinanet/preprocessing/generator.py", line 217, in next
    return self.compute_input_output(group)
  File "build/bdist.linux-x86_64/egg/keras_retinanet/preprocessing/generator.py", line 195, in compute_input_output
    image_group, annotations_group = self.preprocess_group(image_group, annotations_group)
  File "build/bdist.linux-x86_64/egg/keras_retinanet/preprocessing/generator.py", line 115, in preprocess_group
    image, annotations = random_transform(image, annotations, self.image_data_generator)
  File "build/bdist.linux-x86_64/egg/keras_retinanet/utils/image.py", line 76, in random_transform
    boxes[index, 0] = float(min(j))
 32/873 [>.............................] - ETA: 10:45 - loss: 1.1265 - regression_loss: 0.0020 - classification_loss: 1.1245Traceback (most recent call last):
  File "train_csv.py", line 117, in <module>
    keras.callbacks.ReduceLROnPlateau(monitor='loss', factor=0.1, patience=2, verbose=1, mode='auto', epsilon=0.0001, cooldown=0, min_lr=0),
  File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 2046, in fit_generator
    generator_output = next(output_generator)
StopIteration

---

(2) @de-vri-es kindly repied my reported error

from @de-vri-es "@Weixing-Zhang: The error message you posted is incomplete. The actual exception is missing. That is probably because of the way keras handles exception in data generators (would be improved by fchollet/keras#8485) . I do not think this is the same issue though.

If this is a reproducible error, could you update Keras to latest master (so that you get fchollet/keras#8485) and open a new issue with the updated error message?"

Then I upgraded Keras to latest master from 2.0.9 as @de-vri-es suggested. Another error showed up: (p.s. the reason I downgraded keras before because ValueError: Error when checking target: expected regression to have shape (None, None, 4) but got array with shape (1, 102438, 5), it seems this error is fixed now).

=============
None
Epoch 1/20
391/873 [============>.................] - ETA: 3:48 - loss: 0.5860 - regression_loss: 0.0018 - classification_loss: 0.5841Traceback (most recent call last):
  File "train_csv.py", line 117, in <module>
    keras.callbacks.ReduceLROnPlateau(monitor='loss', factor=0.1, patience=2, verbose=1, mode='auto', epsilon=0.0001, cooldown=0, min_lr=0),
  File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 2115, in fit_generator
    generator_output = next(output_generator)
  File "/usr/local/lib/python2.7/dist-packages/keras/utils/data_utils.py", line 735, in get
    six.reraise(value.__class__, value, value.__traceback__)
  File "/usr/local/lib/python2.7/dist-packages/keras/utils/data_utils.py", line 635, in data_generator_task
    generator_output = next(self._generator)
  File "build/bdist.linux-x86_64/egg/keras_retinanet/preprocessing/generator.py", line 217, in next
  File "build/bdist.linux-x86_64/egg/keras_retinanet/preprocessing/generator.py", line 195, in compute_input_output
  File "build/bdist.linux-x86_64/egg/keras_retinanet/preprocessing/generator.py", line 115, in preprocess_group
  File "build/bdist.linux-x86_64/egg/keras_retinanet/utils/image.py", line 76, in random_transform
ValueError: min() arg is an empty sequence

Any help would be appreciated!!

[Weixing-Zhang]

@hgaiser You are right. Your code runs perfectly after I filter out those wrong annotations. I am really sorry for wasting your time. Please allow me have all of your in the acknowledgement section of my research article.

[hgaiser]

Good to hear it’s working, and thanks 😃