Misleading warning when exiting iteration while loop

I get the following warning in each epoch

Data iterator can not provide data anymore but required total number of iterations to run is not reached.

Looking into it, I see that the exception handler on #L416 for StopIteration, upon reaching the end of the data loader, sets should_exit = True. Then, on the next attempt to iterate, it goes through the except block again and it checks if the entire run is done by running self._is_done. This function checks if all iterations have been completed, not per epoch but for all epochs. Therefore, unless we are in the final epoch, this warning is shown for each epoch.

Is this the intended behaviour or have I done something wrong? I am seeing this behaviour while using an iterator that must be manually reset to the first element. I do this reset in a handler that runs on EPOCH_COMPLETED so after this loop has been exited, making the warning seemingly unavoidable through the API.

[kilianbreathnach]

Yes, I ,manually call .reset() using a function defined to run on EPOCH_COMPLETED, as I mentioned in the original issue. I see now that if I were to use the auto_reset option, then the iterator would reset on #L436 and the loop would continue. However, the manual reset will not happen until the _run_once_on_dataset method has completed and the EPOCH_COMPLETED event is fired in the _internal_run method.

It turns out that my issue is that the iter_counter condition on #L457 is not being met. In the DaliGenericIterator, there are some options for how to fill or not fill the last batch, outlined here. The default option will lead to batches being processed at a rate that is actually quicker than the epoch_length. It seems the best option is probably to set the iterator with

fill_last_batch = False, last_batch_padded = True

but maybe you will figure out better options when you make a DALI example.

[vfdev-5]

Ok I see the problem. Thanks for the feedback!