Engine - Continue run throws error because epoch_length is None

🐛 Bug description

I am using an engine such that it runs a previously undefined number of iterations with an infinite iterator. Epoch will be terminated in the process function with engine.terminate().

I am always running the engine for only one epoch, because I need to do some calculations outside the engines framework and then continue the engine to run one further epoch with engine.run(itertools.counter(start=0), max_epochs=engine.state.epoch + 1).

I get an error in engine._is_done because engine.state.epoch_length is None. Is this the intended behavior? Do I have to set epoch_length explicitly?

Here is a minimum example to reproduce the error:

import itertools

from ignite.engine import Engine


def _run_iteration(engine, counter):
    if counter > 2:
        # same behavior observed for engine.terminate_epoch()
        engine.terminate()


if __name__ == '__main__':
    engine = Engine(_run_iteration)
    # run one epoch
    engine.run(itertools.count(start=0), max_epochs=engine.state.epoch + 1)
    # epoch length is None after run:
    print(engine.state.epoch_length)
    # running another epoch throws the error:
    engine.run(itertools.count(start=0), max_epochs=engine.state.epoch + 1)

Thanks in advance!

Environment

• PyTorch Version (e.g., 1.4): 1.5.1
• Ignite Version (e.g., 0.3.0): 0.4.2
• OS (e.g., Linux): Windows
• How you installed Ignite (conda, pip, source): pip
• Python version: 3.7.7
• Any other relevant information: -

[vfdev-5]

@fco-dv could you please take a look at this issue and propose a way to fix it along the way with https://github.com/pytorch/ignite/pull/1333 which actually shouldn’t introduce state.restart as it was discussed in the corresponding issue.

[vfdev-5]

@alxlampe thanks for the details !

Yes, that’s true that engine.state.max_epochs = None is to restart the Engine from the begining and not continue. Currently, if we would like to continue, epoch_length is sort of implicitly required. It is a bug as in your case user can stop the execution during the first epoch of a run with epoch_length=None and epoch_length is not set… Let me think what can be done for that. Maybe, a solution is to set unknown epoch_length on engine.terminate_epoch() or engine.terminate()…

I was already thinking about creating a new issue to ask if it is possible to separate the event handler mechanics from the engine and build the engine on top it. That would allow to create fully customized engines, e.g. engines that only require an iteration loop but no epoch loop. But I still have to dive deeper into the code to check if this makes any sense and if so, would create an issue with an initial proposal 😃

Yes, this is something to explore

In the optimal case, Engine would provide me some interface to split the run method into subcommands …

A refactor like that may also make sense… Let’s see what is more interesting: this point or above one.