Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Eval gets stuck forever in the Trainer Component

See original GitHub issue

When I’m in the Trainer component, eval gets stuck forever:

[2019-05-13 17:40:53,499] {logging_mixin.py:95} INFO - [2019-05-13 17:40:53,499] {saver.py:1270} INFO - Restoring parameters from /home/benjamintan/workspace/darkrai/logs/shapes_768_1024_20190513T1740/model.ckpt-100
[2019-05-13 17:40:54,151] {logging_mixin.py:95} INFO - [2019-05-13 17:40:54,151] {session_manager.py:491} INFO - Running local_init_op.
[2019-05-13 17:40:54,198] {logging_mixin.py:95} INFO - [2019-05-13 17:40:54,198] {session_manager.py:493} INFO - Done running local_init_op.

Strangely, if I replace the path eval example path with the training example path, it manages to make progress to the model validator (though it fails model validation).

Any pointers on how to debug this?

Issue Analytics

State:
Created 4 years ago
Comments:9 (7 by maintainers)

Top GitHub Comments

1reaction

ruoyu90commented, May 16, 2019

@benjamintanweihao Thanks for reporting this! Actually this helps surfaced a bug in our trainer executor. The train_files and eval_files should have consistent type. Will send a PR to fix this soon.

0reactions

benjamintanweihaocommented, May 16, 2019

I think I found the problem! I parsed the filenames parameter from eval_input_fn(filenames, transform_output) wrongly. It turns out that filenames is a string and not a list?