question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Bug] #1252 seems causing `KeyError: 'data_time'` when evaluation interval is a multiple of logging interval

See original GitHub issue

By default, log_config.interval=50. When evaluation.interval={50,200}, there is an error KeyError: ‘data_time’ (error log pasted below). When evaluation.interval=120, the error does not occur at first several iterations (because the evaluation is quite long, I was unable to try testing when it reaches iteration 600, but I guess that it might trigger the error when it reaches there). Not sure if there is anything that I am not doing correctly. Thank you for your great work and effort in maintaining this!

  File "/home/haotian/code/mmcv/mmcv/runner/hooks/logger/text.py", line 96, in _log_info
    log_str += f'time: {log_dict["time"]:.3f}, ' \
KeyError: 'data_time'
Traceback (most recent call last):
  File "./tools/train.py", line 223, in <module>
    main()
  File "./tools/train.py", line 219, in main
    meta=meta)
  File "/home/haotian/code/mmdetection3d/mmdet3d/apis/train.py", line 25, in train_model
    meta=meta)
  File "/home/haotian/code/mmsegmentation/mmseg/apis/train.py", line 116, in train_segmentor
    runner.run(data_loaders, cfg.workflow)
  File "/home/haotian/code/mmcv/mmcv/runner/iter_based_runner.py", line 133, in run
    iter_runner(iter_loaders[i], **kwargs)
  File "/home/haotian/code/mmcv/mmcv/runner/iter_based_runner.py", line 66, in train
    self.call_hook('after_train_iter')
  File "/home/haotian/code/mmcv/mmcv/runner/base_runner.py", line 307, in call_hook
    getattr(hook, fn_name)(self)
  File "/home/haotian/code/mmcv/mmcv/runner/hooks/logger/base.py", line 152, in after_train_iter
    self.log(runner)
  File "/home/haotian/code/mmcv/mmcv/runner/hooks/logger/text.py", line 177, in log
    self._log_info(log_dict, runner)
  File "/home/haotian/code/mmcv/mmcv/runner/hooks/logger/text.py", line 96, in _log_info
    log_str += f'time: {log_dict["time"]:.3f}, ' \

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:9 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
zhouzaidacommented, Aug 12, 2021

hi @haotian-liu , you should also lower the priority of EvalHook in your codebase. Please refer to https://github.com/open-mmlab/mmsegmentation/pull/766

0reactions
haotian-liucommented, Aug 13, 2021

Thanks, I have opened the pull request: https://github.com/open-mmlab/mmdetection/pull/5882.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Azure Machine Learning Python SDK release notes
Fixed a bug that was causing forecasting generated code training jobs to fail. 2022-10-25. Azure Machine Learning SDK for Python v1.47.0.
Read more >
Release Notes — Airflow Documentation
The new logic generally orders by data interval, but a custom ordering can be applied by setting the DAG to use a custom...
Read more >
IBM Informix Messages and Corrections
For example, this error appears when a second user tries to alter a table ... Review the DATETIME and INTERVAL literals in this...
Read more >
Customization Guide - WeeWX
Structure. The time at the end of the archive interval in unix epoch time. This is the primary key in the database. It...
Read more >
Appendix C. Errors, Error Codes, and Common Problems
If after you have examined all other possibilities and you have concluded that the MySQL server or a MySQL client is causing the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found