[Bug] #1252 seems causing `KeyError: 'data_time'` when evaluation interval is a multiple of logging interval
See original GitHub issueBy default, log_config.interval=50
.
When evaluation.interval={50,200}
, there is an error KeyError: ‘data_time’ (error log pasted below).
When evaluation.interval=120
, the error does not occur at first several iterations (because the evaluation is quite long, I was unable to try testing when it reaches iteration 600, but I guess that it might trigger the error when it reaches there).
Not sure if there is anything that I am not doing correctly. Thank you for your great work and effort in maintaining this!
File "/home/haotian/code/mmcv/mmcv/runner/hooks/logger/text.py", line 96, in _log_info
log_str += f'time: {log_dict["time"]:.3f}, ' \
KeyError: 'data_time'
Traceback (most recent call last):
File "./tools/train.py", line 223, in <module>
main()
File "./tools/train.py", line 219, in main
meta=meta)
File "/home/haotian/code/mmdetection3d/mmdet3d/apis/train.py", line 25, in train_model
meta=meta)
File "/home/haotian/code/mmsegmentation/mmseg/apis/train.py", line 116, in train_segmentor
runner.run(data_loaders, cfg.workflow)
File "/home/haotian/code/mmcv/mmcv/runner/iter_based_runner.py", line 133, in run
iter_runner(iter_loaders[i], **kwargs)
File "/home/haotian/code/mmcv/mmcv/runner/iter_based_runner.py", line 66, in train
self.call_hook('after_train_iter')
File "/home/haotian/code/mmcv/mmcv/runner/base_runner.py", line 307, in call_hook
getattr(hook, fn_name)(self)
File "/home/haotian/code/mmcv/mmcv/runner/hooks/logger/base.py", line 152, in after_train_iter
self.log(runner)
File "/home/haotian/code/mmcv/mmcv/runner/hooks/logger/text.py", line 177, in log
self._log_info(log_dict, runner)
File "/home/haotian/code/mmcv/mmcv/runner/hooks/logger/text.py", line 96, in _log_info
log_str += f'time: {log_dict["time"]:.3f}, ' \
Issue Analytics
- State:
- Created 2 years ago
- Comments:9 (4 by maintainers)
Top Results From Across the Web
Azure Machine Learning Python SDK release notes
Fixed a bug that was causing forecasting generated code training jobs to fail. 2022-10-25. Azure Machine Learning SDK for Python v1.47.0.
Read more >Release Notes — Airflow Documentation
The new logic generally orders by data interval, but a custom ordering can be applied by setting the DAG to use a custom...
Read more >IBM Informix Messages and Corrections
For example, this error appears when a second user tries to alter a table ... Review the DATETIME and INTERVAL literals in this...
Read more >Customization Guide - WeeWX
Structure. The time at the end of the archive interval in unix epoch time. This is the primary key in the database. It...
Read more >Appendix C. Errors, Error Codes, and Common Problems
If after you have examined all other possibilities and you have concluded that the MySQL server or a MySQL client is causing the...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
hi @haotian-liu , you should also lower the priority of EvalHook in your codebase. Please refer to https://github.com/open-mmlab/mmsegmentation/pull/766
Thanks, I have opened the pull request: https://github.com/open-mmlab/mmdetection/pull/5882.