question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

PAA training error

See original GitHub issue

Thanks for your error report and we appreciate it a lot.

Checklist

  1. I have searched related issues but cannot get the expected help.
  2. I have read the FAQ documentation but cannot get the expected help.
  3. The bug has not been fixed in the latest version.

Describe the bug PAA detector training breaks.

Reproduction

  1. What command or script did you run?
python tools/train.py configs/paa/paa_r50_fpn_1x_coco.py
  1. Did you make any modifications on the code or config? Did you understand what you have modified? No
  2. What dataset did you use? COCO

Environment

  1. Please run python mmdet/utils/collect_env.py to collect necessary environment information and paste it here.
  2. You may add addition that may be helpful for locating the problem, such as
    • How you installed PyTorch [e.g., pip, conda, source]
    • Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)

Error traceback If applicable, paste the error trackback here.

Traceback (most recent call last):
  File "/home/user/mmdetection/tools/train.py", line 242, in <module>
    main()
  File "/home/user/mmdetection/tools/train.py", line 231, in main
    train_detector(
  File "/home/user/mmdetection/mmdet/apis/train.py", line 244, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/user/mmcv/mmcv/runner/epoch_based_runner.py", line 127, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/user/mmcv/mmcv/runner/epoch_based_runner.py", line 50, in train
    self.run_iter(data_batch, train_mode=True, **kwargs)
  File "/home/user/mmcv/mmcv/runner/epoch_based_runner.py", line 29, in run_iter
    outputs = self.model.train_step(data_batch, self.optimizer,
  File "/home/user/mmcv/mmcv/parallel/data_parallel.py", line 75, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/user/mmdetection/mmdet/models/detectors/base.py", line 248, in train_step
    losses = self(**data)
  File "/home/user/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/user/mmcv/mmcv/runner/fp16_utils.py", line 110, in new_func
    return old_func(*args, **kwargs)
  File "/home/user/mmdetection/mmdet/models/detectors/base.py", line 172, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/home/user/mmdetection/mmdet/models/detectors/single_stage.py", line 83, in forward_train
    losses = self.bbox_head.forward_train(x, img_metas, gt_bboxes,
  File "/home/user/mmdetection/mmdet/models/dense_heads/base_dense_head.py", line 335, in forward_train
    losses = self.loss(*loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore)
  File "/home/user/mmcv/mmcv/runner/fp16_utils.py", line 198, in new_func
    return old_func(*args, **kwargs)
  File "/home/user/mmdetection/mmdet/models/dense_heads/paa_head.py", line 152, in loss
    reassign_bbox_weights, num_pos = multi_apply(
  File "/home/user/mmdetection/mmdet/core/utils/misc.py", line 30, in multi_apply
    return tuple(map(list, zip(*map_results)))
  File "/home/user/mmdetection/mmdet/models/dense_heads/paa_head.py", line 349, in paa_reassign
    gmm.fit(pos_loss_gmm)
  File "/home/user/anaconda3/lib/python3.9/site-packages/sklearn/mixture/_base.py", line 193, in fit
    self.fit_predict(X, y)
  File "/home/user/anaconda3/lib/python3.9/site-packages/sklearn/mixture/_base.py", line 246, in fit_predict
    self._m_step(X, log_resp)
  File "/home/user/anaconda3/lib/python3.9/site-packages/sklearn/mixture/_gaussian_mixture.py", line 691, in _m_step
    self.precisions_cholesky_ = _compute_precision_cholesky(
  File "/home/user/anaconda3/lib/python3.9/site-packages/sklearn/mixture/_gaussian_mixture.py", line 333, in _compute_precision_cholesky
    raise ValueError(estimate_precision_error_message)
ValueError: Fitting the mixture model failed because some components have ill-defined empirical covariance (for instance caused by singleton or collapsed samples). Try to decrease the number of components, or increase reg_covar.

Bug fix If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:8 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
FDInSkycommented, Sep 1, 2022
0reactions
github-actions[bot]commented, Sep 27, 2022

This issue is closed because it has been stale for 5 days. Please open a new issue if you have similar issues or you have any new updates now.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Training Error - an overview | ScienceDirect Topics
In this strategy, the best parameter setting yielding lowest validation error is reserved and when training is terminated (with lowest training error), the ......
Read more >
What is a training and test error? - Quora
Training error is the error that you get when you run the trained model back on the training data. Remember that this data...
Read more >
paa.iis.module.dll error - Ping Identity Support Portal
Hi Team,. After successful Ping Agent installation on IIS 2012 webs erver while try to attempting APP URL getting below error in Event ......
Read more >
DCIPS PAA Demonstration Training
Note: If the certification box is not checked, user will receive the following error message: Confirmation message is received. Select the 'Yes' ...
Read more >
3. Training error vs Test error - YouTube
3. Training error vs Test error. 8.9K views 2 years ago Machine Learning. Model assessment and model selection.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found