Test anomalib with unsupervised dataset
See original GitHub issueIβm trying to adapt the current pipeline for some ad hoc dataset (ie EuroSAT) in order to test anomalib and make it more robust. For using the FolderDataset
for unsupervised segmentation task.
So Iβm using the EuroSAT RGB Dataset (https://madm.dfki.de/files/sentinel/EuroSAT.zip), Residential as normal and Industrial as abnormal, and of course no mask/groundtruth.
As everything is fine now on development branch with torchmetrics, I will use 24ccb585319dd10ef81778364350eb972bf1cf41.
Updating config with EuroSAT
Here is my dataset section Iβm adding using to cflow
, ganomaly
, padim
and patchcore
(others settings are not modified)
dataset: name: eurosat format: folder path: ./datasets/EuroSAT normal_dir: Residential abnormal_dir: Industrial task: segmentation normal_test_dir: null mask: null extensions: .jpg split_ratio: 0.2 seed: 0 image_size: 64 train_batch_size: 16 test_batch_size: 16 num_workers: 4 transform_config: train: null val: null create_validation_set: false tiling: apply: false tile_size: null stride: null remove_border_count: 0 use_random_tiling: False random_tile_count: 16 inference_batch_size: 16 fiber_batch_size: 32 #for padim
Cflow: metric pixel_AUROC
not available
Epoch 0: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 344/344 [04:10<00:00, 1.38it/s]Traceback (most recent call last): File "tools/train.py", line 83, in <module> train() File "tools/train.py", line 73, in train trainer.fit(model=model, datamodule=datamodule) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit self._call_and_handle_interrupt( File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt return trainer_fn(*args, **kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl results = self._run(model, ckpt_path=self.ckpt_path) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run results = self._run_stage() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage return self._run_train() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1351, in _run_train self.fit_loop.run() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 205, in run self.on_advance_end() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 297, in on_advance_end self.trainer._call_callback_hooks("on_train_epoch_end") File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1634, in _call_callback_hooks fn(self, self.lightning_module, *args, **kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/callbacks/early_stopping.py", line 179, in on_train_epoch_end self._run_early_stopping_check(trainer) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/callbacks/early_stopping.py", line 190, in _run_early_stopping_check if trainer.fast_dev_run or not self._validate_condition_metric( # disable early_stopping with fast_dev_run File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/callbacks/early_stopping.py", line 145, in _validate_condition_metric raise RuntimeError(error_msg) RuntimeError: Early stopping conditioned on metric `pixel_AUROC` which is not available. Pass in or modify your `EarlyStopping` callback to use any of the following: `image_F1Score`, `image_AUROC`
Change early_stopping > metric: pixel_AUROC
to metric: image_AUROC
. βοΈ
Padim: KeyError: 'mask'
Epoch 0: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 375/375 [00:14<00:00, 25.17it/s, loss=nan] Training took 14.995270490646362 seconds LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] Testing DataLoader 0: 0%| | 0/225 [00:00<?, ?it/s]Traceback (most recent call last): File "tools/train.py", line 83, in <module> train() File "tools/train.py", line 79, in train trainer.test(model=model, datamodule=datamodule) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 936, in test return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt return trainer_fn(*args, **kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 983, in _test_impl results = self._run(model, ckpt_path=self.ckpt_path) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run results = self._run_stage() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1318, in _run_stage return self._run_evaluate() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1363, in _run_evaluate eval_loop_results = self._evaluation_loop.run() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, **kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 153, in advance dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, **kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 133, in advance self._on_evaluation_batch_end(output, **kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 263, in _on_evaluation_batch_end self.trainer._call_callback_hooks(hook_name, output, *kwargs.values()) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1634, in _call_callback_hooks fn(self, self.lightning_module, *args, **kwargs) File "/home/gsd/Desktop/julien-blanchon/trash/anomalib/anomalib/utils/callbacks/visualizer_callback.py", line 141, in on_test_batch_end true_mask = outputs["mask"][i].cpu().numpy() * 255 KeyError: 'mask' Testing DataLoader 0: 0%| | 0/225 [00:00<?, ?it/s]
I get an KeyError: 'mask'
after the first epoch, as ground truth is not provide the mask visualization step of on_batch_end must be disable. https://github.com/openvinotoolkit/anomalib/blob/24ccb585319dd10ef81778364350eb972bf1cf41/anomalib/utils/callbacks/visualizer_callback.py#L140
After a small fix (ie skip ground truth visualization) everything work fine. βοΈ
#TODO: Make true_mask visualisation optional as ground thruth is not always provide
if self.task == "segmentation" and "mask" in outputs:
true_mask = outputs["mask"][i].cpu().numpy() * 255
visualizer.add_image(image=true_mask, color_map="gray", title="Ground Truth")
Patchcore
Traceback (most recent call last):βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 194/194 [01:23<00:00, 2.32it/s] File "tools/train.py", line 83, in <module> train() File "tools/train.py", line 73, in train trainer.fit(model=model, datamodule=datamodule) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit self._call_and_handle_interrupt( File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt return trainer_fn(*args, **kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl results = self._run(model, ckpt_path=self.ckpt_path) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run results = self._run_stage() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage return self._run_train() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1351, in _run_train self.fit_loop.run() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, **kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 269, in advance self._outputs = self.epoch_loop.run(self._data_fetcher) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 205, in run self.on_advance_end() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 255, in on_advance_end self._run_validation() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 309, in _run_validation self.val_loop.run() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 211, in run output = self.on_run_end() File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 187, in on_run_end self._evaluation_epoch_end(self._outputs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 309, in _evaluation_epoch_end self.trainer._call_lightning_module_hook("validation_epoch_end", output_or_outputs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1593, in _call_lightning_module_hook output = fn(*args, **kwargs) File "/home/gsd/Desktop/julien-blanchon/trash/anomalib/anomalib/models/components/base/anomaly_module.py", line 133, in validation_epoch_end self._collect_outputs(self.image_metrics, self.pixel_metrics, outputs) File "/home/gsd/Desktop/julien-blanchon/trash/anomalib/anomalib/models/components/base/anomaly_module.py", line 159, in _collect_outputs image_metric.update(output["pred_scores"], output["label"].int()) File "/home/gsd/Desktop/julien-blanchon/trash/anomalib/anomalib/utils/metrics/collection.py", line 37, in update super().update(*args, **kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/collections.py", line 153, in update m.update(*args, **m_kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/metric.py", line 312, in wrapped_func update(*args, **kwargs) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/classification/stat_scores.py", line 185, in update tp, fp, tn, fn = _stat_scores_update( File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/functional/classification/stat_scores.py", line 154, in _stat_scores_update preds, target, _ = _input_format_classification( File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/utilities/checks.py", line 407, in _input_format_classification case = _check_classification_inputs( File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/utilities/checks.py", line 268, in _check_classification_inputs case, implied_classes = _check_shape_and_type_consistency(preds, target) File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/utilities/checks.py", line 114, in _check_shape_and_type_consistency raise ValueError( ValueError: Either `preds` and `target` both should have the (same) shape (N, ...), or `target` should be (N, ...) and `preds` should be (N, C, ...). Epoch 0: 100%|ββββββββββ| 344/344 [01:44<00:00, 3.29it/s, loss=nan]
Shape error π¦, any idea ? β
Issue Analytics
- State:
- Created a year ago
- Comments:10 (10 by maintainers)
Top GitHub Comments
Great! Weβll try to simplify the parameters a further to make it more intuitive.
Exactly, as you said, task ensures the right performance evaluation and saving the images.
In addition to this, tasks saves the images accordingly
You could refer to this link to see which models support both classification and segmentation (The list has the terms detection vs segmentation, but the same idea).
Does it make any more sense?