Issue in Patch-Core

Background

I have tried to run anomalib on the kaggle environment (also preparing a tutorial to contribute here for newcomers to get started with anomalib 😉). However, with the patch-core model, I’m facing an issue.

Describe the Behaviour

When I run the script with patch-core, it kinda stuck in the following stage. And I’ve noticed that the coreset is processing on CPU (usages were high). So, it might be stuck or take lots of time to process on the CPU, which should not be.

Epoch 0:   8%|█▉           | 7/90 [01:20<15:58, 11.54s/it, loss=nan]
Validation: 0it [00:00, ?it/s]A
Creating CoreSet Sampler via k-Center Greedy
Getting the coreset from the main embedding.

Reproducible Code

• Kaggle Notebook. (make sure to enable GPU quote)

Other info.

Not sure if it’s the current know issue with patchcore. I’ve run the code successfully with the padim model (didn’t test the rest of the models).

I’m considering myself a beginner in pytorch/pytorch-lighting, maybe I’m missing something obvious. Also, there are lots of pytorch-lighting related UserWarning messages. IMHO, it’s annoying, can it be removed, like verbose = 0?

[daMichaelB]

Meanwhile the code from fix/sa/configs is merged. I have a slightly different setup and still does not work.

First I trained PatchCore on a single-GPU machine. There i always ran into Cuda-OutOfMemory as my dataset was to big for the patch-core training step. So i switched to a 8-GPU instance. I changed:

trainer:
  accelerator: dp # parallel training
  gpus: 8 # 8 gpus used

if i use the settings from fix/sa/configs it only uses one gpu and goes into OOO in training step.

Training Step 🆗

Now the training_step works very well with all the data. I also found that the GPU-Mem is not used anymore at all (maybe because of dp setting.

Validation Step

The validation Step however does not use any of the GPUs and it takes forever.

I get the log Applying core-set subsampling to get the embedding. and then it computes forever on CPU.

Is there any idea how to get this working @samet-akcay ? Thank you very much in advance!

[innat]

@samet-akcay running from fix/sa/configs branch fix the issue. Thanks. 😄

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│        image_AUROC        │            1.0            │
│         image_F1          │            1.0            │
│        pixel_AUROC        │     0.984413743019104     │
│         pixel_F1          │    0.7208729982376099     │
└───────────────────────────┴───────────────────────────┘