Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

PatchCore: Unable to reproduce the pixel-wise reported performance

See original GitHub issue

Hi everybody,

I tried to use the PatchCore model on the MVTec dataset (bottle class), but the pixel-wise performance was well bellow the reported. I used the default config file (setting the seed to 42):

DATALOADER:0 TEST RESULTS
{'image_AUROC': 1.0,
 'image_F1': 1.0,
 'pixel_AUROC': 0.8852562308311462,
 'pixel_F1': 0.49169835448265076}

However, using the PaDiM model I was able to reproduce the reported benchmark performance:

DATALOADER:0 TEST RESULTS
{'image_AUROC': 0.9936507940292358,
 'image_F1': 0.9763779044151306,
 'pixel_AUROC': 0.9830061197280884,
 'pixel_F1': 0.7220250964164734}

Is this unexpected result due to a bug or am I missing something?

Issue Analytics

State:
Created a year ago
Reactions:1
Comments:5 (4 by maintainers)

Top GitHub Comments

2reactions

alexriedel1commented, Apr 9, 2022

https://github.com/openvinotoolkit/anomalib/commit/2dfa0a7a6b97ffd0557c562b4beae06ff755c1e7 This was when apply_tiling was set True on default for patchcore. It’s the reason why the pixel wise F1 drops signifacantly.

2reactions

samet-akcaycommented, Apr 8, 2022

Hi @mvidela31, this must be related to one of our recent PRs. We’ll investigate this.

Top Results From Across the Web

Towards Total Recall in Industrial Anomaly Detection

PatchCore offers competitive inference times while achieving state-of-the-art performance for both detection and localization.

Towards Total Recall in Industrial Anomaly Detection

PatchCore offers competitive inference times while achieving state-of-the-art performance for both detection and localization.

amazon-science/patchcore-inspection

This repository contains the implementation for PatchCore as proposed in Roth et al. (2021), https://arxiv.org/abs/2106.08265.

Registration based Few-Shot Anomaly Detection

07/15/22 - This paper considers few-shot anomaly detection (FSAD), a practical yet under-studied setting for anomaly detection (AD), ...

[Reproducibility Report] Explainable Deep One-Class ...

This study verifies comparable performance of FCDD on fashion-MNIST and CIFAR-10 and outperforms existing methods on MVTec-AD on the pixel-wise task. However, ...