Issues with heatmap scaling when training without masks

I ran into an issue when training on a dataset (subset of MVTec), where I set all ground truth masks to zero (to simulate training on a dataset for which I have no ground truth masks). When training with the actual ground truth masks, the model produces heatmaps as expected as in the first image below (produced with tools/inference.py). However, when training with the zero masks, the heatmaps seem to be scaled differently as in the second image below. The confidence score seems unaffected.

This behaviour is the same for both PADIM and PatchCore. I haven’t tested the other models.

This is my model config for PADIM

dataset:
  name: mvtec_test
  format: folder
  path: ./datasets/mvtec_test/images
  normal: normal
  abnormal: abnormal
  task: segmentation
  mask: ./datasets/mvtec_test/masks_orig
  extensions: null
  seed: 0  
  image_size: 224
  train_batch_size: 32
  test_batch_size: 1
  num_workers: 16
  transform_config: null
  split_ratio: 0.2
  create_validation_set: true
  tiling:
    apply: false
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

model:
  name: padim
  backbone: resnet18
  layers:
    - layer1
    - layer2
    - layer3
  metric: auc
  normalization_method: min_max # options: [none, min_max, cdf]
  threshold:
    image_default: 3
    pixel_default: 3
    adaptive: true

project:
  seed: 42
  path: ./results
  log_images_to: ["local"]
  logger: false
  save_to_csv: false

optimization:
  openvino:
    apply: false

# PL Trainer Args. Don't add extra parameter here.
trainer:
  accelerator: null
  accumulate_grad_batches: 1
  amp_backend: native
  auto_lr_find: false
  auto_scale_batch_size: false
  auto_select_gpus: false
  benchmark: false
  check_val_every_n_epoch: 1 # Don't validate before extracting features.
  checkpoint_callback: true
  default_root_dir: null
  deterministic: false
  fast_dev_run: false
  gpus: 1
  gradient_clip_val: 0
  limit_predict_batches: 1.0
  limit_test_batches: 1.0
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  log_every_n_steps: 50
  log_gpu_memory: null
  max_epochs: 1
  max_steps: -1
  min_epochs: null
  min_steps: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle
  num_nodes: 1
  num_processes: 1
  num_sanity_val_steps: 0
  overfit_batches: 0.0
  plugins: null
  precision: 32
  prepare_data_per_node: true
  process_position: 0
  profiler: null
  progress_bar_refresh_rate: null
  replace_sampler_ddp: true
  stochastic_weight_avg: false
  sync_batchnorm: false
  terminate_on_nan: false
  tpu_cores: null
  track_grad_norm: -1
  val_check_interval: 1.0 # Don't validate before extracting features.
  weights_save_path: null
  weights_summary: top

[samet-akcay]

If I don’t have the ground truth, how should I change the config for the custom dataset? Also, any tips about the best practices to get optimal results?

@innat, if you don’t have the mask annotations, you could set the mask path to null. Something like the following:

dataset:
  name: mvtec #options: [mvtec, btech, folder]
  format: folder # mvtec
  path: C:/Users/..../bottle/test # ./datasets/MVTec
  normal: good
  abnormal: defected_type_1 
  split_ratio: 0.2 
  seed: 0
->task: classification
->mask: null
  extensions: null
  # category: bottle
  image_size: 224
  train_batch_size: 32
  test_batch_size: 1
  num_workers: 1 # 36
  transform_config: null
  create_validation_set: false
  tiling:
    apply: false # true
    tile_size: null # 64
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

Since there is no mask annotations, the task would become classification. It’s currently set manually, but we’ll be automatically adjusting this

[samet-akcay]

Hi @LukasBommes and @innat, it’s because you guys set the adaptive threshold to True. It calculates the threshold based on the best f1 score based on the prediction and annotations. When you provide zero masks, the model assumes that this is a segmentation task and computes the threshold based on the zero-masks and get confused. That’s why you see wrong heatmap as the output.

If you want to use a custom dataset you should use folder dataset. Here is sample config how to use it