[Feature Request] 3D support for PanopticQualityMetric + provide option to return confusion matrix

** Edit ** Feature request:

• 3D support for PanopticQualityMetric
• provide option to return confusion matrix

Describe the bug Not sure whether one should report bugs for weekly versions?

I am playing around with PanopticQualityMetric.

The metric_name is ignored, I always get the same output, no matter if I use:

    metric_name=("pq")
    metric_name=("sq")
    metric_name=("rq")
    metric_name=("pq", "sq", "rq")

Also there is no option to turn on the confusion matrix output already, correct?

When I turn it on in the code I get a tensor like:

score1: tensor([153.0000,  24.0000,  21.0000, 139.7191])

If I am not mistaken these values represent the following?

tp, fp, fn, iou_sum = f[..., 0], f[..., 1], f[..., 2], f[..., 3]

feature request: I believe there should also be an option to directly compute mean iou as it is done in the manuscript.

To Reproduce Compute PanopticQualityMetric with different metric_name parameters. Environment

python -c 'import monai; monai.config.print_debug_info()'

python -c ‘import monai; monai.config.print_debug_info()’

================================ Printing MONAI config…

MONAI version: 1.1.dev2250 Numpy version: 1.23.5 Pytorch version: 1.13.0+cu117 MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False MONAI rev id: 1d25ea145532319feb57647797ed3a8c2e7e9eb4 MONAI file: /home/florian/miniconda3/envs/emcaps/lib/python3.10/site-packages/monai/init.py

Optional dependencies: Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION. Nibabel version: 4.0.2 scikit-image version: 0.19.3 Pillow version: 9.3.0 Tensorboard version: 2.11.0 gdown version: 4.5.4 TorchVision version: NOT INSTALLED or UNKNOWN VERSION. tqdm version: 4.64.1 lmdb version: NOT INSTALLED or UNKNOWN VERSION. psutil version: 5.9.4 pandas version: 1.5.1 einops version: NOT INSTALLED or UNKNOWN VERSION. transformers version: NOT INSTALLED or UNKNOWN VERSION. mlflow version: NOT INSTALLED or UNKNOWN VERSION. pynrrd version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit: https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================ Printing system config…

System: Linux Linux version: Ubuntu 20.04.5 LTS Platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.31 Processor: x86_64 Machine: x86_64 Python version: 3.10.8 Process name: python Command: [‘python’, ‘-c’, ‘import monai; monai.config.print_debug_info()’] Open files: [popenfile(path=‘/home/florian/.vscode-server/data/logs/20221211T175743/remoteagent.log’, fd=19, position=20132, mode=‘a’, flags=33793), popenfile(path=‘/home/florian/.vscode-server/data/logs/20221211T175743/ptyhost.log’, fd=20, position=4094, mode=‘a’, flags=33793), popenfile(path=‘/home/florian/.vscode-server/bin/5235c6bb189b60b01b1f49062f4ffa42384f8c91/vscode-remote-lock.florian.5235c6bb189b60b01b1f49062f4ffa42384f8c91’, fd=99, position=0, mode=‘w’, flags=32769)] Num physical CPUs: 16 Num logical CPUs: 32 Num usable CPUs: 32 CPU usage (%): [32.1, 20.7, 16.2, 17.0, 45.5, 30.8, 36.7, 36.4, 40.6, 17.9, 40.6, 23.4, 18.9, 15.2, 16.0, 86.0, 16.8, 30.8, 18.9, 18.1, 22.6, 34.3, 32.1, 34.0, 56.5, 35.5, 15.2, 27.4, 18.5, 16.8, 16.0, 30.8] CPU freq. (MHz): 2008 Load avg. in last 1, 5, 15 mins (%): [11.3, 14.0, 14.8] Disk usage (%): 35.1 Avg. sensor temp. (Celsius): UNKNOWN for given OS Total physical memory (GB): 125.7 Available memory (GB): 114.8 Used memory (GB): 9.5

================================ Printing GPU config…

Num GPUs: 2 Has CUDA: True CUDA version: 11.7 cuDNN enabled: True cuDNN version: 8500 Current device: 0 Library compiled for CUDA architectures: [‘sm_37’, ‘sm_50’, ‘sm_60’, ‘sm_70’, ‘sm_75’, ‘sm_80’, ‘sm_86’] GPU 0 Name: NVIDIA RTX A5000 GPU 0 Is integrated: False GPU 0 Is multi GPU board: False GPU 0 Multi processor count: 64 GPU 0 Total memory (GB): 23.7 GPU 0 CUDA capability (maj.min): 8.6 GPU 1 Name: Quadro RTX 8000 GPU 1 Is integrated: False GPU 1 Is multi GPU board: False GPU 1 Multi processor count: 72 GPU 1 Total memory (GB): 47.5 GPU 1 CUDA capability (maj.min): 7.5

[yiheng-wang-nv]

do you know why there’s a constraint for 2d here @yiheng-wang-nv?

https://github.com/Project-MONAI/MONAI/blob/5200cc6e687910118198933b2b6de4ecf97bb881/monai/metrics/panoptic_quality.py#L93-L100

looking at the rest of the logic I don’t find anything requiring 2D

Hi @wyli , the initial purpose to implement this metric is for the HoverNet pipeline. Considering that the reference repo uses only 2d logic, I decided to follow their logic. I think we can enhance it for 3d if needed after 1.1 release.

[yiheng-wang-nv]

Hi @neuronflow , could you please re-edit this ticket to make it more clear? I think it is a feature request of making PanopticQualityMetric support 3d, rather than a bug. Thanks