Discussion on prediction filenames

Issue description

This issue aims to open a design discussion about the naming of the prediction files from ivadomed and was brought up in ADS meeting while working on integrating ivadomed’s model to ADS.

It concerns the naming of the prediction files in ivadomed with:

• the command --segment (run_segment_command in main.py)
• the pred_to_png function (used to convert NiFTI prediction files to PNG for microscopy)

Current behavior

Currently, the output file names for the predictions are: <original_image_filename>_<target_suffix>_pred.nii.gz

For example, with target_suffix = ["seg-manual", "_csfseg-manual"], the predictions are named: sub-01_T2w_seg-manual_pred.nii.gz sub-01_T2w_csfseg-manual_pred.nii.gz

This naming is odd as the target suffix often contains the word “manual” as per our naming convention for derivatives, and can be seen as ambiguous (i.e. is it a manual label or an automatic prediction from a model?).

Expected behavior

I don’t have a specific “expected” behavior at the moment, hence the design discussion card. As far as I know, the target_suffix is the only part of the config file that identifies the structures segmented in each class of the model.

Options:

1. Get rid of the “manual” word in the output predictions (ex: sub-01_T2w_seg_pred.nii.gz and ex: sub-01_T2w_csfseg_pred.nii.gz)
2. Name the predictions with a class number instead of the name of the structure (ex: sub-01_T2w_class-0_pred.nii.gz)
3. Others? (please, feel free to add suggestions here)

In ADS and SCT (via the segment_volume API):

• In ADS, we chose to follow the ivadomed convention for the moment for consistency (tagging @mathieuboudreau)
• As far as I understand the SCT code, they seem to do the same in sct_deepseg here (tagging @joshuacwnewton)

[mariehbourget]

We might have different and more complex "target_suffix" scenarios in the future of ivadomed, and a new field "prediction_suffix" in the config file would also solve these. To make sure old ivadomed configs stay compatible, we can make this field optional s.t.

• if it is not filled, then "prediction_suffix" will just be equivalent to "target_suffix" which will reserve the old behavior
• if it is filled, we will use it to generate the prediction filenames

Hi @uzaymacar and @jcohenadad, I’m not sure about adding another prediction_suffix field in the config file.

As suggested by Uzay, it would work for our needs and covers potential retrocompatibility issues. However, I think it opens the door to errors/misunderstanding from the users, either not filling the field and defaulting to the old behavior, or not updating the field to correspond to the target suffixes (potential mismatch between prediction and filename).

Moreover, it produces different output filenames depending if the field is present or not in the model (from a programming standpoint, I think it would be easier to deal with consistent filenames (independent from the model/task) for downstream tools, instead of disparate filenames.

My preferred alternative would be to keep the prediction filename agnostic to the model/task by having the files named according to the class (ex: sub-01_T2w_class-0_pred.nii.gz). In my opinion, this is a more robust to different scenarios, lessen the potential for errors with less intervention from the users, and is easier to explain/document for the users. Note: we are already using the class0, class1, etc. notation for the columns of the evaluation metrics in evaluation_3Dmetrics.csv.

[uzaymacar]

I second @jcohenadad’s suggestion.

Automated rules for naming is tricky because they might not always produce meaningful results. For example, in #1031 I proposed to get the first element in the list in the case of multiple raters but as @mariehbourget pointed here this doesn’t work well for all scenarios.

We might have different and more complex "target_suffix" scenarios in the future of ivadomed, and a new field "prediction_suffix" in the config file would also solve these. To make sure old ivadomed configs stay compatible, we can make this field optional s.t.

• if it is not filled, then "prediction_suffix" will just be equivalent to "target_suffix" which will reserve the old behavior
• if it is filled, we will use it to generate the prediction filenames

It is also relevant to note that if "prediction_suffix" is not filled for the multi-rater case, then as https://github.com/ivadomed/model_seg_ms_mp2rage/issues/27 and #816 points out there will be an error. We can add a meaningful error message like “looks like you have multiple raters, please specify "prediction_suffix" if this is the case.”

Let me know your thoughts! I am happy to open a PR on this once we settle on a solution as this is critical for the https://github.com/ivadomed/model_seg_ms_mp2rage project.