Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Cannot add multiple labels using MONAI Label server

See original GitHub issue

Describe the bug We are trying to do multi label annotation of liver and spleen. We used the spleen segmentation example and we followed this link to modify our multilabel annotation code. We changed the code as explained in the To Reproduce section,

To Reproduce Steps to reproduce the behavior:

  1. Go to apps/deepedit/main.py

    1. Change the out_channels in the dynunet network to 3. 0- background, 1- spleen, 2- liver
    network_params = {
                "spatial_dims": 3,
                "in_channels": 3,
                "out_channels": 3,

  2. Go to apps/deepedit/train.py

    1. Add the sigmoid=true in loss function
    def loss_function(self, context: Context):
        return DiceLoss(sigmoid=True, squared_pred=True)

  3. Go to apps/deepedit/train.py

    1. Change the to_onehot value to 2.
    def train_post_transforms(self, context: Context):
        return [
            Activationsd(keys="pred", sigmoid=True),
            AsDiscreted(keys="pred", threshold_values=True, to_onehot=3,logit_thresh=0.5),
        ]

  4. Then run the command monailabel start_server --app apps/deepedit --studies dataset/spleen

After running the above command, we selected an image by clicking the Next Sample button in Slicer App. The app pops up a dialog box showing Failed to run inference in MONAI Label server. In the terminal, we are getting the following error

Using PYTHONPATH=/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label:
[2022-02-07 16:05:38,689] [6221] [MainThread] [INFO] (__main__:277) - USING:: app = /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit
[2022-02-07 16:05:38,689] [6221] [MainThread] [INFO] (__main__:277) - USING:: studies = /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:05:38,689] [6221] [MainThread] [INFO] (__main__:277) - USING:: debug = False
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: conf = None
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: host = 0.0.0.0
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: port = 8000
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: log_config = None
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: dryrun = False
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: action = start_server
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:278) -
[2022-02-07 16:05:40,396] [6221] [MainThread] [INFO] (uvicorn.error:64) - Started server process [6221]
[2022-02-07 16:05:40,396] [6221] [MainThread] [INFO] (uvicorn.error:26) - Waiting for application startup.
[2022-02-07 16:05:40,396] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.app:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-07 16:05:40,408] [6221] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelAppFound: <class 'main.MyApp'>
[2022-02-07 16:05:40,937] [6221] [MainThread] [INFO] (main:104) - Pretrained Model Path: https://github.com/Project-MONAI/MONAILabel/releases/download/data/deepedit_dynunet_spleen.pt
[2022-02-07 16:05:40,938] [6221] [MainThread] [INFO] (main:109) - EPISTEMIC Enabled: 0; Samples: 5
[2022-02-07 16:05:40,938] [6221] [MainThread] [INFO] (main:113) - TTA Enabled: 0; Samples: 5
[2022-02-07 16:05:40,938] [6221] [MainThread] [INFO] (monailabel.interfaces.app:115) - Init Datastore for: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:05:40,938] [6221] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
[2022-02-07 16:05:40,941] [6221] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
[2022-02-07 16:05:40,941] [6221] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-02-07 16:05:40,943] [6221] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2022-02-07 16:05:40,943] [6221] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-02-07 16:05:40,944] [6221] [MainThread] [INFO] (monailabel.interfaces.app:396) - App Init - completed
[2022-02-07 16:05:40,944] [6221] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Scoring request: {'method': 'dice'}
[2022-02-07 16:05:40,945] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:57) - COMMAND:: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/bin/python -m monailabel.interfaces.utils.app -m scoring -r {"method":"dice","gpus":"all"}
[2022-02-07 16:05:40,945] [6221] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Scoring request: {'method': 'sum'}
[2022-02-07 16:05:40,945] [timeloop] [INFO] Starting Timeloop..
[2022-02-07 16:05:40,945] [6221] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2022-02-07 16:05:40,964] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete.<locals>.run_schedulerat 0x7fdca9f8f790>
[2022-02-07 16:05:40,964] [6221] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x7fdca9f8f790>
[2022-02-07 16:05:40,965] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2022-02-07 16:05:40,965] [6221] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2022-02-07 16:05:40,965] [6221] [MainThread] [INFO] (uvicorn.error:38) - Application startup complete.
[2022-02-07 16:05:40,966] [6221] [MainThread] [INFO] (uvicorn.error:199) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2022-02-07 16:05:41,108] [6306] [MainThread] [INFO] (__main__:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-07 16:05:43,465] [6306] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelAppFound: <class 'main.MyApp'>
[2022-02-07 16:05:43,990] [6306] [MainThread] [INFO] (main:104) - Pretrained Model Path: https://github.com/Project-MONAI/MONAILabel/releases/download/data/deepedit_dynunet_spleen.pt
[2022-02-07 16:05:43,990] [6306] [MainThread] [INFO] (main:109) - EPISTEMIC Enabled: 0; Samples: 5
[2022-02-07 16:05:43,990] [6306] [MainThread] [INFO] (main:113) - TTA Enabled: 0; Samples: 5
[2022-02-07 16:05:43,990] [6306] [MainThread] [INFO] (monailabel.interfaces.app:115) - Init Datastore for: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:05:43,990] [6306] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
[2022-02-07 16:05:43,993] [6306] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
[2022-02-07 16:05:43,993] [6306] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-02-07 16:05:43,995] [6306] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2022-02-07 16:05:43,995] [6306] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-02-07 16:05:43,996] [6306] [MainThread] [INFO] (__main__:62) - Result: {}
[2022-02-07 16:05:44,262] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:75) - Return code: 0
[2022-02-07 16:05:44,263] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:57) - COMMAND:: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/bin/python -m monailabel.interfaces.utils.app -m scoring -r {"method":"sum","gpus":"all"}
[2022-02-07 16:05:44,404] [6391] [MainThread] [INFO] (__main__:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-07 16:05:46,738] [6391] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelAppFound: <class 'main.MyApp'>
[2022-02-07 16:05:47,294] [6391] [MainThread] [INFO] (main:104) - Pretrained Model Path: https://github.com/Project-MONAI/MONAILabel/releases/download/data/deepedit_dynunet_spleen.pt
[2022-02-07 16:05:47,294] [6391] [MainThread] [INFO] (main:109) - EPISTEMIC Enabled: 0; Samples: 5
[2022-02-07 16:05:47,294] [6391] [MainThread] [INFO] (main:113) - TTA Enabled: 0; Samples: 5
[2022-02-07 16:05:47,294] [6391] [MainThread] [INFO] (monailabel.interfaces.app:115) - Init Datastore for: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:05:47,294] [6391] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
[2022-02-07 16:05:47,298] [6391] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
[2022-02-07 16:05:47,298] [6391] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-02-07 16:05:47,300] [6391] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2022-02-07 16:05:47,300] [6391] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-02-07 16:05:48,962] [6391] [MainThread] [INFO] (__main__:62) - Result: {"spleen_22": {"sum": 1487375, "slices": 193}, "spleen_13": {"sum": 1000, "slices": 10}}
[2022-02-07 16:05:49,218] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:75) - Return code: 0
[2022-02-07 16:06:09,863] [6221] [MainThread] [INFO] (monailabel.endpoints.activelearning:42) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'}
[2022-02-07 16:06:09,864] [6221] [MainThread] [INFO] (monailabel.tasks.activelearning.random:44) - Random: Images: ['spleen_46', 'spleen_22', 'spleen_9', 'spleen_6', 'spleen_13']; Weight: [3515, 3403, 2772, 2388, 2491]
[2022-02-07 16:06:09,865] [6221] [MainThread] [INFO] (monailabel.tasks.activelearning.random:45) - Random: Selected Image:spleen_22
[2022-02-07 16:06:09,865] [6221] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Scoring request: {'method': 'dice'}
[2022-02-07 16:06:09,866] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:57) - COMMAND:: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/bin/python -m monailabel.interfaces.utils.app -m scoring -r {"method":"dice","gpus":"all"}
[2022-02-07 16:06:09,867] [6221] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Scoring request: {'method': 'sum'}
[2022-02-07 16:06:09,897] [6221] [MainThread] [INFO] (monailabel.endpoints.activelearning:58) - Next sample: {'id': 'spleen_22', 'ts': 1643905192, 'checksum': 'SHA256:e2d12bf008afbbbcde50b3817b63e92fda3593999c944be0b52382b62d8cda9d', 'name': 'spleen_22.nii.gz', 'strategy': {'random': {'ts': 1644249969, 'client_id': 'user-xyz'}}, 'path': '/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen/spleen_22.nii.gz'}
[2022-02-07 16:06:10,041] [6475] [MainThread] [INFO] (__main__:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-07 16:06:12,399] [6475] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelAppFound: <class 'main.MyApp'>
[2022-02-07 16:06:12,951] [6475] [MainThread] [INFO] (main:104) - Pretrained Model Path: https://github.com/Project-MONAI/MONAILabel/releases/download/data/deepedit_dynunet_spleen.pt
[2022-02-07 16:06:12,952] [6475] [MainThread] [INFO] (main:109) - EPISTEMIC Enabled: 0; Samples: 5
[2022-02-07 16:06:12,952] [6475] [MainThread] [INFO] (main:113) - TTA Enabled: 0; Samples: 5
[2022-02-07 16:06:12,952] [6475] [MainThread] [INFO] (monailabel.interfaces.app:115) - Init Datastore for: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:06:12,952] [6475] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
[2022-02-07 16:06:12,955] [6475] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
[2022-02-07 16:06:12,955] [6475] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-02-07 16:06:12,957] [6475] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2022-02-07 16:06:12,957] [6475] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-02-07 16:06:12,957] [6475] [MainThread] [INFO] (__main__:62) - Result: {}
[2022-02-07 16:06:13,225] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:75) - Return code: 0
[2022-02-07 16:06:13,226] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:57) - COMMAND:: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/bin/python -m monailabel.interfaces.utils.app -m scoring -r {"method":"sum","gpus":"all"}
[2022-02-07 16:06:13,389] [6559] [MainThread] [INFO] (__main__:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-07 16:06:14,636] [6221] [MainThread] [INFO] (monailabel.endpoints.infer:147) - Infer Request: {'model': 'deepedit_seg', 'image': 'spleen_22', 'client_id': 'user-xyz'}
[2022-02-07 16:06:14,637] [6221] [MainThread] [INFO] (monailabel.interfaces.app:224) - Image => /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen/spleen_22.nii.gz
[2022-02-07 16:06:14,637] [6221] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:221) - Infer Request (final): {'model': 'deepedit_seg', 'image': '/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen/spleen_22.nii.gz', 'client_id': 'user-xyz'}
[2022-02-07 16:06:14,640] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:61) - PRE - Run Transform
[2022-02-07 16:06:14,641] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:62) - PRE - Input Keys: dict_keys(['model', 'image', 'client_id', 'image_path'])
[2022-02-07 16:06:15,714] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (LoadImaged): Time: 1.0716; image: (512, 512, 148)(float32)
[2022-02-07 16:06:15,714] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (SingleLabelSingleModalityd): Time: 0.0000; image: (512, 512, 148)(float32)
[2022-02-07 16:06:15,714] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (AddChanneld): Time: 0.0000; image: (1, 512, 512, 148)(float32)
[2022-02-07 16:06:15,783] [6559] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelAppFound: <class 'main.MyApp'>
[2022-02-07 16:06:16,370] [6559] [MainThread] [INFO] (main:104) - Pretrained Model Path: https://github.com/Project-MONAI/MONAILabel/releases/download/data/deepedit_dynunet_spleen.pt
[2022-02-07 16:06:16,371] [6559] [MainThread] [INFO] (main:109) - EPISTEMIC Enabled: 0; Samples: 5
[2022-02-07 16:06:16,371] [6559] [MainThread] [INFO] (main:113) - TTA Enabled: 0; Samples: 5
[2022-02-07 16:06:16,371] [6559] [MainThread] [INFO] (monailabel.interfaces.app:115) - Init Datastore for: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:06:16,371] [6559] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
[2022-02-07 16:06:16,376] [6559] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
[2022-02-07 16:06:16,376] [6559] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-02-07 16:06:16,381] [6559] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2022-02-07 16:06:16,381] [6559] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-02-07 16:06:18,087] [6559] [MainThread] [INFO] (__main__:62) - Result: {"spleen_22": {"sum": 1487375, "slices": 193}, "spleen_13": {"sum": 1000, "slices": 10}}
[2022-02-07 16:06:18,366] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:75) - Return code: 0
[2022-02-07 16:06:19,511] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (Spacingd): Time: 3.7970; image: (1, 496, 496, 222)(float32)
[2022-02-07 16:06:19,512] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (Orientationd): Time: 0.0010; image: (1, 496, 496, 222)(float32)
[2022-02-07 16:06:20,061] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (NormalizeIntensityd): Time: 0.5483; image: (1, 496, 496, 222)(float32)
[2022-02-07 16:06:20,421] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (Resized): Time: 0.3595; image: (1, 256, 256, 128)(float32)
[2022-02-07 16:06:20,451] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (DiscardAddGuidanced): Time: 0.0297; image: (3, 256, 256, 128)(float32)
[2022-02-07 16:06:20,451] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (ToTensord): Time: 0.0001; image: torch.Size([3, 256, 256, 128])(torch.float32)
[2022-02-07 16:06:20,451] [6221] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:344) - Running Inferer:: SimpleInferer
[2022-02-07 16:06:20,451] [6221] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:297) - Infer model path: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit/model/pretrained_dynunet.pt
[2022-02-07 16:06:24,198] [6221] [MainThread] [ERROR] (uvicorn.error:399) - Exception in ASGI application
Traceback (most recent call last):
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 396, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
    await self.app(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 52, in app
    response = await func(request)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/fastapi/routing.py", line 214, in app
    raw_response = await run_endpoint_function(
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/fastapi/routing.py", line 149, in run_endpoint_function
    return await dependant.call(**values)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/endpoints/infer.py", line 165, in api_run_inference
    return run_inference(background_tasks, model, image, session_id, params, file, label, output)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/endpoints/infer.py", line 148, in run_inference
    result = instance.infer(request)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/interfaces/app.py", line 225, in infer
    result_file_name, result_json = task(request)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/interfaces/tasks/infer.py", line 233, in __call__
    data = self.run_inferer(data, device=device)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/interfaces/tasks/infer.py", line 346, in run_inferer
    network = self._get_network(device)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/interfaces/tasks/infer.py", line 322, in _get_network
    network.load_state_dict(model_state_dict, strict=self.load_strict)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1406, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DynUNet:
        size mismatch for input_block.conv1.conv.weight: copying a param with shape torch.Size([32, 3, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 1, 3, 3, 3]).
        size mismatch for input_block.conv3.conv.weight: copying a param with shape torch.Size([32, 3, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([32, 1, 1, 1, 1]).
        size mismatch for output_block.conv.conv.weight: copying a param with shape torch.Size([1, 32, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 32, 1, 1, 1]).
        size mismatch for output_block.conv.conv.bias: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([3]).
        size mismatch for deep_supervision_heads.0.conv.conv.weight: copying a param with shape torch.Size([1, 64, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 64, 1, 1, 1]).
        size mismatch for deep_supervision_heads.0.conv.conv.bias: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([3]).
        size mismatch for deep_supervision_heads.1.conv.conv.weight: copying a param with shape torch.Size([1, 128, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 128, 1, 1, 1]).
        size mismatch for deep_supervision_heads.1.conv.conv.bias: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([3]).
        size mismatch for deep_supervision_heads.2.conv.conv.weight: copying a param with shape torch.Size([1, 256, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 256, 1, 1, 1]).
        size mismatch for deep_supervision_heads.2.conv.conv.bias: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([3]).
        size mismatch for deep_supervision_heads.3.conv.conv.weight: copying a param with shape torch.Size([1, 320, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 320, 1, 1, 1]).
        size mismatch for deep_supervision_heads.3.conv.conv.bias: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([3]).
        size mismatch for skip_layers.downsample.conv1.conv.weight: copying a param with shape torch.Size([32, 3, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 1, 3, 3, 3]).
        size mismatch for skip_layers.downsample.conv3.conv.weight: copying a param with shape torch.Size([32, 3, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([32, 1, 1, 1, 1]).

Environment

MONAI Label version : 0.3.1

================================
Printing MONAI config...
================================
MONAI version: 0.8.0
Numpy version: 1.22.2
Pytorch version: 1.9.0+cu111
MONAI flags: HAS_EXT = False, USE_COMPILED = False
MONAI rev id: 714d00dffe6653e21260160666c4c201ab66511b

Optional dependencies:
Pytorch Ignite version: 0.4.6
Nibabel version: 3.2.1
scikit-image version: 0.19.1
Pillow version: 9.0.1
Tensorboard version: 2.8.0
gdown version: 4.2.1
TorchVision version: 0.10.0+cu111
tqdm version: 4.62.3
lmdb version: 1.3.0
psutil version: 5.9.0
pandas version: NOT INSTALLED or UNKNOWN VERSION.
einops version: NOT INSTALLED or UNKNOWN VERSION.
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 20.04.3 LTS
Platform: Linux-4.15.0-136-generic-x86_64-with-glibc2.10
Processor: x86_64
Machine: x86_64
Python version: 3.8.12
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 20
Num logical CPUs: 40
Num usable CPUs: 40
CPU usage (%): [9.8, 9.7, 9.1, 9.7, 9.1, 9.0, 9.0, 9.0, 10.5, 9.1, 9.1, 9.8, 9.1, 9.8, 8.4, 9.8, 9.1, 3.0, 9.7, 9.8, 9.7, 9.0, 9.1, 10.4, 9.1, 9.1, 9.8, 9.8, 9.8, 100.0, 11.9, 9.1, 9.1, 4.5, 9.8, 3.8, 9.8, 8.3, 8.4, 9.7]
CPU freq. (MHz): 2
Load avg. in last 1, 5, 15 mins (%): [0.1, 0.3, 0.6]
Disk usage (%): 76.9
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 251.8
Available memory (GB): 245.8
Used memory (GB): 4.0

================================
Printing GPU config...
================================
Num GPUs: 4
Has CUDA: True
CUDA version: 11.1
cuDNN enabled: True
cuDNN version: 8005
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']
GPU 0 Name: Tesla V100-DGXS-32GB
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 80
GPU 0 Total memory (GB): 31.7
GPU 0 CUDA capability (maj.min): 7.0
GPU 1 Name: Tesla V100-DGXS-32GB
GPU 1 Is integrated: False
GPU 1 Is multi GPU board: False
GPU 1 Multi processor count: 80
GPU 1 Total memory (GB): 31.7
GPU 1 CUDA capability (maj.min): 7.0
GPU 2 Name: Tesla V100-DGXS-32GB
GPU 2 Is integrated: False
GPU 2 Is multi GPU board: False
GPU 2 Multi processor count: 80
GPU 2 Total memory (GB): 31.7
GPU 2 CUDA capability (maj.min): 7.0
GPU 3 Name: Tesla V100-DGXS-32GB
GPU 3 Is integrated: False
GPU 3 Is multi GPU board: False
GPU 3 Multi processor count: 80
GPU 3 Total memory (GB): 31.7
GPU 3 CUDA capability (maj.min): 7.0

Any suggestions is appreciated. Thanks in advance

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Reactions:1
  • Comments:11

github_iconTop GitHub Comments

2reactions
SachidanandAllecommented, Mar 7, 2022

if you are using multilabel… and 512x512xN kind of higher spatial size… with 32GB GPU… this won’t be enough… you can switch to small spatial size… please remember… this is more of resource related choices and not related to application logic.

1reaction
Suchi97commented, Mar 11, 2022

Thanks @SachidanandAlle and @diazandr3s for your reply. We trained the model using GPU of 32GB on images having the spatial size 128x128x128. Its working now.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Segmentation example on multiple labels · Issue #415 - GitHub
Several of my projects involve multiple labels - e.g. the right and left lateral ... Cannot add multiple labels using MONAI Label server...
Read more >
Quickstart — MONAI Label 0.6.0rc6 Documentation
Install MONAI Label Plugin in 3D Slicer¶ · Navigate Edit -> Application Settings · Under the Modules panel drag MONAI Label into Favorite...
Read more >
Source code for monailabel.datastore.local - Project MONAI
[docs]class LocalDatastore(Datastore): """ Class to represent a datastore local to the MONAI-Label Server Attributes ---------- `name: str` The name of the ...
Read more >
monailabel.datastore.local module - Project MONAI
Create a new model by parsing and validating input data from keyword arguments. Raises ValidationError if the input data cannot be parsed to...
Read more >
MONAI Label 0.6.0rc6 Documentation
The Leading Open Platform for Medical Data Labeling with AI ... where custom labeling apps are exposed as a service through the MONAI...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Training on during active learning is failing

Next page

Documentation of `--conf`