question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

RuntimeError: Unknown type: itkMatrixF44

See original GitHub issue

Describe the bug MONAILabel fails when running spleen_deepedit_annotation on in 3D Slicer 5.0.3 and with the MONAILabel plugin version: 7cd40d8 (2022-10-10).

To Reproduce Steps to reproduce the behavior:

  1. Run MONAI Label on the server:
# install MONAI Label
pip install monailabel

# download Bundle sample app to local directory
monailabel apps --name monaibundle --download --output apps

# download a local study images, sample dataset such as spleen:
monailabel datasets --download --name Task09_Spleen --output datasets

monailabel start_server --app apps/monaibundle --studies datasets/Task09_Spleen/imagesTr --conf models "spleen_ct_segmentation_v0.3.1,spleen_deepedit_annotation_v0.3.0"
  1. On client, start 3D Slicer and MONAI label, input the server (e.g. http://cp0293:8000)
  2. Click Active Learning > Next Sample
  3. Click Auto Segmentation > Run for spleen_ct_segmentation_v0.3.1
  4. Click SmartEdit / Deepgrow > Landmarks for spleen_deepedit_annotation_v0.3.0 and click on a spot in the image just outside of the automatic segmentation
  5. Slicer error because of 500 Internal server error and an error in the server log:
(monai) [benjaminr@cp0293 MONAILabel]$ monailabel start_server --app workspace/monaibundle --studies datasets/Task09_Spleen/imagesTr --conf models "spleen_ct_segmentation_v0.3.1,spleen_deepedit_annotation_v0.3.0"
Using PYTHONPATH=/srv/scratch/benjaminr/anaconda3/envs::'/monaibundle/model/renalStructures_UNEST_segmentation_v0.2.0/scripts:'/monaibundle/model/renalStructures_UNEST_segmentation_v0.1.3/scripts:/srv/scratch/benjaminr/monai/MONAILabel/sample-apps/monaibundle/model/renalStructures_UNEST_segmentation_v0.1.3/scripts:/srv/scratch/benjaminr/monai/MONAILabel/sample-apps/monaibundle/model/renalStructures_UNEST_segmentation_v0.1.3/scripts

2022-10-12 16:43:45,007 - USING:: version = False
2022-10-12 16:43:45,007 - USING:: app = /srv/scratch/benjaminr/monai/MONAILabel/workspace/monaibundle
2022-10-12 16:43:45,007 - USING:: studies = /srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr
2022-10-12 16:43:45,007 - USING:: verbose = INFO
2022-10-12 16:43:45,007 - USING:: conf = [['models', 'spleen_ct_segmentation_v0.3.1,spleen_deepedit_annotation_v0.3.0']]
2022-10-12 16:43:45,007 - USING:: host = 0.0.0.0
2022-10-12 16:43:45,007 - USING:: port = 8000
2022-10-12 16:43:45,007 - USING:: uvicorn_app = monailabel.app:app
2022-10-12 16:43:45,007 - USING:: ssl_keyfile = None
2022-10-12 16:43:45,007 - USING:: ssl_certfile = None
2022-10-12 16:43:45,007 - USING:: ssl_keyfile_password = None
2022-10-12 16:43:45,007 - USING:: ssl_ca_certs = None
2022-10-12 16:43:45,007 - USING:: workers = None
2022-10-12 16:43:45,007 - USING:: limit_concurrency = None
2022-10-12 16:43:45,007 - USING:: access_log = False
2022-10-12 16:43:45,007 - USING:: log_config = None
2022-10-12 16:43:45,007 - USING:: dryrun = False
2022-10-12 16:43:45,007 - USING:: action = start_server
2022-10-12 16:43:45,007 - ENV SETTINGS:: MONAI_LABEL_API_STR = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_PROJECT_NAME = MONAILabel
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_APP_DIR = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_STUDIES = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_AUTH_ENABLE = False
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_AUTH_DB = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_APP_CONF = '{}'
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_TASKS_TRAIN = True
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_TASKS_STRATEGY = True
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_TASKS_SCORING = True
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_TASKS_BATCH_INFER = True
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_URL = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_USERNAME = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PASSWORD = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_API_KEY = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_CACHE_PATH = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PROJECT = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_ASSET_PATH = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_DSA_ANNOTATION_GROUPS = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_USERNAME = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PASSWORD = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_PATH = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_QIDO_PREFIX = None
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_WADO_PREFIX = None
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_STOW_PREFIX = None
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_FETCH_BY_FRAME = False
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CONVERT_TO_NIFTI = True
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_SEARCH_FILTER = '{"Modality": "CT"}'
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_EXPIRY = 180
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PROXY_TIMEOUT = 30.0
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_READ_TIMEOUT = 5.0
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_AUTO_RELOAD = True
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_FILE_EXT = '["*.nii.gz", "*.nii", "*.nrrd", "*.jpg", "*.png", "*.tif", "*.svs", "*.xml"]'
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_SERVER_PORT = 8000
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_CORS_ORIGINS = '[]'
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_SESSIONS = True
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_SESSION_PATH = 
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_SESSION_EXPIRY = 3600
2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_INFER_CONCURRENCY = -1
2022-10-12 16:43:45,009 - ENV SETTINGS:: MONAI_LABEL_INFER_TIMEOUT = 600
2022-10-12 16:43:45,009 - ENV SETTINGS:: MONAI_LABEL_AUTO_UPDATE_SCORING = True
2022-10-12 16:43:45,009 - 
Allow Origins: ['*']
[2022-10-12 16:43:45,422] [49797] [MainThread] [INFO] (uvicorn.error:75) - Started server process [49797]
[2022-10-12 16:43:45,423] [49797] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup.
[2022-10-12 16:43:45,423] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.app:38) - Initializing App from: /srv/scratch/benjaminr/monai/MONAILabel/workspace/monaibundle; studies: /srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr; conf: {'models': 'spleen_ct_segmentation_v0.3.1,spleen_deepedit_annotation_v0.3.0'}
[2022-10-12 16:43:45,646] [49797] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2022-10-12 16:43:45,751] [49797] [MainThread] [INFO] (main:81) - +++ Adding Bundle from Zoo: spleen_ct_segmentation_v0.3.1 => {'checksum': '271cc5248be5b527b9a06f95e63cbbf91e81681e', 'source': 'https://github.com/Project-MONAI/model-zoo/releases/download/hosting_storage_v1/spleen_ct_segmentation_v0.3.1.zip'} => /srv/scratch/benjaminr/monai/MONAILabel/workspace/monaibundle/model/spleen_ct_segmentation_v0.3.1
[2022-10-12 16:43:45,751] [49797] [MainThread] [INFO] (main:81) - +++ Adding Bundle from Zoo: spleen_deepedit_annotation_v0.3.0 => {'checksum': '235ca2f1614f8ee189a309d7cefe1615d86a5e59', 'source': 'https://github.com/Project-MONAI/model-zoo/releases/download/hosting_storage_v1/spleen_deepedit_annotation_v0.3.0.zip'} => /srv/scratch/benjaminr/monai/MONAILabel/workspace/monaibundle/model/spleen_deepedit_annotation_v0.3.0
[2022-10-12 16:43:45,751] [49797] [MainThread] [INFO] (main:90) - +++ Using Models: ['spleen_ct_segmentation_v0.3.1', 'spleen_deepedit_annotation_v0.3.0']
[2022-10-12 16:43:45,751] [49797] [MainThread] [INFO] (monailabel.interfaces.app:129) - Init Datastore for: /srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr
[2022-10-12 16:43:45,752] [49797] [MainThread] [INFO] (monailabel.datastore.local:128) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2022-10-12 16:43:45,762] [49797] [MainThread] [INFO] (monailabel.datastore.local:574) - Invalidate count: 0
[2022-10-12 16:43:45,763] [49797] [MainThread] [INFO] (monailabel.datastore.local:148) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-10-12 16:43:46,097] [49797] [MainThread] [INFO] (main:108) - +++ Adding Inferer:: spleen_ct_segmentation_v0.3.1 => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x7fed02e8a4a0>
[2022-10-12 16:43:46,769] [49797] [MainThread] [INFO] (main:108) - +++ Adding Inferer:: spleen_deepedit_annotation_v0.3.0 => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x7fed02e89990>
[2022-10-12 16:43:46,770] [49797] [MainThread] [INFO] (main:122) - +++ Adding Trainer:: spleen_ct_segmentation_v0.3.1 => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x7fed02e89960>
[2022-10-12 16:43:46,771] [49797] [MainThread] [INFO] (main:122) - +++ Adding Trainer:: spleen_deepedit_annotation_v0.3.0 => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x7fed000793c0>
[2022-10-12 16:43:46,771] [49797] [MainThread] [INFO] (main:132) - Active Learning Strategies:: ['random', 'first']
[2022-10-12 16:43:46,771] [49797] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/benjaminr/.cache/monailabel/sessions
[2022-10-12 16:43:46,771] [49797] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-10-12 16:43:46,771] [49797] [MainThread] [INFO] (monailabel.interfaces.app:467) - App Init - completed
[2022-10-12 16:43:46,773] [timeloop] [INFO] Starting Timeloop..
[2022-10-12 16:43:46,773] [49797] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2022-10-12 16:43:46,774] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x7fed02e9dd80>
[2022-10-12 16:43:46,774] [49797] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x7fed02e9dd80>
[2022-10-12 16:43:46,775] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2022-10-12 16:43:46,775] [49797] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2022-10-12 16:43:46,775] [49797] [MainThread] [INFO] (uvicorn.error:59) - Application startup complete.
[2022-10-12 16:43:46,777] [49797] [MainThread] [INFO] (uvicorn.error:206) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2022-10-12 16:44:07,564] [49797] [MainThread] [INFO] (monailabel.endpoints.activelearning:43) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'}
[2022-10-12 16:44:07,570] [49797] [MainThread] [INFO] (monailabel.tasks.activelearning.random:45) - Random: Selected Image: spleen_45; Weight: 1665585847
[2022-10-12 16:44:07,582] [49797] [MainThread] [INFO] (monailabel.endpoints.activelearning:59) - Next sample: {'id': 'spleen_45', 'weight': 1665585847, 'path': '/srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr/spleen_45.nii.gz', 'ts': 1665494588, 'name': 'spleen_45.nii.gz'}
[2022-10-12 16:44:56,001] [49797] [MainThread] [INFO] (monailabel.endpoints.infer:160) - Infer Request: {'model': 'spleen_ct_segmentation_v0.3.1', 'image': 'spleen_45', 'device': 'cuda:0', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'}
[2022-10-12 16:44:56,002] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:258) - Infer Request (final): {'device': 'cuda:0', 'model': 'spleen_ct_segmentation_v0.3.1', 'image': '/srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr/spleen_45.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'A pre-trained model for volumetric (3D) segmentation of the spleen from CT image'}
[2022-10-12 16:44:56,010] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - PRE - Run Transform(s)
[2022-10-12 16:44:56,010] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path']
[2022-10-12 16:44:57,789] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (LoadImaged): Time: 1.7768; image: (512, 512, 112)(torch.float32)
[2022-10-12 16:44:57,790] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (EnsureChannelFirstd): Time: 0.0007; image: (1, 512, 512, 112)(torch.float32)
[2022-10-12 16:44:57,795] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (Orientationd): Time: 0.0042; image: (1, 512, 512, 112)(torch.float32)
[2022-10-12 16:44:59,816] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (Spacingd): Time: 2.0205; image: (1, 328, 328, 223)(torch.float32)
[2022-10-12 16:44:59,950] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (ScaleIntensityRanged): Time: 0.1329; image: (1, 328, 328, 223)(torch.float32)
[2022-10-12 16:44:59,950] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (EnsureTyped): Time: 0.0002; image: (1, 328, 328, 223)(torch.float32)
[2022-10-12 16:44:59,951] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:426) - Inferer:: cuda:0 => SlidingWindowInferer => {'roi_size': [96, 96, 96], 'sw_batch_size': 4, 'overlap': 0.5, 'mode': constant, 'sigma_scale': 0.125, 'padding_mode': constant, 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'roi_weight_map': None}
[2022-10-12 16:44:59,952] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:375) - Infer model path: /srv/scratch/benjaminr/monai/MONAILabel/workspace/monaibundle/model/spleen_ct_segmentation_v0.3.1/models/model.pt
[2022-10-12 16:45:07,481] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - POST - Run Transform(s)
[2022-10-12 16:45:07,482] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'image_meta_dict', 'latencies', 'pred']
[2022-10-12 16:45:07,489] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (Activationsd): Time: 0.0066; image: (1, 328, 328, 223)(torch.float32); pred: (2, 328, 328, 223)(torch.float32)
[2022-10-12 16:45:08,324] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (Invertd): Time: 0.834; image: (1, 328, 328, 223)(torch.float32); pred: (2, 512, 512, 112)(torch.float32)
[2022-10-12 16:45:13,926] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (AsDiscreted): Time: 5.6012; image: (1, 328, 328, 223)(torch.float32); pred: (1, 512, 512, 112)(torch.float32)
[2022-10-12 16:45:13,927] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (Restored): Time: 0.0006; image: (1, 328, 328, 223)(torch.float32); pred: (512, 512, 112)(torch.float32)
[2022-10-12 16:45:13,927] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:458) - Writing Result...
[2022-10-12 16:45:13,930] [49797] [MainThread] [INFO] (monailabel.transform.writer:188) - Result ext: .nrrd; write_to_file: True; dtype: uint8
[2022-10-12 16:45:19,251] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:290) - ++ Latencies => Total: 23.2501; Pre: 3.9487; Inferer: 7.5193; Invert: 0.0000; Post: 6.4573; Write: 5.3241
[2022-10-12 16:45:19,252] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:314) - Result File: /tmp/tmprudwjr3j.nrrd
[2022-10-12 16:45:19,252] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:315) - Result Json Keys: ['label_names', 'latencies']
[2022-10-12 16:46:12,929] [49797] [MainThread] [INFO] (monailabel.endpoints.infer:160) - Infer Request: {'model': 'spleen_deepedit_annotation_v0.3.0', 'image': 'spleen_45', 'spleen': [[140, 256, 82]], 'label': 'spleen', 'device': 'cuda:0', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'}
[2022-10-12 16:46:12,932] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:258) - Infer Request (final): {'device': 'cuda:0', 'model': 'spleen_deepedit_annotation_v0.3.0', 'image': '/srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr/spleen_45.nii.gz', 'spleen': [[140, 256, 82]], 'label': 'spleen', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'This is a pre-trained model for 3D segmentation of the spleen organ from CT images using DeepEdit.'}
[2022-10-12 16:46:12,947] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - PRE - Run Transform(s)
[2022-10-12 16:46:12,948] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - PRE - Input Keys: ['device', 'model', 'image', 'spleen', 'label', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path']
[2022-10-12 16:46:14,737] [49797] [MainThread] [ERROR] (uvicorn.error:369) - Exception in ASGI application
Traceback (most recent call last):
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/fastapi/applications.py", line 269, in __call__
    await super().__call__(scope, receive, send)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/exceptions.py", line 93, in __call__
    raise exc
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in __call__
    await self.app(scope, receive, sender)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/routing.py", line 670, in __call__
    await route.handle(scope, receive, send)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/routing.py", line 266, in handle
    await self.app(scope, receive, send)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/routing.py", line 65, in app
    response = await func(request)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app
    raw_response = await run_endpoint_function(
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/endpoints/infer.py", line 179, in api_run_inference
    return run_inference(background_tasks, model, image, session_id, params, file, label, output)
  File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/endpoints/infer.py", line 161, in run_inference
    result = instance.infer(request)
  File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/interfaces/app.py", line 299, in infer
    result_file_name, result_json = task(request)
  File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/tasks/infer/basic_infer.py", line 267, in __call__
    data = self.run_pre_transforms(data, pre_transforms)
  File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/tasks/infer/basic_infer.py", line 343, in run_pre_transforms
    return run_transforms(data, transforms, log_prefix="PRE", use_compose=False)
  File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/interfaces/utils/transform.py", line 93, in run_transforms
    data = t(data)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/transforms/io/dictionary.py", line 154, in __call__
    data = self._loader(d[key], reader)
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/transforms/io/array.py", line 271, in __call__
    meta_data = switch_endianness(meta_data, "<")
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/transforms/io/array.py", line 84, in switch_endianness
    data = {k: switch_endianness(v, new) for k, v in data.items()}
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/transforms/io/array.py", line 84, in <dictcomp>
    data = {k: switch_endianness(v, new) for k, v in data.items()}
  File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/transforms/io/array.py", line 86, in switch_endianness
    raise RuntimeError(f"Unknown type: {type(data).__name__}")
RuntimeError: Unknown type: itkMatrixF44

Expected behavior

After step 5, MONAI Label server should return a new segmentation where the old segmentation was grown to the region of the user click.

Environment

Ensuring you use the relevant python executable, please paste the output of:

================================
Printing MONAI config...
================================
MONAI version: 1.0.0
Numpy version: 1.23.2
Pytorch version: 1.12.1+cu113
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 170093375ce29267e45681fcec09dfa856e1d7e7
MONAI __file__: /srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.10
Nibabel version: 4.0.2
scikit-image version: 0.19.3
Pillow version: 9.2.0
Tensorboard version: 2.10.0
gdown version: 4.5.1
TorchVision version: 0.13.1+cu113
tqdm version: 4.64.1
lmdb version: 1.3.0
psutil version: 5.9.2
pandas version: NOT INSTALLED or UNKNOWN VERSION.
einops version: 0.4.1
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: 0.4.3

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: CentOS Linux 7 (Core)
Platform: Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-glibc2.17
Processor: x86_64
Machine: x86_64
Python version: 3.10.6
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: [popenfile(path='/home/benjaminr/.vscode-server/data/logs/20221012T155029/ptyhost.log', fd=19, position=2188, mode='a', flags=33793), popenfile(path='/home/benjaminr/.vscode-server/data/logs/20221012T155029/remoteagent.log', fd=20, position=1958, mode='a', flags=33793), popenfile(path='/home/benjaminr/.vscode-server/bin/74b1f979648cc44d385a2286793c226e611f59e7/vscode-remote-lock.benjaminr.74b1f979648cc44d385a2286793c226e611f59e7', fd=99, position=0, mode='w', flags=32769)]
Num physical CPUs: 16
Num logical CPUs: 32
Num usable CPUs: 32
CPU usage (%): [90.8, 14.9, 17.0, 13.9, 13.0, 12.0, 13.4, 13.9, 82.1, 18.3, 17.8, 7.8, 14.6, 18.8, 6.0, 17.1, 7.8, 8.3, 6.9, 6.9, 10.0, 6.4, 5.5, 6.0, 6.0, 5.5, 16.7, 8.3, 5.5, 7.3, 100.0, 6.4]
CPU freq. (MHz): 1515
Load avg. in last 1, 5, 15 mins (%): [10.5, 7.8, 6.5]
Disk usage (%): 87.9
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 345.8
Available memory (GB): 285.6
Used memory (GB): 59.0

================================
Printing GPU config...
================================
Num GPUs: 8
Has CUDA: True
CUDA version: 11.3
cuDNN enabled: True
cuDNN version: 8302
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']
GPU 0 Name: NVIDIA A30
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 56
GPU 0 Total memory (GB): 23.7
GPU 0 CUDA capability (maj.min): 8.0
GPU 1 Name: NVIDIA GeForce RTX 3090
GPU 1 Is integrated: False
GPU 1 Is multi GPU board: False
GPU 1 Multi processor count: 82
GPU 1 Total memory (GB): 23.7
GPU 1 CUDA capability (maj.min): 8.6
GPU 2 Name: NVIDIA TITAN V
GPU 2 Is integrated: False
GPU 2 Is multi GPU board: False
GPU 2 Multi processor count: 80
GPU 2 Total memory (GB): 11.8
GPU 2 CUDA capability (maj.min): 7.0
GPU 3 Name: NVIDIA TITAN V
GPU 3 Is integrated: False
GPU 3 Is multi GPU board: False
GPU 3 Multi processor count: 80
GPU 3 Total memory (GB): 11.8
GPU 3 CUDA capability (maj.min): 7.0
GPU 4 Name: NVIDIA GeForce RTX 3090
GPU 4 Is integrated: False
GPU 4 Is multi GPU board: False
GPU 4 Multi processor count: 82
GPU 4 Total memory (GB): 23.7
GPU 4 CUDA capability (maj.min): 8.6
GPU 5 Name: NVIDIA GeForce RTX 3090
GPU 5 Is integrated: False
GPU 5 Is multi GPU board: False
GPU 5 Multi processor count: 82
GPU 5 Total memory (GB): 23.7
GPU 5 CUDA capability (maj.min): 8.6
GPU 6 Name: NVIDIA GeForce RTX 3090
GPU 6 Is integrated: False
GPU 6 Is multi GPU board: False
GPU 6 Multi processor count: 82
GPU 6 Total memory (GB): 23.7
GPU 6 CUDA capability (maj.min): 8.6
GPU 7 Name: NVIDIA GeForce RTX 3090
GPU 7 Is integrated: False
GPU 7 Is multi GPU board: False
GPU 7 Multi processor count: 82
GPU 7 Total memory (GB): 23.7
GPU 7 CUDA capability (maj.min): 8.6

Additional context Running 3D Slicer 5.0.3 r30893 / 7ea0f43 on a MacBook Pro (14-inch, 2021). MONAILabel plugin version: 7cd40d8 (2022-10-10)

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:7 (1 by maintainers)

github_iconTop GitHub Comments

2reactions
berombaucommented, Oct 13, 2022

Thank you for your quick response, using python=3.9 worked! Here are the steps for reference. My setup is that the MONAI Label server runs on a CentOS workstation with GPUs and I access it over the network with 3D Slicer using a Mac laptop.

mamba create -n monai -c pytorch python=3.9 pytorch torchvision torchaudio cudatoolkit=11.3 
conda activate monai
pip install monailabel
monailabel start_server ...

In my previous setup, I installed monailabel with 3.10 binaries using the itk pre-release:

pip install --upgrade --pre itk 
2reactions
tangy5commented, Oct 12, 2022

@berombau Thanks for reporting this. I tried to reproduce the error with my local env. It works with my env, but as Sachi mentioned, it’s not 3.10.6, it’s 3.9.12. I also tried to create an env with python 3.10.6, but I can’t build wheel with the “pip install monailabel”. There might be env version compatible issue. Here is my working env, if this help you solve the problem:

================================
Printing MONAI config...
================================
MONAI version: 1.0.0
Numpy version: 1.21.5
Pytorch version: 1.12.0+cu102
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 170093375ce29267e45681fcec09dfa856e1d7e7
MONAI __file__: /home/yucheng/anaconda3/lib/python3.9/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.10
Nibabel version: 3.1.1
scikit-image version: 0.19.2
Pillow version: 9.0.1
Tensorboard version: 2.9.1
gdown version: 4.5.1
TorchVision version: 0.13.0+cu102
tqdm version: 4.63.0
lmdb version: 1.3.0
psutil version: 5.8.0
pandas version: 1.4.2
einops version: 0.4.1
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: 0.4.3

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 20.04.4 LTS
Platform: Linux-5.15.0-48-generic-x86_64-with-glibc2.31
Processor: x86_64
Machine: x86_64
Python version: 3.9.12
Process name: ipython
Command: ['/home/yucheng/anaconda3/bin/python', '/home/yucheng/anaconda3/bin/ipython']
Open files: [popenfile(path='/home/yucheng/.ipython/profile_default/history.sqlite', fd=4, position=0, mode='r+', flags=688130), popenfile(path='/home/yucheng/.ipython/profile_default/history.sqlite', fd=5, position=0, mode='r+', flags=688130), popenfile(path='/home/yucheng/.ipython/profile_default/history.sqlite-journal', fd=30, position=0, mode='r+', flags=688130)]
Num physical CPUs: 4
Num logical CPUs: 8
Num usable CPUs: 8
CPU usage (%): [10.3, 10.9, 12.7, 11.8, 10.5, 11.0, 10.7, 37.8]
CPU freq. (MHz): 1702
Load avg. in last 1, 5, 15 mins (%): [23.8, 31.0, 33.6]
Disk usage (%): 33.5
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 31.2
Available memory (GB): 20.5
Used memory (GB): 9.6

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 10.2
cuDNN enabled: True
cuDNN version: 7605
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70']
GPU 0 Name: NVIDIA GeForce GTX 1080
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 20
GPU 0 Total memory (GB): 7.9
GPU 0 CUDA capability (maj.min): 6.1

Thanks

Read more comments on GitHub >

github_iconTop Results From Across the Web

"Unknown type constructor" error in TorchScript #29094 - GitHub
Bug Scripting functions and classes with initialisation of List, Dict, Tuple fails, if import typing is used instead of from typing import.
Read more >
MONAI Label - Training from Scratch - 3D Slicer Community
__name__}") RuntimeError: Unknown type: itkMatrixF44 The above exception was the direct cause of the following exception: Traceback (most recent call last): ...
Read more >
Torchscripted Pytorch Lightning Module Fails to Load
I'm attempting to launch a triton server instance with a torchscripted module. The module was trained with assistance from pytorch lightning ...
Read more >
RuntimeError: Unknown type name 'nn.Module' - jit
Hi All, I was just wondering how torch.jit.script can be used on functions that take nn.Module as an argument as well as Tensor?...
Read more >
Getting Unknown type annotation error when JIT saving ...
The issue turned out to be the that I was using class variable names which got mangled. Example: class Generator(nn.Module): __main: nn.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found