Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Inability to train model on monailabel

See original GitHub issue

Dear Experts,

Greetings from Mumbai, India.

At the outset thank you for the open source application for image annotation and segmentation.

I attempted segmentation model training on monailabel server deployed on WSL, however, training failed. Please find the entire log below for perusal and inputs:

monailabel start_server --app apps/radiology --studies /mnt/h/Cases/TMH/Liver/ --conf models segmentation --host 0.0.0.0 --port 8080
Using PYTHONPATH=/home/amitjc:

Failed to load image Python extension: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
2022-09-25 04:23:11,658 - USING:: version = False
2022-09-25 04:23:11,658 - USING:: app = /home/amitjc/apps/radiology
2022-09-25 04:23:11,658 - USING:: studies = /mnt/h/Cases/TMH/Liver
2022-09-25 04:23:11,658 - USING:: verbose = INFO
2022-09-25 04:23:11,658 - USING:: conf = [['models', 'segmentation']]
2022-09-25 04:23:11,658 - USING:: host = 0.0.0.0
2022-09-25 04:23:11,658 - USING:: port = 8080
2022-09-25 04:23:11,658 - USING:: uvicorn_app = monailabel.app:app
2022-09-25 04:23:11,658 - USING:: ssl_keyfile = None
2022-09-25 04:23:11,658 - USING:: ssl_certfile = None
2022-09-25 04:23:11,658 - USING:: ssl_keyfile_password = None
2022-09-25 04:23:11,658 - USING:: ssl_ca_certs = None
2022-09-25 04:23:11,658 - USING:: workers = None
2022-09-25 04:23:11,658 - USING:: limit_concurrency = None
2022-09-25 04:23:11,658 - USING:: access_log = False
2022-09-25 04:23:11,658 - USING:: log_config = None
2022-09-25 04:23:11,658 - USING:: dryrun = False
2022-09-25 04:23:11,658 - USING:: action = start_server
2022-09-25 04:23:11,658 - ENV SETTINGS:: MONAI_LABEL_API_STR =
2022-09-25 04:23:11,658 - ENV SETTINGS:: MONAI_LABEL_PROJECT_NAME = MONAILabel
2022-09-25 04:23:11,658 - ENV SETTINGS:: MONAI_LABEL_APP_DIR =
2022-09-25 04:23:11,658 - ENV SETTINGS:: MONAI_LABEL_STUDIES =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_AUTH_ENABLE = False
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_AUTH_DB =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_APP_CONF = '{}'
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_TASKS_TRAIN = True
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_TASKS_STRATEGY = True
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_TASKS_SCORING = True
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_TASKS_BATCH_INFER = True
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DATASTORE =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_URL =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_USERNAME =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PASSWORD =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_API_KEY =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_CACHE_PATH =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PROJECT =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_ASSET_PATH =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_DSA_ANNOTATION_GROUPS =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_USERNAME =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PASSWORD =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_PATH =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_QIDO_PREFIX =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_WADO_PREFIX =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_STOW_PREFIX =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_FETCH_BY_FRAME = False
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_SEARCH_FILTER = '{"Modality": "CT"}'
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_EXPIRY = 180
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_READ_TIMEOUT = 5.0
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_AUTO_RELOAD = True
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_FILE_EXT = '["*.nii.gz", "*.nii", "*.nrrd", "*.jpg", "*.png", "*.tif", "*.svs", "*.xml"]'
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_SERVER_PORT = 8000
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_CORS_ORIGINS = '[]'
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_SESSIONS = True
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_SESSION_PATH =
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_SESSION_EXPIRY = 3600
2022-09-25 04:23:11,659 - ENV SETTINGS:: MONAI_LABEL_INFER_CONCURRENCY = -1
2022-09-25 04:23:11,660 - ENV SETTINGS:: MONAI_LABEL_INFER_TIMEOUT = 600
2022-09-25 04:23:11,660 - ENV SETTINGS:: MONAI_LABEL_AUTO_UPDATE_SCORING = True
2022-09-25 04:23:11,660 -
Allow Origins: ['*']
[2022-09-25 04:23:12,084] [393] [MainThread] [INFO] (uvicorn.error:75) - Started server process [393]
[2022-09-25 04:23:12,084] [393] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup.
[2022-09-25 04:23:12,084] [393] [MainThread] [INFO] (monailabel.interfaces.utils.app:38) - Initializing App from: /home/amitjc/apps/radiology; studies: /mnt/h/Cases/TMH/Liver; conf: {'models': 'segmentation'}
[2022-09-25 04:23:12,354] [393] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2022-09-25 04:23:12,361] [393] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2022-09-25 04:23:12,361] [393] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2022-09-25 04:23:12,362] [393] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2022-09-25 04:23:12,362] [393] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2022-09-25 04:23:12,366] [393] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2022-09-25 04:23:12,366] [393] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2022-09-25 04:23:12,367] [393] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2022-09-25 04:23:12,367] [393] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2022-09-25 04:23:12,367] [393] [MainThread] [INFO] (main:86) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2022-09-25 04:23:12,406] [393] [MainThread] [INFO] (main:90) - +++ Using Models: ['segmentation']
[2022-09-25 04:23:12,406] [393] [MainThread] [INFO] (monailabel.interfaces.app:128) - Init Datastore for: /mnt/h/Cases/TMH/Liver
[2022-09-25 04:23:12,406] [393] [MainThread] [INFO] (monailabel.datastore.local:126) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2022-09-25 04:23:12,474] [393] [MainThread] [INFO] (monailabel.datastore.local:540) - Invalidate count: 0
[2022-09-25 04:23:12,474] [393] [MainThread] [INFO] (monailabel.datastore.local:146) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-09-25 04:23:12,602] [393] [MainThread] [INFO] (main:116) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x7f413c5adeb0>
[2022-09-25 04:23:12,602] [393] [MainThread] [INFO] (main:172) - {'segmentation': <lib.infers.segmentation.Segmentation object at 0x7f413c5adeb0>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x7f4134049850>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x7f4134049880>}
[2022-09-25 04:23:12,602] [393] [MainThread] [INFO] (main:185) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x7f4133ff67c0>
[2022-09-25 04:23:12,603] [393] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/amitjc/.cache/monailabel/sessions
[2022-09-25 04:23:12,603] [393] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-09-25 04:23:12,603] [393] [MainThread] [INFO] (monailabel.interfaces.app:465) - App Init - completed
[2022-09-25 04:23:12,603] [timeloop] [INFO] Starting Timeloop..
[2022-09-25 04:23:12,603] [393] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2022-09-25 04:23:12,603] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x7f4133fff160>
[2022-09-25 04:23:12,603] [393] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x7f4133fff160>
[2022-09-25 04:23:12,603] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2022-09-25 04:23:12,603] [393] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2022-09-25 04:23:12,603] [393] [MainThread] [INFO] (uvicorn.error:59) - Application startup complete.
[2022-09-25 04:23:12,603] [393] [MainThread] [INFO] (uvicorn.error:206) - Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
[2022-09-25 04:31:28,445] [393] [MainThread] [INFO] (monailabel.endpoints.datastore:67) - Image: CV17056; File: <starlette.datastructures.UploadFile object at 0x7f414e2283d0>; params: {"client_id": "user-xyz"}
[2022-09-25 04:31:28,513] [393] [MainThread] [INFO] (monailabel.datastore.local:402) - Adding Image: CV17056 => /tmp/tmpyt7z7vu_.nii.gz
[2022-09-25 04:36:30,193] [393] [MainThread] [INFO] (monailabel.endpoints.datastore:100) - Saving Label for CV17056 for tag: final by admin
[2022-09-25 04:36:30,194] [393] [MainThread] [INFO] (monailabel.endpoints.datastore:111) - Save Label params: {"label_info": [{"name": "spleen", "idx": 1}, {"name": "right kidney", "idx": 2}, {"name": "left kidney", "idx": 3}, {"name": "gallbladder", "idx": 4}, {"name": "esophagus", "idx": 5}, {"name": "liver", "idx": 6}, {"name": "stomach", "idx": 7}, {"name": "aorta", "idx": 8}, {"name": "inferior vena cava", "idx": 9}, {"name": "portal vein and splenic vein", "idx": 10}, {"name": "pancreas", "idx": 11}, {"name": "right adrenal gland", "idx": 12}, {"name": "left adrenal gland", "idx": 13}], "client_id": "user-xyz"}
[2022-09-25 04:36:30,194] [393] [MainThread] [INFO] (monailabel.datastore.local:449) - Saving Label for Image: CV17056; Tag: final; Info: {'label_info': [{'name': 'spleen', 'idx': 1}, {'name': 'right kidney', 'idx': 2}, {'name': 'left kidney', 'idx': 3}, {'name': 'gallbladder', 'idx': 4}, {'name': 'esophagus', 'idx': 5}, {'name': 'liver', 'idx': 6}, {'name': 'stomach', 'idx': 7}, {'name': 'aorta', 'idx': 8}, {'name': 'inferior vena cava', 'idx': 9}, {'name': 'portal vein and splenic vein', 'idx': 10}, {'name': 'pancreas', 'idx': 11}, {'name': 'right adrenal gland', 'idx': 12}, {'name': 'left adrenal gland', 'idx': 13}], 'client_id': 'user-xyz'}
[2022-09-25 04:36:30,194] [393] [MainThread] [INFO] (monailabel.datastore.local:457) - Adding Label: CV17056 => final => /tmp/tmpndizli05.nii.gz
[2022-09-25 04:36:30,212] [393] [MainThread] [INFO] (monailabel.datastore.local:473) - Label Info: {'label_info': [{'name': 'spleen', 'idx': 1}, {'name': 'right kidney', 'idx': 2}, {'name': 'left kidney', 'idx': 3}, {'name': 'gallbladder', 'idx': 4}, {'name': 'esophagus', 'idx': 5}, {'name': 'liver', 'idx': 6}, {'name': 'stomach', 'idx': 7}, {'name': 'aorta', 'idx': 8}, {'name': 'inferior vena cava', 'idx': 9}, {'name': 'portal vein and splenic vein', 'idx': 10}, {'name': 'pancreas', 'idx': 11}, {'name': 'right adrenal gland', 'idx': 12}, {'name': 'left adrenal gland', 'idx': 13}], 'client_id': 'user-xyz', 'ts': 1664060790, 'checksum': 'SHA256:be4f651b196e6c8799deebee7718d2204c55ecd7af144ee7d11ce936e85c06ac', 'name': 'CV17056.nii.gz'}
[2022-09-25 04:36:30,214] [393] [MainThread] [INFO] (monailabel.interfaces.app:489) - New label saved for: CV17056 => CV17056
[2022-09-25 04:36:39,812] [393] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Train request: {'model': 'segmentation', 'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'client_id': 'user-xyz'}
[2022-09-25 04:36:39,812] [393] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:59) - COMMAND:: /home/amitjc/anaconda3/envs/monailabel-env/bin/python -m monailabel.interfaces.utils.app -m train -r {"model":"segmentation","name":"train_01","pretrained":true,"device":"cuda","max_epochs":50,"early_stop_patience":-1,"val_split":0.2,"train_batch_size":1,"val_batch_size":1,"multi_gpu":true,"gpus":"all","dataset":"SmartCacheDataset","dataloader":"ThreadDataLoader","client_id":"user-xyz"}
[2022-09-25 04:36:39,859] [446] [MainThread] [INFO] (__main__:38) - Initializing App from: /home/amitjc/apps/radiology; studies: /mnt/h/Cases/TMH/Liver; conf: {'models': 'segmentation'}
Failed to load image Python extension: libtorch_cuda_cu.so: cannot open shared object file: No such file or directory
[2022-09-25 04:36:41,070] [446] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2022-09-25 04:36:41,072] [446] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2022-09-25 04:36:41,072] [446] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2022-09-25 04:36:41,073] [446] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2022-09-25 04:36:41,073] [446] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2022-09-25 04:36:41,074] [446] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2022-09-25 04:36:41,074] [446] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2022-09-25 04:36:41,074] [446] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2022-09-25 04:36:41,074] [446] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2022-09-25 04:36:41,075] [446] [MainThread] [INFO] (main:86) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2022-09-25 04:36:41,100] [446] [MainThread] [INFO] (main:90) - +++ Using Models: ['segmentation']
[2022-09-25 04:36:41,100] [446] [MainThread] [INFO] (monailabel.interfaces.app:128) - Init Datastore for: /mnt/h/Cases/TMH/Liver
[2022-09-25 04:36:41,100] [446] [MainThread] [INFO] (monailabel.datastore.local:126) - Auto Reload: False; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2022-09-25 04:36:41,158] [446] [MainThread] [INFO] (monailabel.datastore.local:540) - Invalidate count: 0
[2022-09-25 04:36:41,158] [446] [MainThread] [INFO] (main:116) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x7feeaf66edf0>
[2022-09-25 04:36:41,158] [446] [MainThread] [INFO] (main:172) - {'segmentation': <lib.infers.segmentation.Segmentation object at 0x7feeaf66edf0>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x7feea7102970>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x7feea71029a0>}
[2022-09-25 04:36:41,158] [446] [MainThread] [INFO] (main:185) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x7feea7102a30>
[2022-09-25 04:36:41,158] [446] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/amitjc/.cache/monailabel/sessions
[2022-09-25 04:36:41,158] [446] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-09-25 04:36:41,158] [446] [MainThread] [INFO] (monailabel.tasks.train.basic_train:365) - Train Request (input): {'model': 'segmentation', 'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'client_id': 'user-xyz', 'local_rank': 0}
[2022-09-25 04:36:41,158] [446] [MainThread] [INFO] (monailabel.tasks.train.basic_train:375) - CUDA_VISIBLE_DEVICES: None
[2022-09-25 04:36:41,166] [446] [MainThread] [INFO] (monailabel.tasks.train.basic_train:380) - Distributed/Multi GPU is limited
[2022-09-25 04:36:41,166] [446] [MainThread] [INFO] (monailabel.tasks.train.basic_train:395) - Distributed Training = FALSE
[2022-09-25 04:36:41,166] [446] [MainThread] [INFO] (monailabel.tasks.train.basic_train:422) - 0 - Train Request (final): {'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': False, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'model': 'segmentation', 'client_id': 'user-xyz', 'local_rank': 0, 'run_id': '20220925_0436'}
[2022-09-25 04:36:41,166] [446] [MainThread] [INFO] (monailabel.tasks.train.basic_train:521) - 0 - Using Device: cuda; IDX: None
[2022-09-25 04:36:41,167] [446] [MainThread] [INFO] (monailabel.tasks.train.basic_train:343) - Total Records for Training: 1
[2022-09-25 04:36:41,167] [446] [MainThread] [INFO] (monailabel.tasks.train.basic_train:344) - Total Records for Validation: 0
NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
[2022-09-25 04:36:41,871] [446] [MainThread] [INFO] (monailabel.tasks.train.basic_train:608) - 0 - Load Path /home/amitjc/apps/radiology/model/pretrained_segmentation.pt
Loading dataset:   0%|          | 0/1 [00:00<?, ?it/s]
Loading dataset: 100%|██████████| 1/1 [00:12<00:00, 12.42s/it]
Loading dataset: 100%|██████████| 1/1 [00:12<00:00, 12.42s/it]
cache_num is greater or equal than dataset length, fall back to regular monai.data.CacheDataset.
[2022-09-25 04:36:54,299] [446] [MainThread] [INFO] (monailabel.tasks.train.basic_train:228) - 0 - Records for Training: 1
[2022-09-25 04:36:54,302] [446] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:876) - Engine run resuming from iteration 0, epoch 0 until 50 epochs
[2022-09-25 04:36:54,346] [446] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:138) - Restored all variables from /home/amitjc/apps/radiology/model/pretrained_segmentation.pt
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/transform.py", line 90, in apply_transform
    return [_apply_transform(transform, item, unpack_items) for item in data]
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/transform.py", line 90, in <listcomp>
    return [_apply_transform(transform, item, unpack_items) for item in data]
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/transform.py", line 55, in _apply_transform
    return transform(parameters)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/spatial/dictionary.py", line 1227, in __call__
    d[key] = self.flipper(d[key])
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/spatial/array.py", line 765, in __call__
    out = self.forward_image(img, axes)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/spatial/array.py", line 756, in forward_image
    return torch.flip(img, axes)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/data/meta_tensor.py", line 249, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "/home/amitjc/.local/lib/python3.9/site-packages/torch/_tensor.py", line 1121, in __torch_function__
    ret = func(*args, **kwargs)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/data/thread_buffer.py", line 48, in enqueue_values
    for src_val in self.src:
  File "/home/amitjc/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/home/amitjc/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 721, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/amitjc/.local/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/amitjc/.local/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/data/dataset.py", line 105, in __getitem__
    return self._transform(index)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/data/dataset.py", line 878, in _transform
    data = apply_transform(_transform, data)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/transform.py", line 118, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.spatial.dictionary.RandFlipd object at 0x7feea7583760>
Exception in thread Thread-9:
Traceback (most recent call last):
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/transform.py", line 90, in apply_transform
    return [_apply_transform(transform, item, unpack_items) for item in data]
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/transform.py", line 90, in <listcomp>
    return [_apply_transform(transform, item, unpack_items) for item in data]
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/transform.py", line 55, in _apply_transform
    return transform(parameters)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/intensity/dictionary.py", line 421, in __call__
    d[key] = self.shifter(d[key], factor=factor, randomize=False)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/intensity/array.py", line 289, in __call__
    return self._shifter(img, self._offset if factor is None else self._offset * factor)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/intensity/array.py", line 236, in __call__
    out = img + offset
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/data/meta_tensor.py", line 249, in __torch_function__
    ret = super().__torch_function__(func, types, args, kwargs)
  File "/home/amitjc/.local/lib/python3.9/site-packages/torch/_tensor.py", line 1121, in __torch_function__
    ret = func(*args, **kwargs)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/threading.py", line 917, in run
    self._target(*self._args, **self._kwargs)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/data/thread_buffer.py", line 48, in enqueue_values
    for src_val in self.src:
  File "/home/amitjc/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/home/amitjc/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 721, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/amitjc/.local/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/amitjc/.local/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/data/dataset.py", line 105, in __getitem__
    return self._transform(index)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/data/dataset.py", line 878, in _transform
    data = apply_transform(_transform, data)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/transforms/transform.py", line 118, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.intensity.dictionary.RandShiftIntensityd object at 0x7feea7583850>
Data iterator can not provide data anymore but required total number of iterations to run is not reached. Current iteration: 0 vs Total iterations to run : 50
[2022-09-25 04:36:54,629] [446] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:992) - Engine run is terminating due to exception: the data to aggregate must be PyTorch Tensor.
[2022-09-25 04:36:54,629] [446] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:178) - Exception: the data to aggregate must be PyTorch Tensor.
Traceback (most recent call last):
  File "/home/amitjc/.local/lib/python3.9/site-packages/ignite/engine/engine.py", line 965, in _internal_run_as_gen
    self._fire_event(Events.EPOCH_COMPLETED)
  File "/home/amitjc/.local/lib/python3.9/site-packages/ignite/engine/engine.py", line 425, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/home/amitjc/.local/lib/python3.9/site-packages/ignite/metrics/metric.py", line 329, in completed
    result = self.compute()
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/handlers/ignite_metric.py", line 90, in compute
    result = self.metric_fn.aggregate()
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/metrics/meandice.py", line 99, in aggregate
    raise ValueError("the data to aggregate must be PyTorch Tensor.")
ValueError: the data to aggregate must be PyTorch Tensor.
Traceback (most recent call last):
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monailabel/interfaces/utils/app.py", line 132, in <module>
    run_main()
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monailabel/interfaces/utils/app.py", line 117, in run_main
    result = a.train(request)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monailabel/interfaces/app.py", line 416, in train
    result = task(request, self.datastore())
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monailabel/tasks/train/basic_train.py", line 396, in __call__
    res = self.train(0, world_size, req, datalist)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monailabel/tasks/train/basic_train.py", line 458, in train
    context.trainer.run()
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/engines/trainer.py", line 53, in run
    super().run()
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/engines/workflow.py", line 278, in run
    super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
  File "/home/amitjc/.local/lib/python3.9/site-packages/ignite/engine/engine.py", line 892, in run
    return self._internal_run()
  File "/home/amitjc/.local/lib/python3.9/site-packages/ignite/engine/engine.py", line 935, in _internal_run
    return next(self._internal_run_generator)
  File "/home/amitjc/.local/lib/python3.9/site-packages/ignite/engine/engine.py", line 993, in _internal_run_as_gen
    self._handle_exception(e)
  File "/home/amitjc/.local/lib/python3.9/site-packages/ignite/engine/engine.py", line 636, in _handle_exception
    self._fire_event(Events.EXCEPTION_RAISED, e)
  File "/home/amitjc/.local/lib/python3.9/site-packages/ignite/engine/engine.py", line 425, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/handlers/stats_handler.py", line 179, in exception_raised
    raise e
  File "/home/amitjc/.local/lib/python3.9/site-packages/ignite/engine/engine.py", line 965, in _internal_run_as_gen
    self._fire_event(Events.EPOCH_COMPLETED)
  File "/home/amitjc/.local/lib/python3.9/site-packages/ignite/engine/engine.py", line 425, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "/home/amitjc/.local/lib/python3.9/site-packages/ignite/metrics/metric.py", line 329, in completed
    result = self.compute()
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/handlers/ignite_metric.py", line 90, in compute
    result = self.metric_fn.aggregate()
  File "/home/amitjc/anaconda3/envs/monailabel-env/lib/python3.9/site-packages/monai/metrics/meandice.py", line 99, in aggregate
    raise ValueError("the data to aggregate must be PyTorch Tensor.")
ValueError: the data to aggregate must be PyTorch Tensor.
[2022-09-25 04:36:54,988] [393] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:77) - Return code: 1

RTX 3090 pytorch incompatibility error was returned; however,

conda list | grep "torch"

reveals

pytorch                   1.13.0.dev20220924 py3.9_cuda11.7_cudnn8.5.0_0    pytorch-nightly
pytorch-cuda              11.7                 h67b0de4_0    pytorch-nightly
pytorch-mutex             1.0                        cuda    pytorch-nightly
torchaudio                0.13.0.dev20220924      py39_cu117    pytorch-nightly
torchvision               0.13.1                   pypi_0    pypi

Monailabel server was deployed on wsl because, Ubuntu slicer monailabel module lacks “Train” and “Stop” tabs/buttons after deploying monailabel server on Ubuntu 22.04; issue raised over slicer forum

Monailabel server was also deployed successfully over powershell, however, training could not be done through this deployment either. Please find the logs below:

monailabel start_server --app apps/radiology --studies datasets/TMH/Liver/ --conf models segmentation
Using PYTHONPATH=C:\ProgramData\Anaconda3\envs;
""
2022-09-25 05:00:26,485 - USING:: version = False
2022-09-25 05:00:26,485 - USING:: app = N:\ApPz\MONAI\apps\radiology
2022-09-25 05:00:26,486 - USING:: studies = N:\ApPz\MONAI\datasets\TMH\Liver
2022-09-25 05:00:26,488 - USING:: verbose = INFO
2022-09-25 05:00:26,489 - USING:: conf = [['models', 'segmentation']]
2022-09-25 05:00:26,489 - USING:: host = 0.0.0.0
2022-09-25 05:00:26,490 - USING:: port = 8000
2022-09-25 05:00:26,494 - USING:: uvicorn_app = monailabel.app:app
2022-09-25 05:00:26,494 - USING:: ssl_keyfile = None
2022-09-25 05:00:26,495 - USING:: ssl_certfile = None
2022-09-25 05:00:26,495 - USING:: ssl_keyfile_password = None
2022-09-25 05:00:26,496 - USING:: ssl_ca_certs = None
2022-09-25 05:00:26,496 - USING:: workers = None
2022-09-25 05:00:26,496 - USING:: limit_concurrency = None
2022-09-25 05:00:26,497 - USING:: access_log = False
2022-09-25 05:00:26,497 - USING:: log_config = None
2022-09-25 05:00:26,497 - USING:: dryrun = False
2022-09-25 05:00:26,498 - USING:: action = start_server
2022-09-25 05:00:26,499 - ENV SETTINGS:: MONAI_LABEL_API_STR =
2022-09-25 05:00:26,499 - ENV SETTINGS:: MONAI_LABEL_PROJECT_NAME = MONAILabel
2022-09-25 05:00:26,499 - ENV SETTINGS:: MONAI_LABEL_APP_DIR =
2022-09-25 05:00:26,500 - ENV SETTINGS:: MONAI_LABEL_STUDIES =
2022-09-25 05:00:26,500 - ENV SETTINGS:: MONAI_LABEL_AUTH_ENABLE = False
2022-09-25 05:00:26,501 - ENV SETTINGS:: MONAI_LABEL_AUTH_DB =
2022-09-25 05:00:26,501 - ENV SETTINGS:: MONAI_LABEL_APP_CONF = '{}'
2022-09-25 05:00:26,504 - ENV SETTINGS:: MONAI_LABEL_TASKS_TRAIN = True
2022-09-25 05:00:26,507 - ENV SETTINGS:: MONAI_LABEL_TASKS_STRATEGY = True
2022-09-25 05:00:26,508 - ENV SETTINGS:: MONAI_LABEL_TASKS_SCORING = True
2022-09-25 05:00:26,511 - ENV SETTINGS:: MONAI_LABEL_TASKS_BATCH_INFER = True
2022-09-25 05:00:26,512 - ENV SETTINGS:: MONAI_LABEL_DATASTORE =
2022-09-25 05:00:26,512 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_URL =
2022-09-25 05:00:26,513 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_USERNAME =
2022-09-25 05:00:26,513 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PASSWORD =
2022-09-25 05:00:26,513 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_API_KEY =
2022-09-25 05:00:26,514 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_CACHE_PATH =
2022-09-25 05:00:26,514 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PROJECT =
2022-09-25 05:00:26,515 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_ASSET_PATH =
2022-09-25 05:00:26,515 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_DSA_ANNOTATION_GROUPS =
2022-09-25 05:00:26,515 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_USERNAME =
2022-09-25 05:00:26,516 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PASSWORD =
2022-09-25 05:00:26,516 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_PATH =
2022-09-25 05:00:26,517 - ENV SETTINGS:: MONAI_LABEL_QIDO_PREFIX =
2022-09-25 05:00:26,517 - ENV SETTINGS:: MONAI_LABEL_WADO_PREFIX =
2022-09-25 05:00:26,517 - ENV SETTINGS:: MONAI_LABEL_STOW_PREFIX =
2022-09-25 05:00:26,518 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_FETCH_BY_FRAME = False
2022-09-25 05:00:26,521 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_SEARCH_FILTER = '{"Modality": "CT"}'
2022-09-25 05:00:26,525 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_EXPIRY = 180
2022-09-25 05:00:26,525 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_READ_TIMEOUT = 5.0
2022-09-25 05:00:26,526 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_AUTO_RELOAD = True
2022-09-25 05:00:26,526 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_FILE_EXT = '["*.nii.gz", "*.nii", "*.nrrd", "*.jpg", "*.png", "*.tif", "*.svs", "*.xml"]'
2022-09-25 05:00:26,527 - ENV SETTINGS:: MONAI_LABEL_SERVER_PORT = 8000
2022-09-25 05:00:26,527 - ENV SETTINGS:: MONAI_LABEL_CORS_ORIGINS = '[]'
2022-09-25 05:00:26,528 - ENV SETTINGS:: MONAI_LABEL_SESSIONS = True
2022-09-25 05:00:26,528 - ENV SETTINGS:: MONAI_LABEL_SESSION_PATH =
2022-09-25 05:00:26,529 - ENV SETTINGS:: MONAI_LABEL_SESSION_EXPIRY = 3600
2022-09-25 05:00:26,529 - ENV SETTINGS:: MONAI_LABEL_INFER_CONCURRENCY = -1
2022-09-25 05:00:26,529 - ENV SETTINGS:: MONAI_LABEL_INFER_TIMEOUT = 600
2022-09-25 05:00:26,530 - ENV SETTINGS:: MONAI_LABEL_AUTO_UPDATE_SCORING = True
2022-09-25 05:00:26,530 -
Allow Origins: ['*']
[2022-09-25 05:00:27,327] [25360] [MainThread] [INFO] (uvicorn.error:75) - Started server process [25360]
[2022-09-25 05:00:27,328] [25360] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup.
[2022-09-25 05:00:27,328] [25360] [MainThread] [INFO] (monailabel.interfaces.utils.app:38) - Initializing App from: N:\ApPz\MONAI\apps\radiology; studies: N:\ApPz\MONAI\datasets\TMH\Liver; conf: {'models': 'segmentation'}
[2022-09-25 05:00:27,392] [25360] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2022-09-25 05:00:27,408] [25360] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2022-09-25 05:00:27,408] [25360] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2022-09-25 05:00:27,409] [25360] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2022-09-25 05:00:27,411] [25360] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2022-09-25 05:00:27,412] [25360] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2022-09-25 05:00:27,413] [25360] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2022-09-25 05:00:27,418] [25360] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2022-09-25 05:00:27,419] [25360] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2022-09-25 05:00:27,421] [25360] [MainThread] [INFO] (main:86) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2022-09-25 05:00:27,463] [25360] [MainThread] [INFO] (main:90) - +++ Using Models: ['segmentation']
[2022-09-25 05:00:27,463] [25360] [MainThread] [INFO] (monailabel.interfaces.app:128) - Init Datastore for: N:\ApPz\MONAI\datasets\TMH\Liver
[2022-09-25 05:00:27,465] [25360] [MainThread] [INFO] (monailabel.datastore.local:126) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2022-09-25 05:00:27,475] [25360] [MainThread] [INFO] (monailabel.datastore.local:540) - Invalidate count: 0
[2022-09-25 05:00:27,475] [25360] [MainThread] [INFO] (monailabel.datastore.local:146) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-09-25 05:00:27,478] [25360] [MainThread] [INFO] (main:116) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x000002792556EB80>
[2022-09-25 05:00:27,479] [25360] [MainThread] [INFO] (main:172) - {'segmentation': <lib.infers.segmentation.Segmentation object at 0x000002792556EB80>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x0000027926507400>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x00000279265073D0>}
[2022-09-25 05:00:27,479] [25360] [MainThread] [INFO] (main:185) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x0000027926507EE0>
[2022-09-25 05:00:27,481] [25360] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\AMiT\.cache\monailabel\sessions
[2022-09-25 05:00:27,482] [25360] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-09-25 05:00:27,482] [25360] [MainThread] [INFO] (monailabel.interfaces.app:465) - App Init - completed
[2022-09-25 05:00:27,483] [timeloop] [INFO] Starting Timeloop..
[2022-09-25 05:00:27,483] [25360] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2022-09-25 05:00:27,484] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x00000279255FB1F0>
[2022-09-25 05:00:27,484] [25360] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x00000279255FB1F0>
[2022-09-25 05:00:27,485] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2022-09-25 05:00:27,485] [25360] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2022-09-25 05:00:27,489] [25360] [MainThread] [INFO] (uvicorn.error:59) - Application startup complete.
[2022-09-25 05:00:27,490] [25360] [MainThread] [INFO] (uvicorn.error:206) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2022-09-25 05:02:10,277] [25360] [MainThread] [INFO] (monailabel.endpoints.datastore:100) - Saving Label for CV17056 for tag: final by admin
[2022-09-25 05:02:10,278] [25360] [MainThread] [INFO] (monailabel.endpoints.datastore:111) - Save Label params: {"label_info": [{"name": "Liver", "idx": 1}, {"name": "right kidney", "idx": 2}, {"name": "left kidney", "idx": 3}, {"name": "gallbladder", "idx": 4}, {"name": "esophagus", "idx": 5}, {"name": "stomach", "idx": 6}, {"name": "aorta", "idx": 7}, {"name": "inferior vena cava", "idx": 8}, {"name": "portal vein and splenic vein", "idx": 9}, {"name": "pancreas", "idx": 10}, {"name": "right adrenal gland", "idx": 11}, {"name": "left adrenal gland", "idx": 12}], "client_id": "user-xyz"}
[2022-09-25 05:02:10,279] [25360] [MainThread] [INFO] (monailabel.datastore.local:449) - Saving Label for Image: CV17056; Tag: final; Info: {'label_info': [{'name': 'Liver', 'idx': 1}, {'name': 'right kidney', 'idx': 2}, {'name': 'left kidney', 'idx': 3}, {'name': 'gallbladder', 'idx': 4}, {'name': 'esophagus', 'idx': 5}, {'name': 'stomach', 'idx': 6}, {'name': 'aorta', 'idx': 7}, {'name': 'inferior vena cava', 'idx': 8}, {'name': 'portal vein and splenic vein', 'idx': 9}, {'name': 'pancreas', 'idx': 10}, {'name': 'right adrenal gland', 'idx': 11}, {'name': 'left adrenal gland', 'idx': 12}], 'client_id': 'user-xyz'}
[2022-09-25 05:02:10,282] [25360] [MainThread] [INFO] (monailabel.datastore.local:457) - Adding Label: CV17056 => final => C:\Users\AMiT\AppData\Local\Temp\tmpo1idaxoj.nii.gz
[2022-09-25 05:02:10,289] [25360] [MainThread] [INFO] (monailabel.datastore.local:473) - Label Info: {'label_info': [{'name': 'Liver', 'idx': 1}, {'name': 'right kidney', 'idx': 2}, {'name': 'left kidney', 'idx': 3}, {'name': 'gallbladder', 'idx': 4}, {'name': 'esophagus', 'idx': 5}, {'name': 'stomach', 'idx': 6}, {'name': 'aorta', 'idx': 7}, {'name': 'inferior vena cava', 'idx': 8}, {'name': 'portal vein and splenic vein', 'idx': 9}, {'name': 'pancreas', 'idx': 10}, {'name': 'right adrenal gland', 'idx': 11}, {'name': 'left adrenal gland', 'idx': 12}], 'client_id': 'user-xyz', 'ts': 1664062330, 'checksum': 'SHA256:be4f651b196e6c8799deebee7718d2204c55ecd7af144ee7d11ce936e85c06ac', 'name': 'CV17056.nii.gz'}
[2022-09-25 05:02:10,291] [25360] [MainThread] [INFO] (monailabel.interfaces.app:489) - New label saved for: CV17056 => CV17056
[2022-09-25 05:02:20,220] [25360] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Train request: {'model': 'segmentation', 'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'client_id': 'user-xyz'}
[2022-09-25 05:02:20,222] [25360] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:59) - COMMAND:: C:\ProgramData\Anaconda3\envs\monailabel-env\python.exe -m monailabel.interfaces.utils.app -m train -r {"model":"segmentation","name":"train_01","pretrained":true,"device":"cuda","max_epochs":50,"early_stop_patience":-1,"val_split":0.2,"train_batch_size":1,"val_batch_size":1,"multi_gpu":true,"gpus":"all","dataset":"SmartCacheDataset","dataloader":"ThreadDataLoader","client_id":"user-xyz"}
[2022-09-25 05:02:20,328] [4396] [MainThread] [INFO] (__main__:38) - Initializing App from: N:\ApPz\MONAI\apps\radiology; studies: N:\ApPz\MONAI\datasets\TMH\Liver; conf: {'models': 'segmentation'}
[2022-09-25 05:02:22,635] [4396] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2022-09-25 05:02:22,645] [4396] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'>
[2022-09-25 05:02:22,645] [4396] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'>
[2022-09-25 05:02:22,645] [4396] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'>
[2022-09-25 05:02:22,646] [4396] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'>
[2022-09-25 05:02:22,646] [4396] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'>
[2022-09-25 05:02:22,647] [4396] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'>
[2022-09-25 05:02:22,647] [4396] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'>
[2022-09-25 05:02:22,648] [4396] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'>
[2022-09-25 05:02:22,648] [4396] [MainThread] [INFO] (main:86) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation
[2022-09-25 05:02:22,674] [4396] [MainThread] [INFO] (main:90) - +++ Using Models: ['segmentation']
[2022-09-25 05:02:22,674] [4396] [MainThread] [INFO] (monailabel.interfaces.app:128) - Init Datastore for: N:\ApPz\MONAI\datasets\TMH\Liver
[2022-09-25 05:02:22,674] [4396] [MainThread] [INFO] (monailabel.datastore.local:126) - Auto Reload: False; Extensions: ['*.nii.gz', '*.nii', '*.nrrd', '*.jpg', '*.png', '*.tif', '*.svs', '*.xml']
[2022-09-25 05:02:22,680] [4396] [MainThread] [INFO] (monailabel.datastore.local:540) - Invalidate count: 0
[2022-09-25 05:02:22,680] [4396] [MainThread] [INFO] (main:116) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x0000026767A08400>
[2022-09-25 05:02:22,680] [4396] [MainThread] [INFO] (main:172) - {'segmentation': <lib.infers.segmentation.Segmentation object at 0x0000026767A08400>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x0000026769D73E50>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x0000026769D73E20>}
[2022-09-25 05:02:22,681] [4396] [MainThread] [INFO] (main:185) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x0000026769D73EB0>
[2022-09-25 05:02:22,681] [4396] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\AMiT\.cache\monailabel\sessions
[2022-09-25 05:02:22,681] [4396] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-09-25 05:02:22,681] [4396] [MainThread] [INFO] (monailabel.tasks.train.basic_train:365) - Train Request (input): {'model': 'segmentation', 'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': True, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'client_id': 'user-xyz', 'local_rank': 0}
[2022-09-25 05:02:22,681] [4396] [MainThread] [INFO] (monailabel.tasks.train.basic_train:375) - CUDA_VISIBLE_DEVICES: None
[2022-09-25 05:02:22,681] [4396] [MainThread] [INFO] (monailabel.tasks.train.basic_train:380) - Distributed/Multi GPU is limited
[2022-09-25 05:02:22,681] [4396] [MainThread] [INFO] (monailabel.tasks.train.basic_train:395) - Distributed Training = FALSE
[2022-09-25 05:02:22,681] [4396] [MainThread] [INFO] (monailabel.tasks.train.basic_train:422) - 0 - Train Request (final): {'name': 'train_01', 'pretrained': True, 'device': 'cuda', 'max_epochs': 50, 'early_stop_patience': -1, 'val_split': 0.2, 'train_batch_size': 1, 'val_batch_size': 1, 'multi_gpu': False, 'gpus': 'all', 'dataset': 'SmartCacheDataset', 'dataloader': 'ThreadDataLoader', 'model': 'segmentation', 'client_id': 'user-xyz', 'local_rank': 0, 'run_id': '20220925_0502'}
[2022-09-25 05:02:22,681] [4396] [MainThread] [INFO] (monailabel.tasks.train.basic_train:521) - 0 - Using Device: cpu; IDX: None
[2022-09-25 05:02:22,681] [4396] [MainThread] [INFO] (monailabel.tasks.train.basic_train:343) - Total Records for Training: 1
[2022-09-25 05:02:22,681] [4396] [MainThread] [INFO] (monailabel.tasks.train.basic_train:344) - Total Records for Validation: 0
[2022-09-25 05:02:22,688] [4396] [MainThread] [INFO] (monailabel.tasks.train.basic_train:608) - 0 - Load Path N:\ApPz\MONAI\apps\radiology\model\pretrained_segmentation.pt
Loading dataset:   0%|          | 0/1 [00:00<?, ?it/s]
Loading dataset: 100%|##########| 1/1 [00:14<00:00, 14.51s/it]
Loading dataset: 100%|##########| 1/1 [00:14<00:00, 14.51s/it]
cache_num is greater or equal than dataset length, fall back to regular monai.data.CacheDataset.
[2022-09-25 05:02:37,202] [4396] [MainThread] [INFO] (monailabel.tasks.train.basic_train:228) - 0 - Records for Training: 1
torch.cuda.amp.GradScaler is enabled, but CUDA is not available.  Disabling.
[2022-09-25 05:02:37,205] [4396] [MainThread] [INFO] (ignite.engine.engine.SupervisedTrainer:876) - Engine run resuming from iteration 0, epoch 0 until 50 epochs
[2022-09-25 05:02:37,221] [4396] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:992) - Engine run is terminating due to exception: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
[2022-09-25 05:02:37,221] [4396] [MainThread] [ERROR] (ignite.engine.engine.SupervisedTrainer:178) - Exception: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 946, in _internal_run_as_gen
    self._fire_event(Events.STARTED)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\monai\handlers\checkpoint_loader.py", line 107, in __call__
    checkpoint = torch.load(self.load_path, map_location=self.map_location)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 712, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1049, in _load
    result = unpickler.load()
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1019, in persistent_load
    load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1001, in load_tensor
    wrap_storage=restore_location(storage, location),
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 175, in default_restore_location
    result = fn(storage, location)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 152, in _cuda_deserialize
    device = validate_cuda_device(location)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 136, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\monailabel\interfaces\utils\app.py", line 132, in <module>
    run_main()
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\monailabel\interfaces\utils\app.py", line 117, in run_main
    result = a.train(request)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\monailabel\interfaces\app.py", line 416, in train
    result = task(request, self.datastore())
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\monailabel\tasks\train\basic_train.py", line 396, in __call__
    res = self.train(0, world_size, req, datalist)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\monailabel\tasks\train\basic_train.py", line 458, in train
    context.trainer.run()
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\monai\engines\trainer.py", line 53, in run
    super().run()
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\monai\engines\workflow.py", line 278, in run
    super().run(data=self.data_loader, max_epochs=self.state.max_epochs)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 892, in run
    return self._internal_run()
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 935, in _internal_run
    return next(self._internal_run_generator)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 993, in _internal_run_as_gen
    self._handle_exception(e)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 636, in _handle_exception
    self._fire_event(Events.EXCEPTION_RAISED, e)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\monai\handlers\stats_handler.py", line 179, in exception_raised
    raise e
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 946, in _internal_run_as_gen
    self._fire_event(Events.STARTED)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\ignite\engine\engine.py", line 425, in _fire_event
    func(*first, *(event_args + others), **kwargs)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\monai\handlers\checkpoint_loader.py", line 107, in __call__
    checkpoint = torch.load(self.load_path, map_location=self.map_location)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 712, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1049, in _load
    result = unpickler.load()
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1019, in persistent_load
    load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 1001, in load_tensor
    wrap_storage=restore_location(storage, location),
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 175, in default_restore_location
    result = fn(storage, location)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 152, in _cuda_deserialize
    device = validate_cuda_device(location)
  File "C:\ProgramData\Anaconda3\envs\monailabel-env\lib\site-packages\torch\serialization.py", line 136, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
[2022-09-25 05:02:37,573] [25360] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:77) - Return code: 1

Look forward to guidance for working with monai.

Thanks again.

Best Regards, Amit.

Issue Analytics

State:
Created a year ago
Comments:22

Top GitHub Comments

2reactions

SachidanandAllecommented, Sep 28, 2022

If you are using slicer… enable developer model under Edit => Application Settings => Monai Label

After that you can see config options (which includes epochs for training…) you can change and trigger training (NOTE these are not persistent due to design limitations… when you refresh, they get reloaded from the server)

1reaction

diazandr3scommented, Sep 29, 2022

Hi @r00tdotexe,

the problem is not about using dicom or niftii. Rather, it is a problem with docker itself. When I run the monailabel image, it will only accessible within root user … somehow, I cannot work on the studies with slicer, since it is not on root. But I will figure it out.

ooh sorry, I misunderstood. I guess what you need is to mount a folder in the Docker container: https://www.howtogeek.com/devops/how-to-mount-a-host-directory-into-a-docker-container/

Another quesion: When I run training, will MONAI Label train the model on every label inside “final”?

Good question. The answer is yes, it’ll use all the labels available in the labels/final folder. By default 20% of those labels will be used for validation. You can change this percentage in the Options tab in the MONAI Label module.

And is there a GPU, used on MONAI server? Or is the local GPU used?

MONAI Label uses the GPU - this means the GPU should be available where the server is running

Hope that helps,

Top Results From Across the Web

The problem with the accuracy of training from scratch #922

Since my data belongs to PET CT images, it is not implemented in the current pre-training model, so I plan to train a...

How to start with monailabel for new models - Support

To train a new model first organize your data. Create a folder (e.g., myMonai) and put all your volumes in there. Within this...

STUDIES Path/Directory NOT provided, when trying to ...

Unable to start Monai Server, I get this error: Path Not Provided can ... STUDIES Path/Directory NOT provided, when trying to monailabel ......

monailabel.tasks.train.basic_train module - Project MONAI

monailabel.tasks.train.basic_train module¶ · model_dir – Base Model Dir to save the model checkpoints, events etc… · description – Description for this task.

Annotate DICOM images and build an ML model using the ...

A SageMaker notebook can retrieve the outputs of labeling jobs and use them to train a supervised ML model. We have built a...