13 minutes to load next image

Describe the bug The time between clicking “Next Sample” and it being rendered to screen with inferred segmentation was 13 minutes. I’m running the server on a remote machine with 24GB GPU, connected locally via SSH.

1. I expected some lag because of data transfer over SSH. It would be useful for this to be logged in the debug output if possible.
2. Some transforms seem to take far longer than normal (e.g., post AsDiscreted took 17s and inverse CropForegroundd took 66s)
3. Without timestamps for all events, it’s hard to figure out where the bottleneck is. Although there was TimeTotal: 189.1067; Pre: 8.2437; Inferer: 25.0135; Invert: 114.6922; Post: 20.1229; Write: 21.0341, there was a further 10 minute wait after the last time stamp.

I’m attaching the relevant section of the log output, I would be grateful if you could point me in the right direction.

Log output

- [2022-02-03 17:01:21,010] [3286715] [MainThread] [INFO] (monailabel.endpoints.activelearning:42) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'}
- [2022-02-03 17:01:21,024] [3286715] [MainThread] [INFO] (monailabel.tasks.activelearning.random:44) - Random: Images: ['sub-verse006_ct']; Weight: [92248]
- [2022-02-03 17:01:21,025] [3286715] [MainThread] [INFO] (monailabel.tasks.activelearning.random:45) - Random: Selected Image: sub-verse006_ct
- [2022-02-03 17:01:21,026] [3286715] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Scoring request: {'method': 'dice'}
- [2022-02-03 17:01:21,034] [3286715] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:57) - COMMAND:: /mnt/data/rbrown/Documents/Code/miniconda3/envs/monai/bin/python -m monailabel.interfaces.utils.app -m scoring -r {"method":"dice","gpus":"all"}
- [2022-02-03 17:01:21,037] [3286715] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Scoring request: {'method': 'sum'}
- [2022-02-03 17:01:21,208] [3286715] [MainThread] [INFO] (monailabel.endpoints.activelearning:58) - Next sample: {'id': 'sub-verse006_ct', 'ts': 1643806105, 'checksum': 'SHA256:d5d08401724139f9c4dcd239962db91eeeb5593ec130783063b03fb2364118ae', 'name': 'sub-verse006_ct.nii.gz', 'strategy': {'random': {'ts': 1643907681, 'client_id': 'user-xyz'}}, 'path': '/mnt/data/mmodat/Spine/VerSe_MONAILabel_3ims/sub-verse006_ct.nii.gz'}
- [2022-02-03 17:01:25,685] [3297053] [MainThread] [INFO] (__main__:38) - Initializing App from: /mnt/data/rbrown/Documents/Code/MONAILabel/sample-apps/deepedit_multilabel_VerSe; studies: /mnt/data/mmodat/Spine/VerSe_MONAILabel_3ims; conf: {}
- [2022-02-03 17:01:27,326] [3286715] [MainThread] [INFO] (monailabel.endpoints.infer:159) - Infer Request: {'model': 'deepedit_seg', 'image': 'sub-verse006_ct', 'client_id': 'user-xyz'}
- [2022-02-03 17:01:27,327] [3286715] [MainThread] [INFO] (monailabel.interfaces.app:230) - Image => /mnt/data/mmodat/Spine/VerSe_MONAILabel_3ims/sub-verse006_ct.nii.gz
- [2022-02-03 17:01:27,327] [3286715] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:221) - Infer Request (final): {'model': 'deepedit_seg', 'image': '/mnt/data/mmodat/Spine/VerSe_MONAILabel_3ims/sub-verse006_ct.nii.gz', 'client_id': 'user-xyz'}
- [2022-02-03 17:01:27,340] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:61) - PRE - Run Transform
- [2022-02-03 17:01:27,340] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:62) - PRE - Input Keys: dict_keys(['model', 'image', 'client_id', 'image_path'])
- [2022-02-03 17:01:32,131] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (LoadImaged): Time: 4.7789; image: (915, 1189, 121)(float32)
- [2022-02-03 17:01:32,131] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (AddChanneld): Time: 0.0001; image: (1, 915, 1189, 121)(float32)
- [2022-02-03 17:01:32,398] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (CropForegroundd): Time: 0.2669; image: (1, 390, 1188, 121)(float32)
- [2022-02-03 17:01:34,789] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (VarSpacingd): Time: 2.3899; image: (1, 151, 458, 139)(float32)
- [2022-02-03 17:01:34,865] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (ScaleIntensityd): Time: 0.0740; image: (1, 151, 458, 139)(float32)
- [2022-02-03 17:01:34,893] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (Orientationd): Time: 0.0270; image: (1, 458, 151, 139)(float32)
- [2022-02-03 17:01:34,917] [3297053] [MainThread] [INFO] (numexpr.utils:147) - Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
- [2022-02-03 17:01:34,918] [3297053] [MainThread] [INFO] (numexpr.utils:159) - NumExpr defaulting to 8 threads.
- [2022-02-03 17:01:34,927] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (DivisiblePadd): Time: 0.0332; image: (1, 464, 160, 144)(float32)
- [2022-02-03 17:01:35,568] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (DiscardAddGuidanced): Time: 0.6409; image: (30, 464, 160, 144)(float32)
- [2022-02-03 17:01:35,571] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (ToTensord): Time: 0.0021; image: torch.Size([30, 464, 160, 144])(torch.float32)
- [2022-02-03 17:01:35,571] [3286715] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:355) - Running Inferer:: SimpleInferer
- [2022-02-03 17:01:35,573] [3286715] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:297) - Infer model path: /mnt/data/mmodat/Spine/RB_VerSe_UNet_final.pth
- [2022-02-03 17:01:58,081] [3297053] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
- [2022-02-03 17:01:58,228] [3297053] [MainThread] [INFO] (main:96) - Working with Network UNet
- [2022-02-03 17:01:58,229] [3297053] [MainThread] [INFO] (main:119) - EPISTEMIC Enabled: 0; Samples: 5
- [2022-02-03 17:01:58,229] [3297053] [MainThread] [INFO] (main:123) - TTA Enabled: 0; Samples: 5
- [2022-02-03 17:01:58,229] [3297053] [MainThread] [INFO] (monailabel.interfaces.app:114) - Init Datastore for: /mnt/data/mmodat/Spine/VerSe_MONAILabel_3ims
- [2022-02-03 17:01:58,230] [3297053] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
- [2022-02-03 17:01:58,252] [3297053] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
- [2022-02-03 17:01:58,252] [3297053] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
- [2022-02-03 17:01:58,298] [3297053] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /mnt/data/rbrown/.cache/monailabel/sessions
- [2022-02-03 17:01:58,299] [3297053] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
- [2022-02-03 17:01:58,300] [3297053] [MainThread] [INFO] (__main__:62) - Result: {}
- [2022-02-03 17:02:01,357] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:61) - INV - Run Transform
- [2022-02-03 17:02:01,357] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:62) - INV - Input Keys: dict_keys(['model', 'image', 'client_id', 'image_path', 'image_meta_dict', 'foreground_start_coord', 'foreground_end_coord', 'image_transforms', 'pred'])
- [2022-02-03 17:02:02,073] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - INV - Transform (ToTensord): Time: 0.7146; image: (29, 464, 160, 144)(float32); pred: torch.Size([29, 464, 160, 144])(torch.float32)
- [2022-02-03 17:02:02,585] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - INV - Transform (DivisiblePadd): Time: 0.5117; image: (29, 458, 151, 139)(float32); pred: torch.Size([29, 464, 160, 144])(torch.float32)
- [2022-02-03 17:02:03,479] [3286715] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:75) - Return code: 0
- [2022-02-03 17:02:03,482] [3286715] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:57) - COMMAND:: /mnt/data/rbrown/Documents/Code/miniconda3/envs/monai/bin/python -m monailabel.interfaces.utils.app -m scoring -r {"method":"sum","gpus":"all"}
- [2022-02-03 17:02:04,933] [3297579] [MainThread] [INFO] (__main__:38) - Initializing App from: /mnt/data/rbrown/Documents/Code/MONAILabel/sample-apps/deepedit_multilabel_VerSe; studies: /mnt/data/mmodat/Spine/VerSe_MONAILabel_3ims; conf: {}
- [2022-02-03 17:02:06,512] [3297579] [MainThread] [INFO] (numexpr.utils:147) - Note: NumExpr detected 32 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
- [2022-02-03 17:02:06,513] [3297579] [MainThread] [INFO] (numexpr.utils:159) - NumExpr defaulting to 8 threads.
- [2022-02-03 17:02:06,739] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - INV - Transform (Orientationd): Time: 4.1532; image: (29, 151, 458, 139)(float32); pred: torch.Size([29, 464, 160, 144])(torch.float32)
- [2022-02-03 17:02:13,578] [3297579] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
- [2022-02-03 17:02:13,750] [3297579] [MainThread] [INFO] (main:96) - Working with Network UNet
- [2022-02-03 17:02:13,750] [3297579] [MainThread] [INFO] (main:119) - EPISTEMIC Enabled: 0; Samples: 5
- [2022-02-03 17:02:13,750] [3297579] [MainThread] [INFO] (main:123) - TTA Enabled: 0; Samples: 5
- [2022-02-03 17:02:13,751] [3297579] [MainThread] [INFO] (monailabel.interfaces.app:114) - Init Datastore for: /mnt/data/mmodat/Spine/VerSe_MONAILabel_3ims
- [2022-02-03 17:02:13,751] [3297579] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
- [2022-02-03 17:02:13,758] [3297579] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
- [2022-02-03 17:02:13,758] [3297579] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
- [2022-02-03 17:02:13,769] [3297579] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /mnt/data/rbrown/.cache/monailabel/sessions
- [2022-02-03 17:02:13,769] [3297579] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
- [2022-02-03 17:02:18,709] [3297579] [MainThread] [INFO] (__main__:62) - Result: {"sub-verse006_ct": {"sum": 723510272, "slices": 390}, "sub-verse005_ct": {"sum": 6887338, "slices": 103}, "sub-verse004_ct": {"sum": 4185567, "slices": 107}}
- [2022-02-03 17:02:20,307] [3286715] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:75) - Return code: 0
- [2022-02-03 17:02:48,950] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - INV - Transform (VarSpacingd): Time: 42.2099; image: (29, 390, 1188, 121)(float32); pred: torch.Size([29, 464, 160, 144])(torch.float32)
- [2022-02-03 17:03:55,276] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - INV - Transform (CropForegroundd): Time: 66.3255; image: (29, 915, 1189, 121)(float32); pred: torch.Size([29, 464, 160, 144])(torch.float32)
- [2022-02-03 17:03:55,284] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:61) - POST - Run Transform
- [2022-02-03 17:03:55,285] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:62) - POST - Input Keys: dict_keys(['model', 'image', 'client_id', 'image_path', 'image_meta_dict', 'foreground_start_coord', 'foreground_end_coord', 'image_transforms', 'pred'])
- [2022-02-03 17:03:55,286] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - POST - Transform (ToTensord): Time: 0.0002; image: torch.Size([30, 464, 160, 144])(torch.float32); pred: torch.Size([29, 915, 1189, 121])(torch.float32)
- [2022-02-03 17:03:57,778] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - POST - Transform (Activationsd): Time: 2.4921; image: torch.Size([30, 464, 160, 144])(torch.float32); pred: torch.Size([29, 915, 1189, 121])(torch.float32)
- [2022-02-03 17:04:15,119] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - POST - Transform (AsDiscreted): Time: 17.3401; image: torch.Size([30, 464, 160, 144])(torch.float32); pred: torch.Size([1, 915, 1189, 121])(torch.float32)
- [2022-02-03 17:04:15,120] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - POST - Transform (SqueezeDimd): Time: 0.0004; image: torch.Size([30, 464, 160, 144])(torch.float32); pred: torch.Size([915, 1189, 121])(torch.float32)
- [2022-02-03 17:04:15,120] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - POST - Transform (ToNumpyd): Time: 0.0001; image: torch.Size([30, 464, 160, 144])(torch.float32); pred: (915, 1189, 121)(float32)
- [2022-02-03 17:04:15,121] [3286715] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - POST - Transform (Restored): Time: 0.0002; image: torch.Size([30, 464, 160, 144])(torch.float32); pred: (915, 1189, 121)(float32)
- [2022-02-03 17:04:15,400] [3286715] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:390) - Writing Result
- [2022-02-03 17:04:15,402] [3286715] [MainThread] [INFO] (monailabel.transform.writer:88) - Result ext: .nii.gz
- [2022-02-03 17:04:36,434] [3286715] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:249) - ++ Latencies => Total: 189.1067; Pre: 8.2437; Inferer: 25.0135; Invert: 114.6922; Post: 20.1229; Write: 21.0341
- [2022-02-03 17:04:36,434] [3286715] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:263) - Result File: /tmp/tmpos10l3wh.nii.gz
- [2022-02-03 17:04:36,434] [3286715] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:264) - Result Json: {'label_names': {'Ver: 1': 1, 'Ver: 2': 2, 'Ver: 3': 3, 'Ver: 4': 4, 'Ver: 5': 5, 'Ver: 6': 6, 'Ver: 7': 7, 'Ver: 8': 8, 'Ver: 9': 9, 'Ver: 10': 10, 'Ver: 11': 11, 'Ver: 12': 12, 'Ver: 13': 13, 'Ver: 14': 14, 'Ver: 15': 15, 'Ver: 16': 16, 'Ver: 17': 17, 'Ver: 18': 18, 'Ver: 19': 19, 'Ver: 20': 20, 'Ver: 21': 21, 'Ver: 22': 22, 'Ver: 23': 23, 'Ver: 24': 24, 'Ver: 25': 25, 'Ver: 26': 26, 'Ver: 27': 27, 'Ver: 28': 28, 'background': 0}}
- [2022-02-03 17:04:36,459] [3286715] [MainThread] [INFO] (monailabel.datastore.local:443) - Saving Label for Image: sub-verse006_ct; Tag: original; Info: {'label_names': {'Ver: 1': 1, 'Ver: 2': 2, 'Ver: 3': 3, 'Ver: 4': 4, 'Ver: 5': 5, 'Ver: 6': 6, 'Ver: 7': 7, 'Ver: 8': 8, 'Ver: 9': 9, 'Ver: 10': 10, 'Ver: 11': 11, 'Ver: 12': 12, 'Ver: 13': 13, 'Ver: 14': 14, 'Ver: 15': 15, 'Ver: 16': 16, 'Ver: 17': 17, 'Ver: 18': 18, 'Ver: 19': 19, 'Ver: 20': 20, 'Ver: 21': 21, 'Ver: 22': 22, 'Ver: 23': 23, 'Ver: 24': 24, 'Ver: 25': 25, 'Ver: 26': 26, 'Ver: 27': 27, 'Ver: 28': 28, 'background': 0}}
- [2022-02-03 17:04:36,460] [3286715] [MainThread] [INFO] (monailabel.datastore.local:451) - Adding Label: sub-verse006_ct => original => /tmp/tmpos10l3wh.nii.gz
- [2022-02-03 17:04:36,911] [3286715] [MainThread] [INFO] (monailabel.datastore.local:467) - Label Info: {'label_names': {'Ver: 1': 1, 'Ver: 2': 2, 'Ver: 3': 3, 'Ver: 4': 4, 'Ver: 5': 5, 'Ver: 6': 6, 'Ver: 7': 7, 'Ver: 8': 8, 'Ver: 9': 9, 'Ver: 10': 10, 'Ver: 11': 11, 'Ver: 12': 12, 'Ver: 13': 13, 'Ver: 14': 14, 'Ver: 15': 15, 'Ver: 16': 16, 'Ver: 17': 17, 'Ver: 18': 18, 'Ver: 19': 19, 'Ver: 20': 20, 'Ver: 21': 21, 'Ver: 22': 22, 'Ver: 23': 23, 'Ver: 24': 24, 'Ver: 25': 25, 'Ver: 26': 26, 'Ver: 27': 27, 'Ver: 28': 28, 'background': 0}, 'ts': 1643907876, 'checksum': 'SHA256:b003daedf8bb63cae6c0a77076d1d7fc91183ebb12d0d03b1006fa5346733449', 'name': 'sub-verse006_ct.nii.gz'}
- No timestamp for successfully loading image, but this was 17:14

[SachidanandAlle]

closing this issue… as several possible improvements (running post transform in gpu) were suggested above when running very large image + multi label

[lassoan]

I guess you mean bitfield (each bit in a voxel indicates if that voxel belongs to the corresponding segment), which we briefly experimented many years ago and we ran into two very serious issues:

1. None of the existing processing and visualization methods work on bitfield images: you need to implement and maintain new ones, which is a lot of effort; and they are very slow, because to get or set a value you need to keep using multiple operations instead of just one.
2. Does not scale well with the number of segments (in atlases or radiation therapy plans you often have 100+ segments), which would mean that you need to process 12+ bytes for modifying a single voxel of a single segment (which is very slow), and it also requires lots of memory.

If you don’t mean bitfield but that each voxel would actually contain a pointer to a list object then that would solve the second issue. However, the processing overhead, memory locality, memory fragmentation issues would become so much worse that it would not be acceptable for most applications.