Add option to set determinism with bundle configs
See original GitHub issueIs your feature request related to a problem? Please describe. There should be a configuration option to enable deterministic training with monai bundle. Currently, a user has to add python code in the “training” portion of the config:
"training": [
"$monai.utils.set_determinism(seed=123)",
"$setattr(torch.backends.cudnn, 'benchmark', True)",
"$@train#trainer.run()"
]
This can be difficult to use in security contrainted environments such as in federated learning where one might want to disable the python code execution in the bundle.
Describe the solution you’d like There should be an option to set the deterministic behavior using configuration style options, e.g., the below to achieve the same behavior as above:
"determinism": {
"random_seed": 123,
"benchmark": True
}
Describe alternatives you’ve considered MonaiAlgo for executing FL training does not run the “training” section in the config. Therefore the code there is not being executed. One can achieve determinism by using the “required” option for the SupervisedTrainer but this will not work if python code is disabled.
Additional context Especially important feature to support FL.
Issue Analytics
- State:
- Created a year ago
- Comments:8 (6 by maintainers)
Top GitHub Comments
maybe one valid feature request is to create two types of permissions for the
run_eval
flag, https://github.com/Project-MONAI/MONAI/blob/1941283b1862e3f578ccfeb2bdfa864171125029/monai/bundle/config_item.py#L318so that when option 1 is disabled at a system level, the user can still have the flexibility to use the 2nd option to run some predefined commonly used commands.
Yes, perhaps we have this hardcoded in monaialgo for FL, and later extending this to the primary bundle parser if we receive feature requests? cc @Nic-Ma