Suggestion: Support CPU-only environments/remove CUDA requirement

Hi,

I’m training neural networks on a small CPU-only workstation, and I wanted to use the DeepSpeedCPUAdam optimizer to accelerate the training, but noticed that CUDA is required:

 File "...python3.9/site-packages/deepspeed/ops/op_builder/builder.py", line 37, in installed_cuda_version
    assert cuda_home is not None, "CUDA_HOME does not exist, unable to compile CUDA op(s)"

Making CUDA optional would be a great feature. In my own field (climate modeling and weather forecasting), machine learning is an increasingly popular topic but GPU’s and deep learning are often overkill. (Right now, I’m training neural nets with 1-2 hidden layers and 10-20 neurons, which is quite efficient on CPU’s). Furthermore, support for non-NVIDIA GPU’s should be relevant in the longer term.

I realize this is presented as a deep learning library, but dropping the CUDA requirement for features which do not really require it might still be useful. Thanks!

[Yeqishen]

hello, I have the same problem now. Is there a solution?

[pme0]

Is there any update on this ? I am getting the same error when using FusedAdam !

I’m also having this issue on deepspeed==0.5.10. My model trains fine with an optimizer from torch.optim but when using deepspeed.ops.adam.FusedAdam I get:

File "[...]/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 38, in installed_cuda_version
    assert cuda_home is not None, "CUDA_HOME does not exist, unable to compile CUDA op(s)"
AssertionError: CUDA_HOME does not exist, unable to compile CUDA op(s)