Error while using Apex

Hi, I am trying to do mixed precision training, but I have countered a problem that seems to be related to the LayerNorm implementation of Apex. I have the following error msg while running the example (same error for my other code).

Traceback (most recent call last): File "run_lm_finetuning.py", line 648, in <module> main() File "run_lm_finetuning.py", line 529, in main model = BertForPreTraining.from_pretrained(args.bert_model) File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 506, in from_pretrained model = cls(config, *inputs, **kwargs) File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 689, in __init__ self.bert = BertModel(config) File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 600, in __init__ self.embeddings = BertEmbeddings(config) File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 183, in __init__ self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12) File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/site-packages/apex-0.1-py3.6.egg/apex/normalization/fused_layer_norm.py", line 126, in __init__ File "/home/chenyang/anaconda3/envs/pytorch10/lib/python3.6/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 994, in _gcd_import File "<frozen importlib._bootstrap>", line 971, in _find_and_load File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked ModuleNotFoundError: No module named 'fused_layer_norm_cuda'

I am wondering if it is related to the version of Apex, so may I know which Apex checkpoint you used.

[thomwolf]

Hi @chenyangh, You need to install apex with the C++ and CUDA extensions:

git clone https://github.com/NVIDIA/apex.git
cd apex
python setup.py install --cuda_ext --cpp_ext

[duxie]

@kbulutozler You can change pytorch docker image version to pytorch/pytorch:1.3-cuda10.1-cudnn7-devel