GPT-J-6B in run_clm.py

Environment info

• transformers version: 4.10.0.dev0
• Platform: Linux-4.19.0-10-cloud-amd64-x86_64-with-debian-10.5
• Python version: 3.7.8
• PyTorch version (GPU?): 1.7.1+cu110 (True)
• Tensorflow version (GPU?): 2.4.1 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: <fill in>
• Using distributed or parallel set-up in script?: <fill in>

Who can help

• text generation: @patrickvonplaten
• trainer: @sgugger
• pipelines: @LysandreJik

–>

Information

The model I am using GPT-J from HuggingFaceHub models, there is KeyError with this model, error listed below:

Traceback (most recent call last): File “run_clm.py”, line 522, in <module> main() File “run_clm.py”, line 320, in main config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs) File “/opt/conda/lib/python3.7/site-packages/transformers/models/auto/configuration_auto.py”, line 514, in from_pretrained config_class = CONFIG_MAPPING[config_dict[“model_type”]] File “/opt/conda/lib/python3.7/site-packages/transformers/models/auto/configuration_auto.py”, line 263, in getitem raise KeyError(key) KeyError: ‘gptj’

[sgugger]

You are trying to use the Adam optimizer with a model of 24Gb. With Adam, you have four copies of your model: model, gradients, and in the optimizer state the gradients averaged and square averaged. Even with fp16, all of that is still stored in FP32 because of mixed precision training (the optimzier update is in full precision). So unless you use DeepSpeed to offload the optimizer state and the gradient copy in FP32, you won’t be able to fit this 4 x 24GB on your 80GB card.

[dimaischenko]

@LysandreJik I agree with you. I think that’s the problem. @johndpope Yes 80gb ram was enough. To be honest, I don’t remember the details anymore, but it seems that it took even less with DeepSpeed.