[BUG] GPT-Neo inference examples broken in Master

Describe the bug Get error AttributeError: module 'transformers.models' has no attribute 'opt'. when trying to run gpt-neo.py from DeepSpeedExamples using master build from DeepSpeed github.

GPT-neo example from this tutorial also fails with the same error https://www.deepspeed.ai/tutorials/inference-tutorial/

To Reproduce Steps to reproduce the behavior:

1. clone deepspeed from github
2. clone DeepSpeeedExamples from github
3. cd DeepSpeedExamples/inference/huggingface
4. python gpt-neo.py
5. results in the following error

***************** Creating model in RANK (0) with WORLD_SIZE = 1 *****************
[2022-08-22 18:06:41,060] [INFO] [logging.py:68:log_dist] [Rank -1] DeepSpeed info: version=0.7.1+a9b3bfa2, git-hash=a9b3bfa2, git-branch=master
[2022-08-22 18:06:41,060] [INFO] [logging.py:68:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
Traceback (most recent call last):
  File "/data/home/pacrook/DeepSpeed/examples/DeepSpeedExamples/inference/huggingface/gpt-neo.py", line 21, in <module>
    generator.model = deepspeed.init_inference(generator.model,
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/__init__.py", line 292, in init_inference
    engine = InferenceEngine(model,
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/inference/engine.py", line 140, in __init__
    self._apply_injection_policy(
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/inference/engine.py", line 333, in _apply_injection_policy
    replace_transformer_layer(
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 779, in replace_transformer_layer
    replaced_module = replace_module(model=model,
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 962, in replace_module
    _ = plcy(None)
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/module_inject/replace_policy.py", line 451, in __init__
    transformers.models.opt.configuration_opt.OPTConfig):
AttributeError: module 'transformers.models' has no attribute 'opt'. Did you mean: 'dpt'?

Expected behavior Expect gpt-neo to output inference result.

ds_report output

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
 [WARNING]  please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/torch']
torch version .................... 1.11.0
torch cuda version ............... 11.3
torch hip version ................ None
nvcc version ..................... 11.0
deepspeed install path ........... ['/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.7.1+a9b3bfa2, a9b3bfa2, master
deepspeed wheel compiled w. ...... torch 1.11, cuda 11.3

System info (please complete the following information):

• OS: Ubuntu 20.04.4 LTS
• GPU count and types: AWS, SLURM run, node with 4 x A100

[skiingpacman]

Can confirm that upgrading transformers from 4.18.0 to 4.21.1 resolves the issue for me.

@arashb thanks for looking into this so promptly.

[arashb]

@skiingpacman you can fix your issue by using the latest transformers package which supports OPT models. If you don’t wish to upgrade your “transformer” package, please wait until I send a PR to make the DSPolicy class backward compatible. I will send the PR by tomorrow the latest.