[BUG] TypeError: argument of type 'ABCMeta' is not iterable

Describe the bug run deepspeed.init_inference and got TypeError: argument of type 'ABCMeta' is not iterable

To Reproduce Steps to reproduce the behavior:

from transformers import AutoModelForCausalLM
import torch
model_name = 'EleutherAI/gpt-neo-2.7B'
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.half).eval()
deepspeed.init_inference(model, dtype=torch.half, replace_method='auto')

output:

[2021-11-25 02:46:40,275] [INFO] [logging.py:69:log_dist] [Rank -1] DeepSpeed info: version=0.5.8+unknown, git-hash=unknown, git-branch=unknown
[2021-11-25 02:46:40,275] [INFO] [engine.py:127:_init_quantization_setting] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
Process stream_worker:
Traceback (most recent call last):
  File "/root/miniconda/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/root/miniconda/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/miniconda/lib/python3.9/site-packages/service_streamer/service_streamer.py", line 327, in run_forever
    self._model.init_model(*self._model_init_args, **self._model_init_kwargs)
  File "/root/algorithms-playground/nlp/sentence_generation/style_transfer.py", line 150, in init_model
    model, tokenizer, pipeline = load_model_on_device(self.gpu_id)
  File "/root/algorithms-playground/nlp/sentence_generation/style_transfer.py", line 127, in load_model_on_device
    deepspeed.init_inference(
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/__init__.py", line 274, in init_inference
    engine = InferenceEngine(model,
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/inference/engine.py", line 86, in __init__
    self._apply_injection_policy(
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/inference/engine.py", line 161, in _apply_injection_policy
    replace_transformer_layer(client_module,
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 464, in replace_transformer_layer
    return replace_module(model=model,
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 561, in replace_module
    replaced_module, _ = _replace_module(model, policy)
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 583, in _replace_module
    _, layer_id = _replace_module(child, policies, layer_id=layer_id)
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 583, in _replace_module
    _, layer_id = _replace_module(child, policies, layer_id=layer_id)
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 578, in _replace_module
    policies[child.__class__][0](child,
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 460, in replace_fn
    new_module = replace_wo_policy(child, _policy)
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 443, in replace_wo_policy
    return _replace_module(module)
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 440, in _replace_module
    _replace_module(child, name)
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 440, in _replace_module
    _replace_module(child, name)
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 435, in _replace_module
    linear_policies[child.__class__](child,
  File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 344, in _replace
    if name in all_reduce_linears:
TypeError: argument of type 'ABCMeta' is not iterable

Expected behavior A clear and concise description of what you expected to happen.

ds_report output

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [YES] ...... [OKAY]
cpu_adagrad ............ [YES] ...... [OKAY]
fused_adam ............. [YES] ...... [OKAY]
fused_lamb ............. [YES] ...... [OKAY]
sparse_attn ............ [YES] ...... [OKAY]
transformer ............ [YES] ...... [OKAY]
stochastic_transformer . [YES] ...... [OKAY]
async_io ............... [YES] ...... [OKAY]
transformer_inference .. [YES] ...... [OKAY]
utils .................. [YES] ...... [OKAY]
quantizer .............. [YES] ...... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/root/miniconda/lib/python3.9/site-packages/torch']
torch version .................... 1.10.0+cu113
torch cuda version ............... 11.3
nvcc version ..................... 11.0
deepspeed install path ........... ['/root/miniconda/lib/python3.9/site-packages/deepspeed']
deepspeed info ................... 0.5.8+unknown, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.10, cuda 11.3

System info (please complete the following information):

• OS: Ubuntu 18.04
• GPU count and types [one machines with x4 V100]
• Python version: 3.9
• Any other relevant info about your setup: I also tried compiled from source

[callzhang]

I solved it by adding: replace_with_kernel_inject=True Is this method recommended for GPT-Neo or not?

[RezaYazdaniAminabadi]

I just pushed a PR to fix this. Thanks, Reza