[BUG] TypeError: argument of type 'ABCMeta' is not iterable
See original GitHub issueDescribe the bug
run deepspeed.init_inference
and got TypeError: argument of type 'ABCMeta' is not iterable
To Reproduce Steps to reproduce the behavior:
from transformers import AutoModelForCausalLM
import torch
model_name = 'EleutherAI/gpt-neo-2.7B'
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.half).eval()
deepspeed.init_inference(model, dtype=torch.half, replace_method='auto')
output:
[2021-11-25 02:46:40,275] [INFO] [logging.py:69:log_dist] [Rank -1] DeepSpeed info: version=0.5.8+unknown, git-hash=unknown, git-branch=unknown
[2021-11-25 02:46:40,275] [INFO] [engine.py:127:_init_quantization_setting] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
Process stream_worker:
Traceback (most recent call last):
File "/root/miniconda/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda/lib/python3.9/site-packages/service_streamer/service_streamer.py", line 327, in run_forever
self._model.init_model(*self._model_init_args, **self._model_init_kwargs)
File "/root/algorithms-playground/nlp/sentence_generation/style_transfer.py", line 150, in init_model
model, tokenizer, pipeline = load_model_on_device(self.gpu_id)
File "/root/algorithms-playground/nlp/sentence_generation/style_transfer.py", line 127, in load_model_on_device
deepspeed.init_inference(
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/__init__.py", line 274, in init_inference
engine = InferenceEngine(model,
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/inference/engine.py", line 86, in __init__
self._apply_injection_policy(
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/inference/engine.py", line 161, in _apply_injection_policy
replace_transformer_layer(client_module,
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 464, in replace_transformer_layer
return replace_module(model=model,
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 561, in replace_module
replaced_module, _ = _replace_module(model, policy)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 583, in _replace_module
_, layer_id = _replace_module(child, policies, layer_id=layer_id)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 583, in _replace_module
_, layer_id = _replace_module(child, policies, layer_id=layer_id)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 578, in _replace_module
policies[child.__class__][0](child,
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 460, in replace_fn
new_module = replace_wo_policy(child, _policy)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 443, in replace_wo_policy
return _replace_module(module)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 440, in _replace_module
_replace_module(child, name)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 440, in _replace_module
_replace_module(child, name)
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 435, in _replace_module
linear_policies[child.__class__](child,
File "/root/miniconda/lib/python3.9/site-packages/deepspeed/module_inject/replace_module.py", line 344, in _replace
if name in all_reduce_linears:
TypeError: argument of type 'ABCMeta' is not iterable
Expected behavior A clear and concise description of what you expected to happen.
ds_report output
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [YES] ...... [OKAY]
cpu_adagrad ............ [YES] ...... [OKAY]
fused_adam ............. [YES] ...... [OKAY]
fused_lamb ............. [YES] ...... [OKAY]
sparse_attn ............ [YES] ...... [OKAY]
transformer ............ [YES] ...... [OKAY]
stochastic_transformer . [YES] ...... [OKAY]
async_io ............... [YES] ...... [OKAY]
transformer_inference .. [YES] ...... [OKAY]
utils .................. [YES] ...... [OKAY]
quantizer .............. [YES] ...... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/root/miniconda/lib/python3.9/site-packages/torch']
torch version .................... 1.10.0+cu113
torch cuda version ............... 11.3
nvcc version ..................... 11.0
deepspeed install path ........... ['/root/miniconda/lib/python3.9/site-packages/deepspeed']
deepspeed info ................... 0.5.8+unknown, unknown, unknown
deepspeed wheel compiled w. ...... torch 1.10, cuda 11.3
System info (please complete the following information):
- OS: Ubuntu 18.04
- GPU count and types [one machines with x4 V100]
- Python version: 3.9
- Any other relevant info about your setup: I also tried compiled from source
Issue Analytics
- State:
- Created 2 years ago
- Comments:5 (1 by maintainers)
Top Results From Across the Web
TypeError: argument of type 'int' is not iterable - Stack Overflow
Here c is the index not the list that you are searching. Since you cannot iterate through an integer, you are getting that...
Read more >'abcmeta' object is not iterable - You.com | The AI Search ...
Describe the bug run deepspeed.init_inference and got TypeError: argument of type 'ABCMeta' is not iterable To Reproduce Steps to reproduce the behavior: ...
Read more >python/typing - Gitter
from abc import ABCMeta, abstractmethod from typing import Tuple class ... has no attribute "__iter__" (not iterable) [attr-defined] pass class B(A): a =...
Read more >Input API for offline - RLlib - Ray
... when is pass "input": JsonReader(logdir-with-json-files) , I get the error TypeError: argument of type 'ABCMeta' is not iterable .
Read more >argument of type 'NoneType' is not iterable - Bugzilla@Mozilla
Closed Bug 930300 Opened 9 years ago Closed 8 years ago. [traceback] TypeError: argument of type 'NoneType' is not iterable.
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
I solved it by adding:
replace_with_kernel_inject=True
Is this method recommended for GPT-Neo or not?I just pushed a PR to fix this. Thanks, Reza