Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Adding accelerate to `transformer` mdoels

See original GitHub issue

Is there a guide to assign accelerate support to models that are already implemented in the transformers library?

Issue Analytics

State:
Created a year ago
Comments:6 (1 by maintainers)

Top GitHub Comments

1reaction

StellaAthenacommented, May 27, 2022

Hmmm I think I miscommunicated. I would like to add support for accelerate to a model on the hub (specifically, GPT-NeoX and GPT-J) that doesn’t currently have it. When I try to run the models with accelerate it says

Traceback (most recent call last):
  File "main.py", line 142, in <module>
    main()
  File "main.py", line 109, in main
    results = evaluator.simple_evaluate(
  File "/home/mchorse/lm-evaluation-harness/lm_eval/utils.py", line 228, in _wrapper
    return fn(*args, **kwargs)
  File "/home/mchorse/lm-evaluation-harness/lm_eval/evaluator.py", line 70, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
  File "/home/mchorse/lm-evaluation-harness/lm_eval/base.py", line 119, in create_from_arg_string
    return cls(**args, **args2)
  File "/home/mchorse/lm-evaluation-harness/lm_eval/models/huggingface.py", line 35, in __init__
    self.model = self.create_auto_model(pretrained, revision, subfolder)
  File "/home/mchorse/lm-evaluation-harness/lm_eval/models/huggingface.py", line 56, in create_auto_model
    return self.AUTO_MODEL_CLASS.from_pretrained(
  File "/home/mchorse/.local/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 446, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/home/mchorse/.local/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2124, in from_pretrained
    raise ValueError(f"{model.__class__.__name__} does not support `device_map='auto'` yet.")
ValueError: GPTNeoXForCausalLM does not support `device_map='auto'` yet.

0reactions

github-actions[bot]commented, Jun 26, 2022

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

Top Results From Across the Web

Distributed training with Accelerate - Hugging Face

At Hugging Face, we created the Accelerate library to help users easily train a Transformers model on any type of distributed setup, whether...

Accelerate Transformer Model Training with Hugging Face ...

Transformer models deliver state-of-the-art performance on a wide range of machine learning tasks, such as natural language processing, ...

Hardware Accelerator for Multi-Head Attention and Position ...

Therefore, this work lays a good foundation for building efficient hardware accelerators for multiple Transformer networks.

Accelerating Transformer-based Deep Learning Models on ...

This paper investigates the column balanced block-wise pruning on Transformer and designs an FPGA acceleration engine to customize the balanced blockwise matrix ...

DOTA: Detect and Omit Weak Attentions for Scalable ...

end-to-end Transformer acceleration using the proposed attention detection mechanism. ... Instead, DOTA provides an efficient abstraction of the model and.