Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Refactor Pytorch `model.generate` method to work on TPU
See original GitHub issueFeature request
Refactor PT version of the method model.generate for text generating models to make it compatible with XLA and speed up inference on TPU.
Motivation
Right now, model.generate on PT is extremely slow on TPU compared to CPU and GPU. This is probably due to the fact that some operations done in the PT version of model.generate are not XLA compatible, and thus the generation process falls back on CPU. This makes inference on TPU infeasible. A major refactoring work has already been done on its TF counterpart, so it would be nice to have the PT version working as well.
A more in-depth discussion with @gante took place in #12322 and on this huggingface discussion.
Your contribution
If there is some interest from the HF team, I can definitely assist during the work.
Issue Analytics
- State:
- Created a year ago
- Reactions:4
- Comments:9 (5 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Added to my
generatetask queue 👍@divyanshuaggarwal it would be part of
transformers!This is not a prioritized feature as you can already use TPUs for generation in Flax and TensorFlow. Since you can easily convert a model from one framework to the other, there is an easy workaround 😃