Improvement to error handling in subclasses
See original GitHub issueFeature request
I encountered a fascinating (though very frustrating 😃) scenario about error handling in transformers
. When subclassing a tokenizer that relies on sentencepiece
, and not having it installed, you will get an unhelpful error message that sends you down a lot of rabbit holes.
Consider this minimal example:
from transformers import MBartTokenizer
class CustomMBartTokenizer(MBartTokenizer):
@classmethod
def from_pretrained(cls, *args, **kwargs):
inst = super().from_pretrained(*args, **kwargs)
# Do other stuff with it...
a = CustomMBartTokenizer.from_pretrained("facebook/mbart-large-cc25")
If you run this in a new environment where sentencepiece is not installed, you get the following error:
AttributeError: ‘super’ object has no attribute ‘from_existing’
This error message had me comparing Windows vs. Linux and python 3.8 vs 3.9 vs 3.10 because I could not figure out why it was working on my home machine and not on our cluster. In the end, the reason was that sentencepiece
was not yet installed on the cluster but the error message does not show that. It seems that the sentencepiece error does not show or does not stop execution, which then leads the class to not be successfully initialized. Although admittedly I have not dug much farther.
Motivation
The error message does not seem to correctly propagate when subclassing a tokenizer. The error message that indicates that sentencepiece is not installed and needs to be installed is not correctly shown. Instead the user gets a vague error message about the from_pretrained
call.
While this may be an exceptional case, I have found that subclassing tokenizers for a specific task is common in research.
Your contribution
I do not have the time to work on figuring out what the exact cause is unfortunately. Posting this here for posterity. For anyone getting this issue: you probably just need to make sure all necessary third party libraries (such as sentencepiece) are installed.
Issue Analytics
- State:
- Created 10 months ago
- Reactions:1
- Comments:9 (3 by maintainers)
I think
DummyObject
should override__getattribute__
instead of__getattr__
to get the expected error.EDIT:
Tested locally, and it works. I’ve linked a PR with the fix.
Thanks! - will update the requirements with that