Cannot load studio-ousia/luke-base for AutoModelForTokenClassification

Environment info

• transformers version: 4.6.0.dev0 (pulled from repo)
• Platform: 3
• Python version: 3.7.10
• PyTorch version (GPU?): 1.7.0 (no)
• Tensorflow version (GPU?): 2.4.1 (no)
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help

Information

I tried loading LUKE’s weight for AutoModelForTokenClassification. I intend to train further for NER. It failed due to a configuration error.

To reproduce

Steps to reproduce the behavior:

from transformers import AutoModelForTokenClassification, TrainingArguments, Trainer

from transformers import AutoTokenizer, AutoModel
  
tokenizer = AutoTokenizer.from_pretrained("studio-ousia/luke-base") #Succesful

model = AutoModel.from_pretrained("studio-ousia/luke-base") #Succesful

model = AutoModelForTokenClassification.from_pretrained("studio-ousia/luke-base", num_labels=39)

Some weights of the model checkpoint at studio-ousia/luke-base were not used when initializing LukeModel: ['embeddings.position_ids']
- This IS expected if you are initializing LukeModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing LukeModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-b4efbc1b7796> in <module>
      7 model = AutoModel.from_pretrained("studio-ousia/luke-base")
      8 
----> 9 model = AutoModelForTokenClassification.from_pretrained("studio-ousia/luke-base", num_labels=39)

/opt/conda/lib/python3.7/site-packages/transformers/models/auto/auto_factory.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    381             return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
    382         raise ValueError(
--> 383             f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"
    384             f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}."
    385         )

ValueError: Unrecognized configuration class <class 'transformers.models.luke.configuration_luke.LukeConfig'> for this kind of AutoModel: AutoModelForTokenClassification.
Model type should be one of BigBirdConfig, ConvBertConfig, LayoutLMConfig, DistilBertConfig, CamembertConfig, FlaubertConfig, XLMConfig, XLMRobertaConfig, LongformerConfig, RobertaConfig, SqueezeBertConfig, BertConfig, MegatronBertConfig, MobileBertConfig, XLNetConfig, AlbertConfig, ElectraConfig, FunnelConfig, MPNetConfig, DebertaConfig, DebertaV2Config, IBertConfig.

Expected behavior

Succesful loading

[patil-suraj]

cc @NielsRogge

[ghost]

Hi, @Sreyan88 Were you able to train LUKE on a custom dataset? I am also working on the same and do not have any progress on this yet. Any help is appreciated. Thanks!