ImportError: cannot import name 'RobertaLMHeadModel'

Hi all, I was just trying to run a text generation script for low-resource languages, and therefore experimented with XLM-R and initially with Roberta, using the documentation for RobertaForCausalLM: here: https://huggingface.co/transformers/model_doc/roberta.html#robertaforcausallm

I am running into the import error shown in the title. See code snippet and error message below. I also experimented with different Tensorflow and transformers versions to no avail. I suspect that the model classes have changed (or the documentation may not be up to date with the current version). I also tried importing RobertaForCausalLM but it returned the same error.

Environment info

• transformers version: 3.1.0
• Platform: Linux-4.15.0-112-generic-x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.6.9
• Tensorflow version: 2.3.1

Who can help

@LysandreJik , @TevenLeScao

Model I am using (Roberta, XLM-Roberta):

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1. Run:

from transformers import RobertaTokenizer, RobertaLMHeadModel, RobertaConfig
import torch
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
config = RobertaConfig.from_pretrained("roberta-base")
config.is_decoder = True
model = RobertaLMHeadModel.from_pretrained('roberta-base', config=config, return_dict=True)
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)
prediction_logits = outputs.logits

Error: ImportError: cannot import name 'RobertaLMHeadModel'

If this script runs succesfully, I 'd like to re-run it for XMLRoberta (changing the imports and model names of course). Many thanks!

[LysandreJik]

Indeed, there’s an issue with the docstrings here. I’m fixing it in #7642.

Have you taken a look at the summary of text generation here?

Please note that RoBERTa has not been trained to do text generation, but to do mask in-filling, so using a pre-trained RoBERTa model to do generation would yield bad results.

[julien-c]

@lighteternal Have you checked whether this model is any good for your use case? https://huggingface.co/nikokons/gpt2-greek?text=Σήμερα+ο+καιρός