LXMERT pre-training tasks

❓ Questions & Help

Hello, congrats to all contributors for the awesome work with LXMERT! It is exciting to see multimodal transformers coming to hugginface/transformers. Of course, I immediately tried it out and played with the demo.

LXMERT pre-trained model, trained on what exactly?

Question: Does the line lxmert_base = LxmertForPreTraining.from_pretrained("unc-nlp/lxmert-base-uncased") load an already pre-trained LXMERT model on the tasks enumerated in the original paper “(1) masked crossmodality language modeling, (2) masked object prediction via RoI-feature regression, (3) masked object prediction via detected-label classification, (4) cross-modality matching, and (5) image question answering.” (Tan & Bansal, 2019)? If the pre-training tasks are not all the ones from the paper, would that line load pre-trained weights at all and if yes, on what?

Thanks in advance! 🤗

A link to original question on the forum/Stack Overflow: Here is the link to the hugginface forum.

[yezhengli-Mr9]

Is there any entry-level example of Lxmert？ Following example from Lxmert.

from transformers import LxmertTokenizer, LxmertModel
import torch

tokenizer = LxmertTokenizer.from_pretrained('unc-nlp/lxmert-base-uncased')
model = LxmertModel.from_pretrained('unc-nlp/lxmert-base-uncased')

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state

comes up

File "/Users/yezli/miniconda3/lib/python3.8/site-packages/transformers/models/lxmert/modeling_lxmert.py", line 933, in forward
    assert visual_feats is not None, "`visual_feats` cannot be `None`"
AssertionError: `visual_feats` cannot be `None` 

[eltoto1219]

Hi, “unc-nlp/lxmert-base-uncased” was trained with all tasks specified in the paper (as aforementioned). We have benchmarked the pre-trained model to make sure it reaches the same performance on all QA tasks. If you do run into any troubles though, please let me know!