[RAG] RagSequenceForGeneration should not load "facebook/rag-token-nq" and RagTokenForGeneration also should not load "facebook/rag-sequence-nq"

Environment info

• transformers version: 3.3.1
• Platform: Linux-4.19.112±x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.6.9
• PyTorch version (GPU?): 1.6.0+cu101 (False)
• Tensorflow version (GPU?): 2.3.0 (False)
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help

@patrickvonplaten

Information

Model I am using (Bert, XLNet …): RAG

The problem arises when using:

• the official example scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)

To reproduce

Following usage of token and sequence models should not be allowed, it may give unintended result in forward pass-

# RagSequenceForGeneration with "facebook/rag-token-nq"
model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)

# RagTokenForGeneration with "facebook/rag-sequence-nq"
model = RagTokenForGeneration.from_pretrained("facebook/rag-sequence-nq", retriever=retriever)

Also please correct example at https://huggingface.co/transformers/master/model_doc/rag.html#ragsequenceforgeneration

Expected behavior

Above usage should throw exception because both the models are incompatible with the each other.

[patrickvonplaten]

@patrickvonplaten

I am seeing very weird behaviour. Various RAG generator and model combination giving me very different output. I am not able to understand why?

Check output of generators for “What is capital of Germany?” -

!pip install git+https://github.com/huggingface/transformers.git
!pip install datasets
!pip install faiss-cpu
!pip install torch torchvision

from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration, RagSequenceForGeneration
import torch
import faiss


tokenizer = RagTokenizer.from_pretrained("facebook/rag-sequence-nq")
retriever = RagRetriever.from_pretrained("facebook/rag-sequence-nq", index_name="exact", use_dummy_dataset=True)


input_dict = tokenizer.prepare_seq2seq_batch("What is capital of Germany?", return_tensors="pt")
input_ids = input_dict["input_ids"]

# RagTokenForGeneration with "facebook/rag-token-nq"
model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
generated_ids = model.generate(input_ids=input_ids)
generated_string = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print("Result of model = ", generated_string)

# RagSequenceForGeneration with "facebook/rag-sequence-nq"
model = RagSequenceForGeneration.from_pretrained("facebook/rag-sequence-nq", retriever=retriever)
generated_ids = model.generate(input_ids=input_ids)
generated_string = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print("Result of model = ", generated_string)

# RagSequenceForGeneration with "facebook/rag-token-nq"
model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
generated_ids = model.generate(input_ids=input_ids)
generated_string = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print("Result of model = ", generated_string)

# RagTokenForGeneration with "facebook/rag-sequence-nq"
model = RagTokenForGeneration.from_pretrained("facebook/rag-sequence-nq", retriever=retriever)
generated_ids = model.generate(input_ids=input_ids)
generated_string = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print("Result of model = ", generated_string)

Output of above run is (it is consistent behaviour) -

Result of model =  [' german capital']
Result of model =  ['']
Result of model =  [' munich']
Result of model =  [' germany']

Hey @lalitpagaria , the models are different in generating the answers - the results are not unexpected 😃 If you take a closer look into the code you can see that both models expect the exact same weights, but have different generate() functions

[lalitpagaria]

@patrickvonplaten

I am seeing very weird behaviour. Various RAG generator and model combination giving me very different output. I am not able to understand why?

Check output of generators for “What is capital of Germany?” -

!pip install git+https://github.com/huggingface/transformers.git
!pip install datasets
!pip install faiss-cpu
!pip install torch torchvision

from transformers import RagTokenizer, RagRetriever, RagTokenForGeneration, RagSequenceForGeneration
import torch
import faiss


tokenizer = RagTokenizer.from_pretrained("facebook/rag-sequence-nq")
retriever = RagRetriever.from_pretrained("facebook/rag-sequence-nq", index_name="exact", use_dummy_dataset=True)


input_dict = tokenizer.prepare_seq2seq_batch("What is capital of Germany?", return_tensors="pt")
input_ids = input_dict["input_ids"]

# RagTokenForGeneration with "facebook/rag-token-nq"
model = RagTokenForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
generated_ids = model.generate(input_ids=input_ids)
generated_string = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print("Result of model = ", generated_string)

# RagSequenceForGeneration with "facebook/rag-sequence-nq"
model = RagSequenceForGeneration.from_pretrained("facebook/rag-sequence-nq", retriever=retriever)
generated_ids = model.generate(input_ids=input_ids)
generated_string = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print("Result of model = ", generated_string)

# RagSequenceForGeneration with "facebook/rag-token-nq"
model = RagSequenceForGeneration.from_pretrained("facebook/rag-token-nq", retriever=retriever)
generated_ids = model.generate(input_ids=input_ids)
generated_string = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print("Result of model = ", generated_string)

# RagTokenForGeneration with "facebook/rag-sequence-nq"
model = RagTokenForGeneration.from_pretrained("facebook/rag-sequence-nq", retriever=retriever)
generated_ids = model.generate(input_ids=input_ids)
generated_string = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
print("Result of model = ", generated_string)

Output of above run is (it is consistent behaviour) -

Result of model =  [' german capital']
Result of model =  ['']
Result of model =  [' munich']
Result of model =  [' germany']