Inference for TFMarianMTModel (en to Romance language translation) is slow and inaccurate

System Info

System macOS Monterey 12.2.1

transformers==4.20.1
tensorflow==2.9.1

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, …)
• My own task or dataset (give details below)

Reproduction

from transformers import TFMarianMTModel, MarianTokenizer
model_name = "Helsinki-NLP/opus-mt-en-ROMANCE"
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = TFMarianMTModel.from_pretrained(model_name)
text_in = ['>>fr<< hello']
batch = tokenizer(text_in, return_tensors='tf', padding=True)
translated = model.generate(**batch) 

Output:

- Qu'est-ce qu'il y a, là-bas, là-bas, là---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Expected behavior

I would expect similar performance to the PyTorch model.

Inference requires about 120s on my machine and outputs an incorrect translation. In contrast, the PyTorch model (replacing TFMarianMTModel with MarianMTModel and changing return_tensors to 'pt' in the code snippet) returns the correct translation (“Bonjour”) and inference requires about 6s on my machine.

[gante]

Hi there @ydshieh @danielenricocahall 👋

None of the Marian models can be successfully converted to TF – they all fail when validating the hidden layers and outputs of the models. This is a shame since there are a ton of Marian models for translation 😦

This means there is something wrong with either the model architecture or with weight cross-loading. I haven’t looked into it, other than noticing the issue when attempting to convert the weights from Helsinki-NLP

[jamie0725]

@ydshieh Hi, I am experiencing the same issue. Expected the TF version would be faster than the PT version.