Behavior of shift_tokens_right on padded input_ids

System Info

• transformers version: 4.20.1

Who can help?

@patrickvonplaten @patil-suraj @sgugger

Reproduction

When I applied shift_tokens_right on a padded input_ids, I get this

from transformers import AutoTokenizer
from transformers.models.bart.modeling_flax_bart import shift_tokens_right

tokenizer = AutoTokenizer.from_pretrained("roberta-base")
labels = tokenizer("My dog is cute", padding='max_length', max_length=8, return_tensors='np').input_ids
decoder_input_ids = shift_tokens_right(labels, tokenizer.pad_token_id, tokenizer.eos_token_id)

print(tokenizer.batch_decode(labels))
# ['<s>My dog is cute</s><pad><pad>']

print(tokenizer.batch_decode(decoder_input_ids))
# ['</s><s>My dog is cute</s><pad>']

Expected behavior

Should the desired behavior of shift_token_right be the following?

print(tokenizer.batch_decode(labels))
# ['<s>My dog is cute</s><pad><pad>']

print(tokenizer.batch_decode(decoder_input_ids))
# ['</s><s>My dog is cute<pad><pad>']

[duongna21]

@patrickvonplaten Thanks for elaborating on it. Totally agree with you that it doesn’t matter with seq2seq training. I just tried to make sure it doesn’t create any side-effect somewhere 😄.

[LysandreJik]

Pinging @patil-suraj and @patrickvonplaten regarding the shift_tokens_right method and its purpose.