CLIP tokenizer inconsistent with OpenAI release

Environment info

• transformers version: 4.6.1
• Platform: Linux-5.4.0-52-generic-x86_64-with-glibc2.10
• Python version: 3.8.5
• PyTorch version (GPU?): 1.8.1 (True)
• Tensorflow version (GPU?): 2.4.1 (False)
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

@patil-suraj

Information

Model I am using (Bert, XLNet …):

The problem arises when using:

• my own modified scripts: (give details below)
• the official example scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

>>> import clip
>>> import transformers
>>> clip.tokenize('hello world')
tensor([[49406,  3306,  1002, 49407,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0]])
>>> tokenizer = transformers.CLIPTokenizerFast.from_pretrained('openai/clip-vit-base-patch32')
>>> tokenizer('hello world')
{'input_ids': [3306, 220, 1002], 'attention_mask': [1, 1, 1]}

The HF CLIPTokenizer seems to add an extra token while dropping the <bos> and <eos> tokens. Am I missing something here?

Thanks!

[nickgkan]

Hi, is there any update/eta on this?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.