Fast tokenizers calculate wrong offsets when special characters are present

Example:

>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True, add_special_tokens=False)
>>> sentence = "A, naïve [MASK] AllenNLP sentence."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in tokenized['offset_mapping']:
...     print(repr(sentence[start:end]))
'A'
','
'naïve'
' [MASK'
' Alle'
'nN'
'L'
' sentenc'
'e'

As you can see, after the word “naïve”, the offsets go off the rails.

[dirkgr]

@mfuntowicz, would it make sense for me to integrate our tokenizer tests into your code, so you can see these things immediately? I’d be happy to do so.

[n1t0]

>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True, add_special_tokens=False)
>>> sentence = "A, naïve [MASK] AllenNLP sentence."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in tokenized['offset_mapping']:
...     print(repr(sentence[start:end]))
'A'
','
'naïve'
'[MASK]'
'Allen'
'NL'
'P'
'sentence'
'.'

and

>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True, add_special_tokens=False)
>>> sentence = "I went to the zoo yesterday, but they had only one animal."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in (t for t in tokenized['offset_mapping'] if t is not None):
...     print(repr(sentence[start:end]))
'I'
'went'
'to'
'the'
'zoo'
'yesterday'
','
'but'
'they'
'had'
'only'
'one'
'animal'
'.' 

and the last one:

from transformers import BertTokenizerFast
tokenizer = BertTokenizerFast.from_pretrained('bert-base-uncased')
tokenizer.add_tokens(['[test]'])
text = "A [test] C"

print(tokenizer.encode(text, add_special_tokens=True))

results = tokenizer.encode_plus(text, 
            return_offsets_mapping=True,
            pad_to_max_length=False,
            max_length=128,
            return_overflowing_tokens=False,
            add_special_tokens=True)

for se in results['offset_mapping']:
  if se:
    print(text[se[0]:se[1]], se)

gives

[101, 1037, 30522, 1039, 102]
 (0, 0)
A (0, 1)
[test] (2, 8)
C (9, 10)
 (0, 0)