KeyError: in convert_tokens_to_ids()

In BertTokenizer’s, convert_tokens_to_ids function gives KeyError. So, I suggest to modify the for loop in the function as follows.

for token in tokens:
    ids.append(self.vocab.get(token, self.vocab['[UNK]']))

[thomwolf]

@congjianluo you should not tokenize the sentence your self but use Bert tokenizer. Please follow the usage example in the readme. In your case:

sent = "hi mary"
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
tokens = tokenizer.tokenize(sent)
tokens_ids = tokenizer.convert_tokens_to_ids(tokens)

[thomwolf]

@wasiahmad ok then I’m closing this issue. Feel free to re-open it if you have a (reproductible) example of such issue.