New GPT2 tokenizer no longer encodes Unicode characters properly in Python 3

In commit 5afa497cbfc53c679a9b22997b6312fad57ee2f8, you changed token.encode('utf-8') to simply token.

This would make the code compatible with Python 2, but now it breaks in Python 3. You’ll get a KeyError when you try to encode a Unicode character that requires more than 1 byte in UTF-8 encoding. For example, this raises a KeyError in Python 3:

from pytorch_pretrained_bert.tokenization_gpt2 import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
tokenizer.encode('你')

I think what you want to do is:

if sys.version_info[0] == 2:
    token = ''.join(self.byte_encoder[ord(b)] for b in token)
else:
    token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8'))

[jamestjw]

I can confirm that this is happening, though it is a different dash.

[lirongyuan]

I can confirm that this is happening, though it is a different dash.

Same here: This is also happening while using GPT2 tokenizer:

Traceback (most recent call last): File "run_lambada_gpt2.py", line 139, in tokenize_and_encode token_ids = tokenizer.encode(obj) File "/data/anaconda/envs/py35/lib/python3.5/site-packages/pytorch_pretrained_bert/tokenization_gpt2.py", line 261, in encode return self.convert_tokens_to_ids(self.tokenize(text)) File "/data/anaconda/envs/py35/lib/python3.5/site-packages/pytorch_pretrained_bert/tokenization_gpt2.py", line 224, in tokenize token = ''.join(self.byte_encoder[ord(b)] for b in token) File "/data/anaconda/envs/py35/lib/python3.5/site-packages/pytorch_pretrained_bert/tokenization_gpt2.py", line 224, in <genexpr> token = ''.join(self.byte_encoder[ord(b)] for b in token) KeyError: 8217

The sys version info is: sys.version_info(major=3, minor=5, micro=5, releaselevel='final', serial=0)