Minimal answer spans are wrong for some examples.

When I try to read the data and print questions and minimal span answers, I noticed that for some examples plaintext_start_byte and plaintext_end_byte of minimal answer span are shifted by k symbols right.

Example: Question: ‘Who created the series Clannad?’ Minimal span: ‘rel’ (it should be ‘Key’, but its values are shifted and it gives ‘rel’ part of the word ‘released’)

... Clannad(クラナド,Kuranado) is a Japanese visual novel developed by Key and released on April 28, 2004 ...

This is how I read the data file:

    with open(path_name) as input_file:
        for line in input_file:
            try:
                json_example = json.loads(line)
                if json_example['language'] not in allowed_langs:
                    continue

                plain_text = json_example['document_plaintext']
...

[dhgarrette]

I suspect that you’re doing a slice on the Python string’s characters instead of bytes. In Python3, string indices are given in unicode characters (unlike Python2, which used bytes). The example you gave has non-ascii characters that take up more than one byte, so the byte and character indices will not be the same:

>>> plaintext = "Clannad(クラナド,Kuranado) is a Japanese visual novel developed by Key and released on"
>>> plaintext_start_byte = 71
>>> plaintext_end_byte = 74
>>> plaintext[plaintext_start_byte:plaintext_end_byte]  # Incorrect
'rel'
>>> plaintext.encode()[plaintext_start_byte:plaintext_end_byte].decode()  # Correct
'Key'

[hashedi]

I think @tomohideshibata code works fine