Inconsistent example output

Installed 6.3.7 from pip as instructed on both latest macOS and ubuntu 16.04 and downloaded frequency_dictionary_en_82_765.txt from the official github. Just ran the examples in the README.md and got inconsistent output on both platforms as follows:

Sample usage (lookup and lookup_compound)

The last number(log_prob_sum) is 11 instead of 10.

members, 226656153, 1
where is to love he had dated for much of the past who couldn't read in six grade and inspired him, 300000, 11

Sample usage (word_segmentation)

The first word the is segmented as t and he which are a bit obvious. Also overt he should be over the. I noticed the last two numbers are different from 8 -34.491167981910635.

t he quick brown fox jumps overt he lazy dog, 10, -52.10066239535173

Next, I tried to segment the test string itwasthebestoftimesitwastheworstoftimesitwastheageofwisdomitwastheageoffoolishness from the official site and the output shows the same error pattern of the as t he.

Any ideas?

[mammothb]

I see, I might have saved it as utf-8 when I was debugging the program and uploaded it as that.

load_dictionary allows you to choose the encoding so you could use that as well.

[farleylai]

I finally figured out the cause which has nothing to do with the installation but some invisible character in the frequency dictionary that I downloaded from the original author wolfgarbe’s repo:

frequency_dictionary_en_82_765.txt

Running diff to compare the files, showing there is an invisible difference in line 1 that is the coincidentally:

1c1
< the 23135851162
---
> the 23135851162

I tried :set list in vim to show the invisible chars but there seems nothing different. Then the first line was print and it happens to be the unicode Byte Order Mark (BOM) causing the issue as discussed in the thread:

>>> f = open('frequency_dictionary_en_82_765.txt.wolfgarbe')
>>> f.readline()
'\ufeffthe 23135851162\n'

One workaround would be to set the encoding argument as follows:

>>> f = open('frequency_dictionary_en_82_765.txt.wolfgarbe', encoding='utf-8-sig')
>>> f.readline()
'the 23135851162\n'