Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Results are different between command line and python script

See original GitHub issue

Hi,

I parsed the same word, but the results differed between the Command Line and Python Script.

Commas Line

$ginza 
mode is C
disabling sentence separator
ICカード
 text = ICカード
1       ICカード        ICカード        NOUN    名詞-普通名詞-一般      _       0       root    _       BunsetuBILabel=B|BunsetuPositionType=ROOT|SpaceAfter=No|NP_B

python script

nlp = spacy.load('ja_ginza')
doc = self.nlp("ICカード")
print(doc.to_json())
{'text': 'ICカード', 'ents': [{'start': 0, 'end': 5, 'label': 'PRODUCT'}], 'sents': [{'start': 0, 'end': 5}], 'tokens': [{'id': 0, 'start': 0, 'end': 2, 'pos': 'PROPN', 'tag': '名詞-固有名詞-一般', 'dep': 'compound', 'head': 1}, {'id': 1, 'start': 2, 'end': 5, 'pos': 'NOUN', 'tag': '名詞-普通名詞-一般', 'dep': 'ROOT', 'head': 1}]}

It was divided. “IC” and “カード”.

Issue Analytics

State:
Created 4 years ago
Comments:5 (3 by maintainers)

Top GitHub Comments

2reactions

yusuke-yaguchicommented, Sep 5, 2019

Hi @KoichiYasuoka, Thank you for your answer.

I saw a sudachi_tokenizer.py Mode is set ‘A’ or ‘B’ or ‘C’, not set Tokenizer.SplitMode.X

I send pull request.

0reactions

hiroshi-matsuda-ritcommented, Sep 29, 2019

@yusuke-yaguchi Thank you for reporting the bug and sending fixing PR. I’ve merged your PR. @KoichiYasuoka Thanks for your analysis to identify the reason of this problem.