leammtizer issue for german words

I am confused about the lemmatizer. For a sentence Ich sehe Bäume (I see trees).

nlp = spacy.load('de_core_news_sm')
doc = nlp(u'Ich sehe Bäume')

for token in doc:
    print(token.text,token.lemma, token.lemma_, token.pos_)
    print("has_vector:", token.has_vector)

token.lemma is just Bäume. I thought it would be lemmatized to the singular form Baum (tree)?

[ines]

Yes, Baum would definitely be correct here. The German lemmatizer only uses lookup tables (and no rule-based process like the English one). This has some limitations – I’ve written a bit more about this in my comment on this thread.

Another problem is that spaCy will always decide on one lemma (and won’t just give you a bunch of options to choose from). This is convenient – but it also means that if the one pick has to be correct. That said, there’s definitely been some suspicious reports around the lemmatization performance that might indicate a bug.

In the meantime, you might want to check out the spacy-iwnlp extensions by @Liebeck and see how it performs on your use case!

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.