Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Question: Does NERModel support sliding_window?

See original GitHub issue

Hello, I was wondering how does the NERModel deal with long documents. Specifically can we provide the sliding_window argument to deal with this?

Alternatively, are there any best practices to handle long documents that have not been implemented into this library?

Thanks!

Issue Analytics

State:
Created 4 years ago
Comments:8 (3 by maintainers)

Top GitHub Comments

1reaction

DebanjanaKarcommented, Jul 28, 2020

That makes sense…thanks a lot 😄

0reactions

ThilinaRajapaksecommented, Jul 27, 2020

Because NER is a token level task, I assumed that it would be more sensitive. If a word gets split in the middle, then it certainly changes the assigned NER tag (it would also create two tags for the two pieces of the word). If a sentence gets split, that might potentially change the tags for each word in the sentence as the meaning of the sentence is likely to change (or stop being meaningful altogether).

For classification, this should be less of an issue as classification generally depends on the sequence as a whole.