SpaCy's `EntityRuler`-like functionality in Spark NLP

Is your feature request related to a problem? Please describe. I would like to annotate documents against a Knowledge Base, like spaCy’s EntityRuler does.

cf https://spacy.io/api/entityruler

Describe the solution you’d like This would be an annotator called EntityRuler, that would be instantiated by loading a local ontology in, let’s say, JSON or CSV format.

Describe alternatives you’ve considered Using spaCy for that, but on Big Data I fear it won’t scale.

Additional context No additional context.

[xegulon]

Hi @danilojsl, much thanks for the speed, the nice work and the notebook!

I’ve been through the notebook, and something I think could be enhanced is the schema of your JSON/CSV. In addition to label and pattern keys, it would be great to have an ID key (that would reference the entity uniquely), and the ability to have any other key/metadata related to the entity.

Moreover, I think taking into account JSONL format in addition to pure JSON could be interesting for speed (SpaCy people recommend that).

An error I had in the notebook:

Thanks again!

[danilojsl]

Hi @xegulon, I’m working on EntityRuler annotator and I have an alpha version of it. If you want to check this out, please see this Colab notebook . We will truly appreciate any feedback on this new feature.