Steps to utilize NeuroNER for other languages

It appears that BART at least is pretty language agnostic. The English specific parts of NeuroNER (afaict), are the recommended glove.6B.100d word vectors, and all of the spacy related tokenizing code, which is used to translate BART format into CoNLL format (correct?)

Am I correct that if I:

1. Supply Korean word vectors in /data/word_vectors
2. Supply CoNLL formatted train, valid, and test data using BART labeled Korean text which I run through my own tokenizer

I will be able to train and use NeuroNER for Korean text?

[Gregory-Howard]

Hi (I’m the guy who uses NeuroNER in French)! These 2 steps are true, but you also need spacy (or nltk) working in Korean. I’m explaining a bit more for SpaCy : You need a SpaCy Korean model. This consist in a tokenizer and a POS Tagging model. Someone asked exactly this question : https://github.com/explosion/spaCy/issues/929 Then you will have to change spacylanguage in parameter.ini I hope I’m clear, if not, feel free to ask.

Steps (for spacy) language : X:

• Check if NLTK or spacy support your language X (you need full support). (https://github.com/explosion/spaCy#spacy-industrial-strength-nlp)
  • if not add your language : https://spacy.io/docs/usage/adding-languages (1-2 week)
• Supply X word vectors in /data/word_vectors
• Supply CoNLL/BRAT(into directory) formatted train, valid, and test data using BART labeled X text which I run through my own tokenizer
• change parameter.ini : token_pretrained_embedding_filepath, token_embedding_dimension, spacylanguage, dataset_text_folder
• run main.py

[Franck-Dernoncourt]

Correct! Note that providing word vectors is optional (it’s typically better if you have some), and that I haven’t tested NeuroNER with languages other than English. I know someone successfully used it in French (after an encoding fix PR 😃), and someone was supposed to try with Bangladeshi but I haven’t heard back from him.

On Jul 3, 2017 9:49 PM, “Sooheon Kim” [e-mail] wrote:

It appears that BART at least is pretty language agnostic. The English specific parts of NeuroNER (afaict), are the recommended glove.6B.100d word vectors, and all of the spacy related tokenizing code, which is used to translate BART format into CoNLL format (correct?)

Am I correct that if I:

1. Supply Korean word vectors in /data/word_vectors
2. Supply CoNLL formatted train, valid, and test data using BART labeled Korean text which I run through my own tokenizer

Will I be able to train and use NeuroNER for Korean text?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Franck-Dernoncourt/NeuroNER/issues/30, or mute the thread https://github.com/notifications/unsubscribe-auth/AAA7447RV_hPWNxKIwrgUb6oHxSekvLUks5sKahDgaJpZM4OM1FP .