Performance difference

Hi there,

Thank you for uploading your implementation of the NER Tagger! Can you please tell me, with which settings it is possible to replicate the performance of glample’s NER tagger on German conll data while using the original embeddings? In 100 epoch, the highest value I get is around 71% (with Theano backend for BiLSTM-CNN-CRF v. 1.2.2) or 70% (with Tensorflow 1.8 backend for BiLSTM-CNN-CRF v. 2.2.0) while using the original configurations

params = {'classifier': 'CRF', 'LSTM-Size': [64], 'dropout': (0.5), 'charEmbeddings': 'LSTM', 'charEmbeddingsSize':'30', 'maxCharLength': 50, 'optimizer': 'sgd', 'earlyStopping': 30}

and further using the IOB Tagging. Do you know how to solve this issue?

Thanks!

[nreimers]

That’s correct, I don’t use the CoNLL 2000 eval script, as it is rather slow and would require that perl is also installed.

I tested my implementation and it produced for me the same results as the CoNLL 2000 eval script if the BIO encoding is valid.

For invalid BIO encoding, the provided code uses two post-editing strategies to ensure a valid BIO encoding: Set invalid tags to O (i.e. O I-PER => O O) or set invalid tags to B (i.e. O I-PER => O B-PER).

I’m not sure how the conll 2000 scripts deals with invalid encoding.

[nreimers]

If all tags are valid, then the evaluation script (implemented in Python) produces the same scores as the CoNLL 2003 perl script.

In the experiments there are two methods how to deal with invalid BIO tags: Set them to O (I-PER I-PER => O O) or start a new tag with B (I-PER I-PER => B-PER I-PER). Both methods ensure, that there are no invalid tags.

If invalid tags are passed to the evaluation script (without the described fix), then they are considered as an error. This is different to the CoNLL 2003 perl script, there, invalid tags are not an error.