Cannot reproduce the IRNet+BERT results

Thank you for sharing the IRNet+bert model. The code is very useful, but I can not reproduce the results from the paper (61.9% dev acc).

Testing the pre-trained model downloaded from #2, I got Acc: 0.572957, Beam Acc: 0.598249 using your eval and 0.572 EM accuracy using the official evaluation script.

Testing the model trained from scratch (bert branch), I got similar results: Acc: 0.564202, Beam Acc: 0.600195 using your eval and 0.573 EM accuracy using the official script.

Do you have similar results from eval.py script? What is the accuracy of the released model?

[brunnurs]

I implemented a BERT encoder myself (https://github.com/brunnurs/valuenet)

[saparina]

@brunnurs did you reproduce IRNet+BERT results in your version? The abstract of your paper sounds very interesting, will read it 😃