New Evaluation: Math

We would like to find or create an eval dataset that tests mathematical knowledge. The GRE exams may be a good source of material.

• Data processing code implemented
• Evaluation implemented

The evaluation code should be modeled after the interface in lm_eval/base.py and the example of the BoolQ task in lm_eval/tasks/suerglue.py

[hendrycks]

The DeepMind mathematics dataset is in the pile.

The GRE exams may be a good source of material.

Some GRE exams (biology, chemistry, maths, physics, cs) are covered in https://arxiv.org/pdf/2009.03300.pdf

Here’s a competition maths dataset: https://drive.google.com/file/d/1RafScUe8O6MxE4K3COnOT4CFR4zvQdXv/view?usp=sharing I’ll have an arXiv paper for it in a month.

[StellaAthena]

OK, though it might be an undergraduate collaborator of mine to add the code.

Understood. Thanks again!