Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

mmmlu tasks improvement

See original GitHub issue

Our human annotators are complaining about the task definitions for the MMMLU tasks.

I think part of the issue is that the tasks don’t explain enough about the desired outputs. For example, here: https://github.com/allenai/natural-instructions-expansion/blob/master/tasks/task667_mmmlu_answer_generation_business_ethics.json

The only thing said is that "You are given a question on business ethics. But then: How do the expected answer relate to business ethics? (should they pick something that is more ethical or less ethical?) What is even “business ethics”, to begin with? We need to improve the instructions to address these.

Issue Analytics

State:
Created 2 years ago
Comments:7 (4 by maintainers)

Top GitHub Comments

1reaction

pulkitverma25commented, Nov 24, 2021

@pulkitverma25 are there any updates on Discussion #467. The conflicts might be difficult to resolve if we’re working on the same tasks.

Since the updates are in different parts of the files, this shouldn’t be difficult. You go ahead and submit your changes, I will take care of the merge. To avoid any confusion: The only issue related to MMMLU tasks that I am currently working on is #569

1reaction

danyaljjcommented, Nov 15, 2021

@Sujan242 could you work on improving these task definitions?

Top Results From Across the Web

Measuring Massive Multitask Language Understanding - arXiv

However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy.

Another negative example for MMLU tasks. · Issue #569 - GitHub

It seems like the annotators needed one negative example that actually had an incorrect answer, not just an answer with an incorrect format....

Paper tables with annotated results for Multi-Task Learning in ...

In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on ...

Is AI Progress Impossible To Predict? - LessWrong

For example, does a task improving rapidly when you go from a small model to a 7B parameter ... Individual MMMLU tasks are...

The 12 tips that will improve your multitasking skills! - Cirkus

Juggling many tasks can become easier and less stressful with practice. Multitasking skills can be learned and improved with every project ...