[example scripts] inconsistency around eval vs val

• val == validation set (split)
• eval == evaluation (mode)

those two are orthogonal to each other - one is a split, another is a model’s run mode.

the trainer args and the scripts are inconsistent around when it’s val and when it’s eval in variable names and metrics.

examples:

• eval_dataset but --validation_file
• eval_* metrics key for validation dataset - why the prediction is then test_* metric keys?
• data_args.max_val_samples vs eval_dataset in the same line

the 3 parallels:

• train is easy - it’s both the process and the split
• prediction is almost never used in the scripts it’s all test - var names and metrics and cl args
• eval vs val vs validation is very inconsistent. when writing tests I’m never sure whether I’m looking up eval_* or val_* key. And one could run evaluation on the test dataset.

Perhaps asking a question would help and then a consistent answer is obvious:

Are metrics reporting stats on a split or a mode? A. split - rename all metrics keys to be train|val|test B. mode - rename all metrics keys to be train|eval|predict

Thank you.

@sgugger, @patil-suraj, @patrickvonplaten

[sgugger]

No the key in the dataset dictionary is “validation”, so it should be validation_file.

[stas00]

Awesome! Thank you, @bhadreshpsavani!

So the changes we need are:

1. use eval instead of val
2. use predict instead of test

in cl args and variable names in example scripts (only the active ones, please ignore legacy/research subdirs).

I hope this will be a last rename in awhile.