Add runtime metrics to `Evaluator`

Currently, the Evaluator class for text-classification computes the model metrics on a given dataset. In addition to model metrics, it would be nice if the Evaluator could also report runtime metrics like eval_runtime (latency) and eval_samples_per_second (throughput).

cc @philschmid

[lvwerra]

I think we can can just add the throughput information to the dict that is returned by the evaluator.

I like the idea of an evaluation report, however, I don’t think we can assume to know e.g. the device a the pipeline is running on: for now it is a transformer pipeline but it could be any callable so we would not know how to get that info. The evaluate.save function lets you store any information and by default also saves some system information. Maybe we could extend this and then let the user add whatever can not be easily inferred (e.g. the device of the pipeline). What do you think?

As for dummy inputs: I think this is something we should let the user handle. Maybe we can extend the docs with a dedicated “Evaluator” section and have a guide “How to measure the performance of your pipeline” section where we show best practices.

[ola13]

I think that’s a great idea! True - it is backend dependent, but it will be very useful for debugging.

I wonder if it would be useful to optionally output not only the metric values but some sort of an evaluation report - basic setup information along with the runtime metrics, e.g. what device it was evaluated on etc.

I think it would be valuable to have these numbers for full evaluation, not only dummy input, as it doesn’t really cost anything and it can provide additional insights (again, mainly in the debugging scenario)