Missing Usage examples in documentation for visualizations

It looks like the documentation is still missing code examples of how to use the visualizations: https://ludwig-ai.github.io/ludwig-docs/api/visualization/#compare_performance

It would be nice to see an example of the test_stats_per_model parameter:

test_stats_per_model (List[dict]): dictionary containing evaluation performance statistics.

My code looks like this:

training_stats, preprocessed_data, output_directory = base_model.train(
  training_set='./datasets/spam_train.csv',
  test_set='./datasets/spam_test.csv'
)

And this is what I’ve tried:

visualize.learning_curves(training_stats, ...)
visualize.learning_curves([training_stats], ...)
visualize.learning_curves((training_stats['model'], ...)

All of them raised errors. I’ve seen a blog where the code example supplied a filename string instead of a List[dict], so what’s the correct way to call train or evaluate and pass the training statistics to the visualize methods?

Thanks for any help on this!

[modernlearner]

@jimthompson5802 I thought .evaluate would be the simplest way to get the stats after loading the model from a file instead of re-training (or trying to load the training stats from a JSON file). I did notice that the evaluate stats were missing the epoch data it wasn’t clear which other visualizations would work with both training and evaluate stats.

Thanks again for the thorough examples! I created a draft PR to document the use of training and evaluate with compare_performance: https://github.com/ludwig-ai/ludwig/pull/1068

Going to close this issue as my questions have been answered. Thanks again!

[jimthompson5802]

@modernlearner In writing about the different content between training_statistics and evaluation_statistics and showing the content of the difference, I forgot to point out Ludwig’s compare_performance visualization that works with evaluation_statistics. Here is an example of its use.