Input for ProbClassificationPerformanceAnalyzer

I posted same question at discord first, but this maybe a more appropriate place. We’ll see 😉

I’m writing some tests for dashboards and profiles but stumbled upon a probable bug in ProbClassificationPerformanceAnalyzer. https://github.com/burkovae/evidently/blob/ab-fix-classification-bugs/evidently/analyzers/test_classification_analyzers.py

Since I’m not sure, how a correct input would look like, I hope that someone can post ist here. I will be happy with a very basic data for an analysis of a single dataframe. Hence, I hope to get some correct input, so that I can narrow down what’s happening and what components fails.

Especially I would like know, what the ProbClassificationPerformanceAnalyzer computes exactly (@emeli-dral it seems you wrote this piece). I would be happy with few inputs and expected outputs.

If you already know the fix to

Error
Traceback (most recent call last):
  File "...\evidently\evidently\analyzers\test_classification_analyzers.py", line 21, in test_single_dataset_with_two_classes
    blubb = analyzer.calculate(df, None, ColumnMapping())
  File "...\evidently\analyzers\prob_classification_performance_analyzer.py", line 32, in calculate
    prediction_labels = [prediction_column[x] for x in prediction_ids]
TypeError: 'numpy.intc' object is not iterable

then I urge you to write the tests with inputs and expected outputs first, please. It helps down the line and improves robustness and safeguard against unwanted changes.

If you can point me in the right direction, I will fix any issues along the road. I just need to know what you compute specifically with some explicit examples.

[Liraim]

Probably my previous answer was misleading:

You are totally right that good tests can be good documentation, and we definitely will write this kind of tests. But it is time-consuming to write proper tests, so I mentioned documentation.

And I didn’t mean that documentation will be substitution for tests in any way, docs that I meant is some kind of high-level description about how given component work, and how you can use it (maybe with code snippets). And it is definitely not a way to test our code.

In short:

• tests is our priority for code-quality
• documentation also our priority but for usability

[burkovae]

thb, 4 years ago, I though writing proper tests would be time consuming as well until a co-worker introduced me to test driven development. Since then I cannot help myself but to write tests first 😃 We all have our journey and the community is here to help. Because I believe in evidently’s approach ❤, I will contribute to tests whenever I can.