How can I reduce the false-positive mutants for strings?

I just ran mutmut over my toy project mpu and got a lot of false positives for strings.

False Positive String Mutations

Exception messages: They should never be mutated; all of the mutants were false-positives like this:

-        raise ValueError("All primes are prime factors of 0.")
+        raise ValueError("XXAll primes are prime factors of 0.XX")

print values: I neither

-    print("Number of datapoints: {datapoints}".format(datapoints=len(df)))
+    print("XXNumber of datapoints: {datapoints}XX".format(datapoints=len(df)))

Valuable String Mutations

The following examples could potentially show issues:

comparisons:

-            if column_type == "other":
+            if column_type == "XXotherXX":

assignments: I see that assignments are hard to tell. The following could either be in a comparison or in a print (it is a print):

     table = [
-        ["Column name", "Non-nan", "mean", "std", "min", "25%", "50%", "75%", "max"]
+        ["Column name", "XXNon-nanXX", "mean", "std", "min", "25%", "50%", "75%", "max"]
     ]

Solutions

Is it possible to configure mutmut to completely disable string mutations or to disable it for those cases?

I have seen Advanced whitelisting and configuraton, but I don’t understand how to apply it in my case.

[blueskyjunkie]

@MartinThoma I disagree that you would “never check this because I’ve designed/implemented to not need the test”. The problem with software development is often not how it is implemented now, but how it changes over time. What if someone adds a new exception in future in the factorize function and reuses the type with a different error message? You want tests to clearly indicate that a new logical error has been introduced by the change (if indeed that is the case). Yes, you could document somewhere that the exception type is not supposed to be reused, but what if that person forgets, doesn’t notice, is careless, whatever; you still want the tests to clearly indicate that a problem has been introduced. Software development has to account for present and future human cognitive frailty as well correctness in delivering business value.

Think of unit/integration tests as the definition of correctness of your software at this moment in time. As the software evolves, the tests should evolve with the software to continue to define correctness. Historically test coverage has been used as a proxy for test quality in absence of other metrics (and it does help), however mutmut introduces the possibility of actually testing the quality of the tests.

[blueskyjunkie]

For the exceptions you should also have tests like this (example uses pytest, you can definitely do equivalent using unittest, and presumably other frameworks):

import pytest

def test_raises():
    with pytest.raises(ValueError, match=r"^Zero cannot be factorized"):
        result = factorize(0)

This tests both that the correct type of exception is raised when you expect it, and that the exception raised has the correct message since you might use the same exception multiple times for different error messages.