"mutmut run 65" doesn't seem to re-test a mutation?

Maybe this is just me not understanding what “mutmut run 65” is supposed to do. I updated a test to kill mutant 65, then ran “mutmut run 65”, but it showed it as still surviving. When I then applied 65, the tests failed:

$ mktmpenv -p python3.7
Running virtualenv with interpreter /usr/local/bin/python3.7
Using base prefix '/usr/local/pythonz/pythons/CPython-3.7.2'
/usr/local/pythonz/pythons/CPython-2.7.14/lib/python2.7/site-packages/virtualenv.py:1047: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
New python executable in /usr/local/virtualenvs/tmp-b4da94a5634daf06/bin/python3.7
Also creating executable in /usr/local/virtualenvs/tmp-b4da94a5634daf06/bin/python
Installing setuptools, pip, wheel...done.
Requirement already up-to-date: pip in /usr/local/virtualenvs/tmp-b4da94a5634daf06/lib/python3.7/site-packages (19.0.3)
Requirement already up-to-date: setuptools in /usr/local/virtualenvs/tmp-b4da94a5634daf06/lib/python3.7/site-packages (40.8.0)
This is a temporary environment. It will be deleted when you run 'deactivate'.

$ git clone [e-mail]:nedbat/cog.git
Cloning into 'cog'...
remote: Enumerating objects: 657, done.
remote: Counting objects: 100% (657/657), done.
remote: Compressing objects: 100% (230/230), done.
remote: Total 657 (delta 458), reused 619 (delta 421), pack-reused 0
Receiving objects: 100% (657/657), 172.29 KiB | 574.00 KiB/s, done.
Resolving deltas: 100% (458/458), done.

$ cd cog

$ git checkout c1f605d
HEAD is now at c1f605d mutmut settings

$ mutmut run

- Mutation testing starting -

These are the steps:
1. A full test suite run will be made to make sure we
   can run the tests successfully and we know how long
   it takes (to detect infinite loops for example)
2. Mutants will be generated and checked

Mutants are written to the cache in the .mutmut-cache
directory. Print found mutants with `mutmut results`.

Legend for output:
🎉 Killed mutants. The goal is for everything to end up in this bucket.
⏰ Timeout. Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious. Tests took a long time, but not long enough to be fatal.
🙁 Survived. This means your tests needs to be expanded.

mutmut cache is out of date, clearing it...
1. Running tests without mutations
⠴ Running... Done

2. Checking mutants
⠋ 309/464  🎉 267  ⏰ 0  🤔 0  🙁 42^C
Aborted!      # this is because of the stuck infinite loop described in #103 

$ mutmut show 65
--- cogapp/cogapp.py
+++ cogapp/cogapp.py
@@ -187,7 +187,7 @@
     def msg(self, s):
         self.prout("Message: "+s)

-    def out(self, sOut='', dedent=False, trimblanklines=False):
+    def out(self, sOut='', dedent=True, trimblanklines=False):
         """ The cog.out function.
         """
         if trimblanklines and ('\n' in sOut):


$ mutmut run 65

- Mutation testing starting -

These are the steps:
1. A full test suite run will be made to make sure we
   can run the tests successfully and we know how long
   it takes (to detect infinite loops for example)
2. Mutants will be generated and checked

Mutants are written to the cache in the .mutmut-cache
directory. Print found mutants with `mutmut results`.

Legend for output:
🎉 Killed mutants. The goal is for everything to end up in this bucket.
⏰ Timeout. Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious. Tests took a long time, but not long enough to be fatal.
🙁 Survived. This means your tests needs to be expanded.

1. Using cached time for baseline tests, to run baseline again delete the cache file

2. Checking mutants
⠹ 1/1  🎉 0  ⏰ 0  🤔 0  🙁 1
# The mutant should still be surviving here, I haven't done anything yet.

# Now apply the change to the test.
$ git checkout 667fe10
Previous HEAD position was c1f605d mutmut settings
HEAD is now at 667fe10 Include dedent=default in a test

$ mutmut run 65

- Mutation testing starting -

These are the steps:
1. A full test suite run will be made to make sure we
   can run the tests successfully and we know how long
   it takes (to detect infinite loops for example)
2. Mutants will be generated and checked

Mutants are written to the cache in the .mutmut-cache
directory. Print found mutants with `mutmut results`.

Legend for output:
🎉 Killed mutants. The goal is for everything to end up in this bucket.
⏰ Timeout. Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious. Tests took a long time, but not long enough to be fatal.
🙁 Survived. This means your tests needs to be expanded.

1. Using cached time for baseline tests, to run baseline again delete the cache file

2. Checking mutants
⠹ 1/1  🎉 0  ⏰ 0  🤔 0  🙁 1
# Shouldn't this have been a killed mutant now?

$ mutmut apply 65

$ tox -e py36
py36 develop-inst-nodeps: /usr/local/virtualenvs/tmp-b4da94a5634daf06/cog
py36 installed: atomicwrites==1.3.0,attrs==19.1.0,-e [e-mail]:nedbat/cog.git@667fe101e97eaec6bad022fd4fb162d96ade392a#egg=cogapp,coverage==4.5.3,more-itertools==6.0.0,pluggy==0.9.0,py==1.8.0,pytest==4.3.1,six==1.12.0
py36 run-test-pre: PYTHONHASHSEED='2230429556'
py36 runtests: commands[0] | coverage run -m pytest
...F..........................................................................................................................                           [100%]
=========================================================================== FAILURES ===========================================================================
______________________________________________________________ CogTestsInMemory.testCogOutDedent _______________________________________________________________

self = <cogapp.test_cogapp.CogTestsInMemory testMethod=testCogOutDedent>

    def testCogOutDedent(self):
        infile = """\
            //[[[cog
            cog.out("This is the first line\\n")
            cog.out('''
                This is dedent=True
            ''', dedent=True, trimblanklines=True)
            cog.out('''
                This is dedent=False
            ''', dedent=False, trimblanklines=True)
            cog.out('''
                This is dedent=default
            ''', trimblanklines=True)
            cog.out("This is the last line\\n")
            //]]]
            This is the first line
            This is dedent=True
                This is dedent=False
                This is dedent=default
            This is the last line
            //[[[end]]]
            """

        infile = reindentBlock(infile)
>       self.assertEqual(Cog().processString(infile), infile)
E       AssertionError: '//[[[398 chars]lse\nThis is dedent=default\nThis is the last [15 chars]]]\n' != '//[[[398 chars]lse\n    This is dedent=default\nThis is the l[19 chars]]]\n'
E         //[[[cog
E         cog.out("This is the first line\n")
E         cog.out('''
E             This is dedent=True
E         ''', dedent=True, trimblanklines=True)
E         cog.out('''
E             This is dedent=False
E         ''', dedent=False, trimblanklines=True)
E         cog.out('''
E             This is dedent=default
E         ''', trimblanklines=True)
E         cog.out("This is the last line\n")
E         //]]]
E         This is the first line
E         This is dedent=True
E             This is dedent=False
E       - This is dedent=default
E       +     This is dedent=default
E       ? ++++
E         This is the last line
E         //[[[end]]]

cogapp/test_cogapp.py:170: AssertionError
=================================================================== short test summary info ====================================================================
FAILED cogapp/test_cogapp.py::CogTestsInMemory::testCogOutDedent
1 failed, 125 passed in 1.13 seconds
ERROR: InvocationError for command '/usr/local/virtualenvs/tmp-b4da94a5634daf06/cog/.tox/py36/bin/coverage run -m pytest' (exited with code 1)
___________________________________________________________________________ summary ____________________________________________________________________________
ERROR:   py36: commands failed

$ git diff
 diff --git a/cogapp/cogapp.py b/cogapp/cogapp.py
 index cb8486d..51122e0 100644
 --- a/cogapp/cogapp.py
 +++ b/cogapp/cogapp.py
 @@ -187,7 +187,7 @@ def evaluate(self, cog, globals, fname):
      def msg(self, s):
          self.prout("Message: "+s)

 -    def out(self, sOut='', dedent=False, trimblanklines=False):
 +    def out(self, sOut='', dedent=True, trimblanklines=False):
          """ The cog.out function.
          """
          if trimblanklines and ('\n' in sOut):

$

[nedbat]

I would think that “mutmut run <id>” should always run the tests. If I just wanted to see the results, why wouldn’t I use “mutmut show <id>”?

[boxed]

Well… if mutmut thinks the tests and code is unchanged, maybe you as a user are confused, or have mutmut configured wrong. In those cases I don’t think it would help the situation to just blindly rerun the test.