CliRunner returns different error codes in different machines

Expected Behavior

CliRunner should be deterministic and always return the same error code for empty parameters.

from unittest.mock import patch, Mock
from click.testing import CliRunner

def test_no_command():
        runner = CliRunner()
        result = runner.invoke(cli, [])
        self.assertEqual(result.exit_code, 0)
        result = runner.invoke(cli, ["get"])
>       self.assertEqual(result.exit_code, 1)
E       AssertionError: 2 != 1

The command definition (snippets):

@click.group()
@click.option('--output',
              '-o',
              "output",
              type=click.Path(writable=True, dir_okay=False),
              required=False,
              default=None,
              help="The output file name where results will be written. "
              "If not specified, results will just be displayed")
@click.option('--download',
              '-d',
              'download',
              type=click.Path(writable=True, dir_okay=True, file_okay=False),
              required=False,
              default=None,
              help="Directory where simulation input and output files will "
              "be downloaded together with the model object.")
@click.pass_context
def cli(ctx, output, download):
    ctx.ensure_object(dict)
    ctx.obj["output"] = output
    ctx.obj["download"] = download
    check_config()

@cli.group(chain=True, no_args_is_help=False)
@click.pass_context
def get(ctx):
    pass

We are testing the error code when cli is called without any subcommand and also when subcommands don’t receive any parameters. All other tests pass, but the no command machine fail in some machines. These machines run tha same pip environment, build and rebuilt many times, with pip freeze identical. The source code didn’t change since September 2019.

This is the 4th time the problem appears, we don’t have any clue about what is causing it. It also happened with Python 3.6.8.

In some machines, the return code is 1 and others it is 2.

Environment

• Python version: Python 3.8.0 on Microsoft Windows 10
• Pip version: 19.3.1
• Click version: 7.0

Pip environment

atomicwrites==1.3.0 attrs==19.3.0 aws-sam-translator==1.20.1 aws-xray-sdk==2.4.3 beautifulsoup4==4.8.2 boto==2.49.0 boto3==1.10.32 botocore==1.13.32 bs4==0.0.1 certifi==2019.11.28 cffi==1.13.2 cfn-lint==0.26.3 chardet==3.0.4 Click==7.0 colorama==0.4.3 coverage==5.0.3 cryptography==2.8 docker==4.1.0 docutils==0.15.2 ecdsa==0.15 entrypoints==0.3 et-xmlfile==1.0.1 flake8==3.7.9 future==0.18.2 idna==2.8 jdcal==1.4.1 Jinja2==2.10.3 jmespath==0.9.4 jsondiff==1.1.2 jsonpatch==1.24 jsonpickle==1.2 jsonpointer==2.0 jsonschema==3.2.0 loglib==1.1.1 lxml==4.4.2 MarkupSafe==1.1.1 mccabe==0.6.1 mock==3.0.5 more-itertools==8.1.0 moto==1.3.14 multidict==4.7.4 mypy==0.740 mypy-extensions==0.4.3 openpyxl==3.0.2 packaging==20.0 Pillow==6.2.1 pluggy==0.13.1 prosumer-common==1.0.2 py==1.8.1 pyasn1==0.4.8 pycodestyle==2.5.0 pycparser==2.19 pyflakes==2.1.1 pynamodb==4.2.0 pyparsing==2.4.6 pypiwin32==223 pyrsistent==0.15.7 pytest==5.3.3 pytest-cov==2.8.1 python-dateutil==2.8.0 python-jose==3.1.0 pytz==2019.3 pywin32==227 PyYAML==5.3 requests==2.22.0 responses==0.10.9 rsa==4.0 s3transfer==0.2.1 sentry-sdk==0.13.4 six==1.14.0 soupsieve==1.9.5 sshpubkeys==3.1.0 typed-ast==1.4.1 typing-extensions==3.7.4.1 urllib3==1.25.7 wcwidth==0.1.8 websocket-client==0.57.0 Werkzeug==0.16.0 wrapt==1.11.2 xmltodict==0.12.0

[ytjohn]

(Click==7.0)

I’m encountering something similar where it’s working locally (python 3.6.9) but on my cicd instance, which I just checked and is python 3.6.8, every CLIRunner based test is failing with result.exit_code being 1, and result.output being empty. I’m hoping to try 3.6.8 and some other things, but if you add a print(f"{result.exit_code} :: {result.output}") to your tests, do you get 1, "" across the board?

[HundredVisionsGuy]

I can’t reproduce this issue with the information provided.