Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

gets stuck in ansitowin32.py on GitLab CI

See original GitHub issue

Issue description

When running pipenv install or pipenv sync on GitLab CI, pipenv gets stuck (at least for 1h, until the job gets killed).

Expected result

It should not get stuck.

Actual result

--verbose does not print any more information. When LANG is set properly, you can’t see any output, otherwise you can read

Warning: the environment variable LANG is not set!
We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.

and after that it hangs.

Steps to replicate

Unfortunately I couldn’t reproduce it on gitlab.com. It happens on our private gitlab in a private repo. Here is a minimal version of a Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
Django = "<2.2.0"

[requires]
# Don't specify the exact python version, because of https://github.com/pypa/pipenv/issues/1050#issuecomment-346203646
python_version = "3"

I only run pipenv install or pipenv sync.

While the job was running / stuck in GitLab CI, I run the exact same command within the docker container, where the CI job was running and it worked. I think, the problem has something to do with how GitLab CI runs the commands. I thought, the problem might be, that stdout is not a tty, but running something like setsid sh -c 'pipenv install' < /dev/null > log 2>&1 did work, too.

I tried running python3 -m trace --trace /usr/bin/pipenv install > trace_log.txt and this is the output

trace_log.txt

I removed many lines. I think, the last ones are interesting. At the end, it seems, that pipenv is looping forever over:

 --- modulename: ansitowin32, funcname: closed
ansitowin32.py(52):         stream = self.__wrapped
ansitowin32.py(53):         return not hasattr(stream, 'closed') or stream.closed

I already spent hours on debugging, but now, I don’t know what to do…

$ pipenv --support

Pipenv version: '2018.11.26'

Pipenv location: '/usr/lib/python3.7/site-packages/pipenv'

Python location: '/usr/bin/python3'

Python installations found:

3.7.2: /usr/bin/python3
3.7.2: /usr/bin/python3.7m
3.6.8: /usr/bin/python3.6m
3.6.8: /usr/bin/python3.6
3.5.6: /usr/bin/python3.5
3.5.6: /usr/bin/python3.5m
3.5.3: /usr/bin/pypy3.5
3.4.9: /usr/bin/python3.4
3.4.9: /usr/bin/python3.4m
2.7.15: /usr/bin/python
2.7.13: /usr/bin/pypy

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.7.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.20.7-200.fc29.x86_64',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Feb 6 19:16:42 UTC 2019',
 'python_full_version': '3.7.2',
 'python_version': '3.7',
 'sys_platform': 'linux'}

System environment variables:

CI_COMMIT_SHORT_SHA
PIPENV_VENV_IN_PROJECT
CI_RUNNER_EXECUTABLE_ARCH
CI_COMMIT_TITLE
CI_JOB_TOKEN
CI_BUILD_REF_NAME
CI_REGISTRY_PASSWORD
CI_RUNNER_TAGS
CI_PIPELINE_URL
GITLAB_CI_PASSWORD
CI_JOB_NAME
CI_SERVER_VERSION
GITLAB_CI
CI_SERVER_REVISION
HOSTNAME
CI_PROJECT_VISIBILITY
CI_DISPOSABLE_ENVIRONMENT
CI_COMMIT_SHA
CI_COMMIT_MESSAGE
CI_BUILD_STAGE
CI_PROJECT_URL
CI_COMMIT_REF_SLUG
CI_SERVER_NAME
CI_RUNNER_VERSION
CI_BUILD_NAME
CI
CI_REGISTRY_USER
CI_PROJECT_ID
CI_PIPELINE_ID
CI_JOB_URL
CI_COMMIT_DESCRIPTION
PWD
GITLAB_FEATURES
HOME
CI_REGISTRY
CI_BUILD_TOKEN
CI_BUILD_ID
GITLAB_USER_NAME
CI_COMMIT_BEFORE_SHA
CI_PROJECT_PATH_SLUG
CI_API_V4_URL
GITLAB_USER_EMAIL
CI_COMMIT_REF_NAME
CI_SERVER_VERSION_PATCH
CI_REGISTRY_IMAGE
CI_PIPELINE_IID
DISTTAG
CI_SERVER_TLS_CA_FILE
CI_RUNNER_ID
FGC
CI_SERVER
CI_JOB_ID
CI_REPOSITORY_URL
GITLAB_USER_LOGIN
CI_RUNNER_REVISION
CI_CONFIG_PATH
CI_PROJECT_NAME
CI_SERVER_VERSION_MINOR
CI_SERVER_VERSION_MAJOR
FF_K8S_USE_ENTRYPOINT_OVER_COMMAND
CI_NODE_TOTAL
SHLVL
CI_RUNNER_DESCRIPTION
NPM_TOKEN
CI_PROJECT_PATH
GITLAB_USER_ID
CI_BUILD_BEFORE_SHA
CI_BUILD_REF
PATH
CI_PIPELINE_SOURCE
CI_PROJECT_NAMESPACE
CI_PROJECT_DIR
CI_JOB_STAGE
CI_BUILD_REF_SLUG
_
OLDPWD
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

PIPENV_VENV_IN_PROJECT: 1

Debug–specific environment variables:

PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD: /

Issue Analytics

State:
Created 5 years ago
Reactions:1
Comments:8 (3 by maintainers)

Top GitHub Comments

1reaction

sedrubalcommented, Feb 17, 2019

FYI pipenv run manage.py check --fail-level=INFO may not work

Thanks for that. In fact it works, when you add a shebang to manage.py and make it executable. But maybe this is not intended 😄

Unfortunately it does not work with any of the variables exported:

ANSI_COLORS_DISABLED=1
VISTIR_DISABLE_COLORS=1
PIPENV_COLORBLIND=1
TERM=dumb
PY_COLORS=0

The colors are of course disabled, when running pipenv with a TTY. But the bug as described above still exists.

As workaround I’ll add an unset CI to my scripts:

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 5205b24..b3c08ff 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -2,6 +2,8 @@

 variables:
   PIPENV_VENV_IN_PROJECT: 1
+  # See https://github.com/pypa/pipenv/issues/3534:
+  CI: ""

 stages:
   - setup

0reactions

techalchemycommented, Mar 18, 2019

Hrm, that’s actually helpful though. Up and down the stack pipenv and https://github.com/sarugaku/vistir (which is what we use for the spinner library) are sensitive to the CI variable which toggles colors on and off

The error logs were even more helpful, I think this was a bug with the old stream wrapper we used to wrap the stderr and stdout streams both in pipenv and in the resolver subprocesses. On the master branch (also in vistir now), we have switched over to a different implementation which provides an uncloseable stream wrapper which I believe was the source of the problem. Interestingly I was not able to reproduce this using Ubuntu-based images but I was successful using fedora based images.

Using the following dockerfile I managed to reproduce the problem, which I then resolved simply by installing a wheel built from the master branch of pipenv – I simply uncommented the specified line:

FROM fedora as test-container
WORKDIR /srv/
RUN set -ex && dnf -y install pipenv python3-tox procps-ng

COPY Pipfile .
COPY pipenv-2018.11.27.dev0-py3-none-any.whl .

# Uncomment this line and things work
# RUN set -ex && python3 -m pip install --upgrade pipenv-2018.11.27.dev0-py3-none-any.whl tox
ENV CI true
ENV PIPENV_VENV_IN_PROJECT 1

# Install python requirements
RUN set -ex && setsid sh -c 'tty; ps -jp "$$"; env; pipenv install' < /dev/null > log 2>&1
RUN set -ex && ps -aux
RUN set -ex && tail log

My pipfile for this build contains:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
django = "<=2.2.0"

[dev-packages]

[requires]
python_version = "3.6"

With the .dev wheel commented out, the docker container builds forever. If I uncomment that line, the container builds in 32 seconds. I am going to close this out as resolved on master, once we cut the next release you will be able to unset your CI: "" workarounds. Thanks a ton for the debugging help, I don’t think we would have tracked this down without it