Current release 47.1.1 breaks supervisor 3.2.0: UserWarning: Setuptools no longer works on Python 2

Prerequisites

• Python 2 is not required for this application.
• I maintain the software that installs Setuptools (if not, please contact that project).
• Setuptools installed with pip 9 or later.
• Pinning Setuptools to setuptools<45 in the environment was unsuccessful.

Environment Details

• Operating System and version: Ubuntu 16.04.5 LTS
• Python version: Python 3.5.2
• Python installed how: apt-get
• Virtualenv version (if using virtualenv): n/a

Command(s) that triggered the warning/error (and output):

root@a84144be0db1:/# python --version
Python 3.5.2
root@a84144be0db1:/# supervisord
/app/python/pkg_resources/py2_warn.py:15: UserWarning: Setuptools no longer works on Python 2
************************************************************
Encountered a version of Setuptools that no longer supports
this version of Python. Please head to
https://bit.ly/setuptools-py2-sunset for support.
************************************************************
  warnings.warn(pre + "*" * 60 + msg + "*" * 60)

Command(s) used to install setuptools (and output):

We use apscheduler which in turn installs the latest setuptools version.

Collecting setuptools>=0.7 (from apscheduler==3.5.3->sensei_sdk==3.0.2)
  Downloading https://artifactory.corp.adobe.com/artifactory/api/pypi/pypi-sensei-framework-release/packages/packages/95/95/f657b6e17f00c3f35b5f68b10e46c3a43af353d8856bd57bfcfb1dbb3e92/setuptools-47.1.1-py3-none-any.whl (583kB)

Output of pip --version when installing setuptools:

pip 19.0.3 from /usr/local/lib/python3.5/dist-packages/pip (python 3.5)

Other notes

This potentially breaks other tools.

[jaraco]

Aha. Yes!

So if I understand correctly, supervisord 3 only runs on Python 2, so even though you’re installing it with apt-get, you’re getting a version that relies on Python 2.

When you install apscheduler, you’re installing that on Python 3.

So far, so good. These systems can interoperate in separate processes (i.e. through subprocess() or through RPC/HTTP calls). The problem becomes if you try to bind them at the library level…

And that’s exactly what will happen if you set PYTHONPATH. PYTHONPATH is honored by all python processes, on Python 2 or 3… and will try to use libraries from one Python environment in another. A simpler example:

draft $ python3 -m pip install -t py3libs setuptools
Collecting setuptools
  Using cached setuptools-47.1.1-py3-none-any.whl (583 kB)
Installing collected packages: setuptools
Successfully installed setuptools-47.1.1
draft $ env PYTHONPATH=py3libs python2 -c 'import setuptools'
/Users/jaraco/draft/py3libs/pkg_resources/py2_warn.py:15: UserWarning: Setuptools no longer works on Python 2
************************************************************
Encountered a version of Setuptools that no longer supports
this version of Python. Please head to
https://bit.ly/setuptools-py2-sunset for support.
************************************************************
  warnings.warn(pre + "*" * 60 + msg + "*" * 60)

In this example, I’ve created a py3libs directory and installed setuptools to it. I’ve then added that target, which could contain any number of libraries that only run on Python 3, to PYTHONPATH and run Python 2. That’s where I’ve basically said, “Python 2, please run with these Python 3 libraries.” Obviously, not a great idea, and no surprise it fails. Here’s an example with another package:

draft $ python3 -m pip install -t py3libs more_itertools
Collecting more_itertools
  Using cached more_itertools-8.3.0-py3-none-any.whl (44 kB)
Installing collected packages: more-itertools
Successfully installed more-itertools-8.3.0
draft $ env PYTHONPATH=py3libs python2 -c 'import more_itertools'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/jaraco/draft/py3libs/more_itertools/__init__.py", line 1, in <module>
    from .more import *  # noqa
  File "/Users/jaraco/draft/py3libs/more_itertools/more.py", line 482
    yield from iterable
             ^
SyntaxError: invalid syntax

It’s just not going to work to have a Python 2 environment pointing at a directory with Python 3 libs, and this doesn’t have anything to do with Setuptools specifically.

Probably your easiest workaround here is when you’re installing apscheduler, just also install setuptools<45. It’s still Python 3 compatible and is probably suitable for anything you’re doing.

draft $ python3 -m pip install -t py3libs 'setuptools<45' apscheduler
Collecting setuptools<45
  Downloading setuptools-44.1.1-py2.py3-none-any.whl (583 kB)
     |████████████████████████████████| 583 kB 1.3 MB/s 
Collecting apscheduler
  Downloading APScheduler-3.6.3-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 6.0 MB/s 
Collecting tzlocal>=1.2
  Downloading tzlocal-2.1-py2.py3-none-any.whl (16 kB)
Collecting six>=1.4.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting pytz
  Using cached pytz-2020.1-py2.py3-none-any.whl (510 kB)
Installing collected packages: setuptools, pytz, tzlocal, six, apscheduler
Successfully installed apscheduler-3.6.3 pytz-2020.1 setuptools-44.1.1 six-1.15.0 tzlocal-2.1
draft $ env PYTHONPATH=py3libs python2 -c 'import setuptools' && echo 'works!'
works!

That should buy you the time until you can migrate to supervisord 4 and bid Python 2 a well-deserved farewell.

Another option might be to unset PYTHONPATH when invoking anything around supervisord. That will avoid injecting the libraries into the supervisord environment.

Do either of those techniques help?

[McSinyx]

So, and you are sure your change made for 47.1.1 is valid and does not break other things?

Again, I am just a concerning user and a seldom contributor happening to hang around this issue tracker and IMHO if a Python 3 program uses Python 2 setuptools that’d be the program’s bug. Based on your previous investigation, for example supervisor fixed that such bug somewhere between version 3.2.0 and 4.2.0.

Indeed version 47 is a breaking change for Python 2 but there’s been version 45 and 46 for the deprecation period. I don’t speak for setuptools but I don’t think it’s its job to support something that’s already declared sunset nor questionable/buggy behaviors. In case Python 2 compatibility is desired, one can always pin to setuptools<45 which AFAIK still receive urgent bug fixes.