ValueError: pickle protocol 4 asked for; the highest available protocol is 2

Observed behavior

While testing https://github.com/learningequality/kolibri/pull/6207 got error related pickle protocols

Expected behavior

no errors

User-facing consequences

unknown

Errors and logs

ERROR    Job 767a126271ec48daabf6d5ff1d1f3129 raised an exception: Traceback (most recent call last):
  File "/Users/d/.pex/install/kolibri-0.13.0b4.dev0+git.7.g1e965215-py2.py3-none-any.whl.1c96f71514c7e69b0d22c18963272db7a4f3d938/kolibri-0.13.0b4.dev0+git.7.g1e965215-py2.py3-none-any.whl/kolibri/core/tasks/worker.py", line 69, in handle_finished_future
    result = future.result()
  File "/Users/d/.pex/install/kolibri-0.13.0b4.dev0+git.7.g1e965215-py2.py3-none-any.whl.1c96f71514c7e69b0d22c18963272db7a4f3d938/kolibri-0.13.0b4.dev0+git.7.g1e965215-py2.py3-none-any.whl/kolibri/dist/py2only/concurrent/futures/_base.py", line 422, in result
    return self.__get_result()
  File "/Users/d/.pex/install/kolibri-0.13.0b4.dev0+git.7.g1e965215-py2.py3-none-any.whl.1c96f71514c7e69b0d22c18963272db7a4f3d938/kolibri-0.13.0b4.dev0+git.7.g1e965215-py2.py3-none-any.whl/kolibri/dist/py2only/concurrent/futures/thread.py", line 62, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/d/.pex/install/kolibri-0.13.0b4.dev0+git.7.g1e965215-py2.py3-none-any.whl.1c96f71514c7e69b0d22c18963272db7a4f3d938/kolibri-0.13.0b4.dev0+git.7.g1e965215-py2.py3-none-any.whl/kolibri/core/tasks/worker.py", line 222, in wrap
    raise e
ValueError: pickle protocol 4 asked for; the highest available protocol is 2

Steps to reproduce

ran pex start command

Context

pex from https://github.com/learningequality/kolibri/pull/6207 on macOS

[rtibbles]

The only reason that it should be asking for a higher pickle version than is available is if there are different Python versions at play - also @micahscopes I assume no one running the pex is using redis, so I feel like it’s not the redis version that is the issue.

Seems more likely to be pickling that’s happening either in diskcache or in SQLAlchemy (for async tasks). The latter seems the most likely, if a task runner running Python2 is still running in the background, but a Python3 server queues something up in the task queue with its pickle protocol.

[indirectlylit]

closing in favor of these three issues:

• https://github.com/learningequality/kolibri/issues/6247 (prevent zombies)
• https://github.com/learningequality/kolibri/issues/6317 (pin pickle protocol)
• https://github.com/learningequality/kolibri/issues/6316 (eventually, stop using pickle)