Python 3.10 compatiblity

Questions

• Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?

  macOS 11.6, 64-bit
  

• Which version of PyQ are you running? Please provide output of pyq --versions, if PyQ isn’t operational, please provide Python interpreter version and PyQ version python -V; python3 -V; pip list | grep pyq:

  5.00
  

• Which version of kdb+ are you using, is it 32-bit or 64-bit?

  64-bit
  

  • If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows.

    $ env | grep QLIC
    QLIC=/Users/user/.lic
    

• Did you use virtual environment to install PyQ? If not, why?

  • yes

• Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows.

  $ env | grep QHOME
  $
  

• Do you use Conda? If so, what version?

  • no

Steps to reproduce the issue

$ brew install python@3.10
$  /usr/local/opt/python@3.10/bin/python3 -mvenv .virtualenvs/310
$ source .virtualenvs/310/bin/activate
$ python3.10 -mpip install -U pip setuptools wheel
$ unzip ~/Downloads/m64.zip -d "${VIRTUAL_ENV}/q"
$ (unset PIP_NO_INDEX; python3.10 -mpip install pyq)
...
Successfully built pyq
Installing collected packages: pyq
Successfully installed pyq-5.0.0
$ pyq --versions
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py", line 4, in <module>
    from collections import Mapping as _Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)
'2021.10.06T12:38:40.403 python

Expected result

pyq --versions will output versions installed on system

Actual result

$ pyq --versions
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py", line 4, in <module>
    from collections import Mapping as _Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)
'2021.10.06T12:38:40.403 python

Workaround

$ cp -pv /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py{,.orig}
/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py -> /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py.orig
$ vi /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py
$ diff /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py{.orig,}
4c4,7
< from collections import Mapping as _Mapping
---
> try:
>     from collections import Mapping as _Mapping
> except ImportError:
>     from collections.abc import Mapping as _Mapping
$ pyq --versions
PyQ 5.0.0
KDB+ 3.5 (2019.01.31) m64
Python 3.10.0 (default, Oct  6 2021, 01:11:32) [Clang 13.0.0 (clang-1300.0.29.3)]

[b-nln]

There are no plans to release 5.0.1, as there is no full python 3.10 support yet.

[sashkab]

After spending hours earlier this week in order to rewrite Python initialization for 3.10 and failing, I came up with a quick workaround for the importing modules installed in the virtual environment when using Python 3.10 and PyQ:

$ pyq
Python 3.10.5 (main, Jul 25 2022, 15:52:08) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import markdown
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.virtualenvs/3/lib/python3.10/site-packages/pyq/__init__.py", line 820, in __import__
    m = _imp(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'markdown'

Unfortunately, due to Python Initialization changes in 3.10, sys.path is not properly initialized when Python is loaded from kdb+. Dirty workaround is to extend sys.path to what it should have been:

>>> import sys
>>> sys.path.extend(['/home/user/.virtualenvs/3/lib/python3.10/site-packages', '/usr/lib/python3.10/site-packages'])
>>> import markdown
>>> markdown.__version__
'3.3.4'

Basically, first path is your $VIRTUAL_ENV/lib/python3.10/site-packages, and second path is where system python 3 packages are installed.