In bundled python package installation is performed in user location

Problem

When the download and install .deb package and try run pip install using pip from bundled python there is information that the package will be installed in user mode because site-package is not writable.

Packages installed in user mode will be available in all python3.8 environments and may introduce unexpected side effects.

pip bundled in app should use different location to install a new packages.

Proposed Solution

set alternative install location in user directory when run in non administrator mode and add this location to PYTHONPATH

[jpivarski]

I’m watching this issue, too, because it’s the one capability that’s necessary to make jupyterlab_app useful. (What’s Python without packages?)

As I understand the current status,

• running pip install X in the JupyterLab terminal or in a notebook cell starting with ! runs pip in some global mode, putting the installed package in ~/.local, rather than in the conda environment
• running conda install X in the JupyterLab terminal or in a notebook cell starting with ! doesn’t work at all because the install area is not writable.

Oddly, considering this error, the pip and conda commands are coming from the right place (this output came from the JupyterLab terminal; the output from a notebook cell is the same):

$ which pip
/opt/JupyterLab/resources/jlab_server/bin/pip
$ which conda
/opt/JupyterLab/resources/jlab_server/bin/conda

Aha! The problem is that the site-packages is not writable, and so that’s why pip is deferring to ~/.local:

$ pip install uproot
Defaulting to user installation because normal site-packages is not writeable
Collecting uproot
  Using cached uproot-4.1.2-py2.py3-none-any.whl (290 kB)
Requirement already satisfied: numpy in /opt/JupyterLab/resources/jlab_server/lib/python3.8/site-packages (from uproot) (1.21.2)
Requirement already satisfied: setuptools in /opt/JupyterLab/resources/jlab_server/lib/python3.8/site-packages (from uproot) (58.0.4)
Installing collected packages: uproot
Successfully installed uproot-4.1.2

We see where it is when uninstalling the package:

$ pip uninstall uproot
Found existing installation: uproot 4.1.2
Uninstalling uproot-4.1.2:
  Would remove:
    /home/jpivarski/.local/lib/python3.8/site-packages/uproot-4.1.2.dist-info/*
    /home/jpivarski/.local/lib/python3.8/site-packages/uproot/*
Proceed (Y/n)? y
  Successfully uninstalled uproot-4.1.2

The error message from conda seems to be from the same underlying cause:

$ conda install uproot
Collecting package metadata (current_repodata.json): failed

NotWritableError: The current user does not have write permissions to a required path.
  path: /opt/JupyterLab/resources/jlab_server/pkgs/urls.txt
  uid: 1000
  gid: 1000

If you feel that permissions on this path are set incorrectly, you can manually
change them by executing

  $ sudo chown 1000:1000 /opt/JupyterLab/resources/jlab_server/pkgs/urls.txt

In general, it's not advisable to use 'sudo conda'.

Interestingly, the conda in JupyterLab_app sees my normal conda environments:

$ conda env info
usage: conda-env [-h] {create,export,list,remove,update,config} ...
conda-env: error: invalid choice: 'info' (choose from 'create', 'export', 'list', 'remove', 'update', 'config')
$ conda env list
# conda environments:
#
                         /home/jpivarski/miniconda3
                         /home/jpivarski/miniconda3/envs/np13
                         /home/jpivarski/miniconda3/envs/np21
                         /home/jpivarski/miniconda3/envs/numba-py38
                         /home/jpivarski/miniconda3/envs/pandas-0.22
                         /home/jpivarski/miniconda3/envs/py27
                         /home/jpivarski/miniconda3/envs/py39
                         /home/jpivarski/miniconda3/envs/rapids
                         /home/jpivarski/miniconda3/envs/test-fastjet
base                  *  /opt/JupyterLab/resources/jlab_server

But the conda I use outside of JuptyerLab_app, in a normal terminal, for instance, does not see the JupyterLab_app one:

% conda env list
# conda environments:
#
base                  *  /home/jpivarski/miniconda3
np13                     /home/jpivarski/miniconda3/envs/np13
np21                     /home/jpivarski/miniconda3/envs/np21
numba-py38               /home/jpivarski/miniconda3/envs/numba-py38
pandas-0.22              /home/jpivarski/miniconda3/envs/pandas-0.22
py27                     /home/jpivarski/miniconda3/envs/py27
py39                     /home/jpivarski/miniconda3/envs/py39
rapids                   /home/jpivarski/miniconda3/envs/rapids
test-fastjet             /home/jpivarski/miniconda3/envs/test-fastjet

If there’s some way now to pip and conda into /opt/JupyterLab/resources/jlab_server, I’d love to know it! If not, this is a bug-fix/feature-request. Thanks!

(Or do I just have to install it not from the .deb? That seemed like the appropriate way to install it, as I’m an end-user, not a developer.)

[Czaki]

@Czaki Ah… seems it is there but I had to restart the kernel before I could see the newly installed packages, whereas typically (for me at least…) packages seem to be available without the need to restart (although a warning that a kernel restart may be required is typically given.)

If I good remember it was caused by installation outside site-packages. I meet this when performing installation in developer mode (pip install -e).