Questions about development environment

@spyder-ide/core-developers, I have some questions about the recommended development environment (see CONTRIBUTING.md).

Spyder’s recommended development environment (spyder-dev) consists of packages installed from requirements/conda.txt, which include release versions of python-lsp-server, spyder-kernels, and qdarkstyle. Then bootstrap.py installs python-lsp-server from external-deps directory in develop mode and the executable is installed in a dedicated directory. Rather than installing spyder-kernels and qdarkstyle in develop mode, bootstrap.py adds them to sys.path. Additionally, rather than install spyder in develop mode, bootstrap.py manually adds plugin entry points and a fake spyder distribution to pkg_resources and adds the repo to sys.path.

So, my questions are

1. Why install python-lsp-server in develop mode but not spyder-kernels or qdarkstyle?
2. Why install pylsp executable in a dedicated directory instead of the standard installation location?
3. Why not install the local spyder repo in develop mode?

Since spyder-dev is a dedicated development environment, wouldn’t it be better practice to install pylsp in a standard location, and install spyder-kernels and qdarkstyle in develop mode as well? Then sys.path does not have to be tampered with, which may be less robust than installing in develop mode. Additionally, note that bootstrap.py redundantly adds python-lsp-server to sys.path since setuptools adds it during develop mode installation. Also, wouldn’t installing spyder in develop mode would be a more robust way to ensure that all the entry points are made via the egg-info?

[ccordoba12]

I missed that this one was fixed by PR #17408, so I’m going to close it.

[CAM-Gerlach]

I’m sorry, but I’m strongly -1 on changing the bootstrap.py behavior, even if it implies more maintenance burden (which I’m willing to do).

For better or worse, you’re not going to have much choice soon than to make at least a lot of these changes (or even more complex/hacky ones), since from a packaging perspective a number of the things the script and lsp utils is currently doing have been discouraged and deprecated for many, many years (thus it is already generating DeprecationWarnings) and are on track to be removed very soon. Furthermore, the hacks used here won’t, in general, work with the modern PEP 517/pyproject.toml-based ecosystem, and could cause much more serious breakage to the environment as a whole than doing things the recommended way.

1. Tampering with the environment to install a bunch of stuff in development mode doesn’t work well with conda, which can lead to a broken env quite easily (e.g. conda can uninstall all spyder-kernels dependencies because that package is not in the env).

TL;DR: What you’re doing now is strictly more dangerous, both with and without Conda, than what is proposed here, and most of it has been deprecated for years and will break completely very soon.

I don’t really follow your reasoning here, as it applies much more strongly to the manual “tampering” with the env via hacks previously used to try to fake a package install than just doing a proper pip install -e --no-deps. While the interaction can get complicated if not handled carefully, Conda does recognize and generally respect pip-installed packages, whereas if it isn’t installed at all, of course it will not.

Even so, it doesn’t uninstall things it is not told to except in specific user-commanded circumstances, and certainly not if the dependencies have actually been specified in the conda requirements.txt (i.e. they were user-specified package specs passed to conda install).

Furthermore, editable installs with pip (and occasionally even tightly coupled dependencies as well) is the standard way packages are developed with conda, and is what I’ve done in dozens of other environments for years now without any of these problems. Could you be more specific here about what particular issues you’ve faced, and with what environment setup specifically?

To the contrary, I’ve heard and seen firsthand all the things that can go wrong with manually trying to hack sys.path, use long-deprecated Egg functionality, rely on executing setup.py directly, legacy Setuptools develop install, etc. And again, all or almost all of those things are actively being removed very soon and will break completely in the very near future, so you’ll have to do what we suggest anyway (or come up with an even more complicated and hackier alternative).

People use bootstrap.py to simply run and test the latest Spyder, not to develop it. In that case, I see no need to leave their envs in very bad shape.

We are already telling them to create an environment specifically for this, which is the rule with conda, having them install our deps in that environment, and then hacking that environment to install PyLSP. Their environment is going to be in better shape (such that they can actually run Spyder by the standard mechanisms) with this script than doing aweful manual hacks that will break soon.

This would make contributing to Spyder more difficult because newbies don’t understand what development mode is and works, and how it can affect their envs.

*editable mode; it only affects the env we specifically told them to create for Spyder development, and otherwise at a basic level, has essentially the same user-facing behavior as the existing bootstrap script, just without all the aweful, likely to break hacks behind the scenes, and with more functionality if they choose to use it. Furthermore, this is the install method that essentially every single other package uses for development use, so it adds more confusion that we’re using bespoke hacks that don’t work as people expect.

@mrclary

Indeed, python-lsp-server does need to be installed; the question was why not also spyder-kernels and qdarkstyle.

Because, as @ccordoba12 mentioned, the LSP entry points needed to be available. However, regardless, I still agree that since we’re modifying the user’s env anyway with this, we may as well do it for the others, and do it properly and not use horrible hacks that are more likely to break things.