Plugins, & getting Lektor out of the package management business

For a long time now, it has become clearer that having Lektor manage the installation of a project’s Python plugins is a significant pain point for maintainers and users. There are a lot of issues that have popped up over the years, from bugs, to missing features. I and some other maintainers are leaning toward minimizing Lektor’s role in installing and managing a project’s plugin Python packages. @mitsuhiko do we have your go-ahead? Thoughts or concerns?

Regardless of the exact goal chosen, there are the normal concerns about breakages and deprecations in order to prepare users for the changes and not make things unnecessarily hard.

We need to decided what exactly this looks like. For example, this could mean:

Rough option 1) - minimize plugin management:

• Removing the lektor plugins command, and it’s sub-commands
• Removing the use of the [packages] section from project files
• Remove the automated use of packages/ subdirectory
• Give specific and simple guidelines for installing packages in the docs, such as how to use a very basic Poetry setup, or requirements.txt.
• Possibly leave in the ability run a similar command to lektor dev new-plugin, though this would likely be modified, at least to match the changed docs instructions. I don’t see this, isolated, to be very burdensome, and is potentially very helpful.

In my opinion this option is the best, because

• At presents everyone with a clear demarcation of responsibility, and lets a dedicated package manager do what it was made for
• It reduces maintenance burden to the minimum on this issue.

This plan has the con of potentially removing some convenience features for users, such as only having to understand and use a single program (Lektor) instead of a minimum of two (Lektor and pip/poetry/etc). The burden of choice at least can be minimized by clear instructions in our docs.

Rough option 2) - do some level of wrapping of a preferred packaging tool

Should we try to keep some pieces of lektor plugins, such as add, remove, and list, and adapt them to match or wrap a chosen tool like Poetry? Should we leave in [packages] and somehow translate that to invoke a tool like Poetry? Should we auto-install things from packages/? These goals are somewhat independent, and they may offload some maintenance burden, but they clearly add some too. Choosing this path could mitigate some of the drawbacks of the option 1). In my opinion, that is not a good enough tradeoff.

N.B. there are also several discussions about using a different package management strategy internally to maintain this repo. This wouldn’t matter much if we choose option 1), above, but the less we approach option 1), the more it might reduce mental overhead to use the same tool for integrations for projects and to manage this repo itself.

[dairiki]

I’m neutral on the idea of adding a new custom plugin loader.

It would be simple enough (i.e. minimal additional code) to do something like: check a particularly directory for .py files (or just check for a single particular .py file); load all matching files; look through each loaded module for subclasses of lektor.pluginsystem.Plugin and treat each of those as a plugin.

At the same time, once one switches to using a package manager (e.g. Poetry) to manage the Lektor plugins for a particularly Lektor project, it’s really not that complicated to manually configure the appropriate entry point(s) in the top-level metadata file for a project.

E.g. here’s a Poetry pyproject.toml that configures a local plugin (as well as installing lektor-atom from PyPI):

[tool.poetry]
# Lektor currently requires that the dist name be "lektor-<plugin-name>".  This should, IMO, be relaxed.
# If it were, we could name this project "test-demo-lektor-website" or something else
# more descriptive.  (We could also configure more than one local plugin.)
name = "lektor-demo"
version = "0.1.0"
description = "Test"
authors = ["Jeff Dairiki <[e-mail]>"]
readme = "README.md"
#packages = [{include = "demo_plugin.py"}]

[tool.poetry.plugins."lektor.plugins"]
# plugin code is in ./lektor_demo/plugin.py
demo = "lektor_demo.plugin:DemoPlugin"

[tool.poetry.dependencies]
python = "^3.7"
Lektor = "^3.3.7"
lektor-atom = "^0.4.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Note that this works just fine with current versions of Lektor.

[frostming]

At the same time, once one switches to using a package manager (e.g. Poetry) to manage the Lektor plugins for a particularly Lektor project, it’s really not that complicated to manually configure the appropriate entry point(s) in the top-level metadata file for a project.

Oh, good point, I didn’t consider it before. Yes, we can define the local plugins directly in the current project’s metadata and after it is installed, the plugins are activated already. Except that it comes with a minor inconvenience – the lektor project itself must be a PyPI-ready package, which is not usually the case for a website. It seems Poetry ensures that when initializing a project but others like Pipenv may not force that.