Proposal to overhaul the plugin loader subsystem

Dear Chris, dear community,

first things first: Thanks a stack for conceiving and maintaining this excellent package. At #681, we discovered an issue with the current implementation of the plugin loader subsystem and Python 3.11, would like to report more details about it, and, at the same time, make a proposal how to improve the situation. We hope you will like it.

With kind regards, Andreas.

Introduction

https://github.com/caronc/apprise/blob/67070e5932a2716b282f297290e095732bedba16/apprise/plugins/__init__.py#L115-L119

I think that mangling the module namespace in this manner is a bit unfortunate, as, for example, apprise.plugins.NotifyGrowl will have a completely different meaning throughout the program lifecycle. While it is physically a module, it will become a reference to a class object later, which actually was ``apprise.plugins.NotifyGrowl.NotifyGrowl`.

The case against

This has a number of huge disadvantages.

• Humans and machines will have a hard time to reason about the correct location of the code, both at editing and linting time, and at runtime.
• No type checker like mypy will be able to correctly reason about its heritage, so it will defy any future attempts to bring sensible type hinting and checking to the code base. IDEs like PyCharm will get confused as well, as everything what is sorted out at runtime will be invisible to them, and will probably guide the programmer wrongly.
• We are now observing that it is already difficult to make the right decision when aiming to mock specific parts for test cases or other purposes. It is a burden for the programmer, and unittest.mock may get confused as well.
• Manipulating Python intrinsics like globals() or __all__, should only be made in emergency situations. I think such manipulation techniques should not be within the core of Apprise, nor its plugin loader subsystem.

Why does it only croak on Python 3.11?

I think the reason why this pops up on Python 3.11 might be recent performance improvements like ^1, which will make the situation even worse to reason about throughout the lifetime of a program.

PEP 659: Specializing Adaptive Interpreter

PEP 659 is one of the key parts of the faster CPython project. The general idea is that while Python is a dynamic language, most code has regions where objects and types rarely change. This concept is known as type stability.

At runtime, Python will try to look for common patterns and type stability in the executing code. Python will then replace the current operation with a more specialized one. This specialized operation uses fast paths available only to those use cases/types, which generally outperform their generic counterparts. This also brings in another concept called inline caching, where Python caches the results of expensive operations directly in the bytecode.

Proposal

Therefore, I am hereby politely asking to improve the plugin loader subsystem, and to remove such tricks from its core.

It’s here that makes it so to add/remove a service to apprise simply involves dropping a new .py file in place in the /plugins path.

I hear you, and I will of course try to keep that feature. However, I think it would be easier to think in terms of packages and module namespaces instead of paths - both for humans and the machine.

Please let me know if you think that loading Python files from arbitrary paths is an absolute must: It can be made possible, but it would not be standardized in any way and difficult to test with mocking, because for that, you would need to know a stable dotted-name reference to the class. Unfortunately, an arbitrary path does not have such a property. For simple modules, it can be faked by adding it to a synthetic module namespace, like apprise.contrib, but I think this should not be mixed with the builtin plugins, which are definitively first class citizens and deserve to have a stable reference, both at import- and at runtime.

In that manner, I think it will be better to make loading plugins by module a first citizen, have all Apprise-builtin plugins addressed coherently by module namespace instead of “by path”, and make loading plugins by path more like a second citizen, because of its intrinsic difficulties.

Talk is cheap, I will check if I can submit a proposal in the form of a patch, if you don’t have any general objections.

[amotl]

I think it is safe to close this issue, but it can by reopened anytime we want to continue the discussion about the general plugin loader architecture. Thank you very much for sharing all the important and valuable insights, Chris. I appreciate that very much.

[caronc]

Well written up; one of the new adaptations is the @notify decorator allowing people to write their own hooks and no longer be limited to just the ones provided by Apprise. A hook could write to an SQL Database, or perform some other personal task for a dev. The @notify decorator is built on the dynamic ability to create modules at run-time as well.

I think that mangling the module namespace in this manner is a bit unfortunate, as, for example, apprise.plugins.NotifyGrowl will have a completely different meaning throughout the program lifecycle. While it is physically a module, it will become a reference to a class object later, which actually was apprise.plugins.NotifyGrowl.NotifyGrowl.

I don’t entirely agree with this; I’d say that this is where the beauty of Python offers us the very functions (well documented) to load modules at runtime. It documents the name-spacing very well too. I love the apprise.contrib idea!

# Consider 
# /dir1/dir2/__init__.py
# /dir/__init__.py
# main.py

./python main.py

Assume:

# dir1/dir2/__init__.py
class bob:
    pass

# ---
# dir1/__init__.py
from dir2 import bob

# ---
# main.py
#  # the below logic works despite the name-spacing of bob being 2 levels deep
from dir1 import bob

Honestly though, I’m okay with acknowledging any issues you found with the namespacing though. If there is a way to improve it, I’m all ears! 😃

Talk is cheap, I will check if I can submit a proposal in the form of a patch, if you don’t have any general objections.

Not at all; It’s been great having you around, i truly value your opinion (as others)! 🙂 I’ll be interested in what you propose. I’ll be the first to admit that I could do things better. Up until now this was one thing i actually liked about Apprise (the dynamic loading of module elements), but I’m really excited to see what you come up with.

I would add that one other thing: since the modules are loaded dynamically, they can be globally turned off and off (once loaded) as well. I don’t think what your suggesting would cause an issue here; but it’s just a heads up. For example while cryptography may not be available in your environment, the only result of this is the Notifications (in the /plugins/ directory) that depend on it would still load, but be turned off (for reporting purposes which i’ll explain)… The modules themselves can illustrate the reason they’re disabled through the CLI and web users who are generating their integration to Apprise dynamically through the apprise.details() function. (not sure if this makes sense, hopefully it does).

I would say my only caveat is we need to remain compatible with Python 3.6 for a little while… at least until RHEL8 is EOL.

Edit: my grammar sucks when i type without proof-reading. I just tidied up a little bit of what i wrote.