RFC separating plugins from nornir "core"

This is a request for comments from the community.

As the project grows the community contributes plugins for different use cases. This is great and it’s certainly good news for the community but it puts a burden on the core maintainers. Some of these plugins relate to systems which we, the core maintainers, are no experts of so review is slow and, in case of bugs/issues, we can’t help much about it.

For this reason I am proposing the following:

1. Ship nornir without any plugins whatsoever, just the engine framework.
2. Plugins will live in their own repo by theme. For instance, nornir-napalm may contain the connection plugin for napalm + its tasks. nornir-netmiko may contain the connection plugin for netmiko + its tasks. nornir-netbox, the inventory plugin for netbox, etc…

This means that if you want to use nornir with netmiko and netbox as the inventory you would have to install both nornir, nornir-netmiko and nornir-netbox. For instance:

pip install nornir nornir-netmiko nornir-netbox

This has a few extra benefits:

1. Installation footprint is as small/big as you need it to be. For instance, no need to install napalm and its tons of dependencies if you don’t plan to use it.
2. It sets a pattern where people can own the plugins under their own organization as it’s not required to live in the nornir repo. As a matter of fact this is not necessary today either but the pattern is to contribute everything to the core repo.
3. Plugins will be able to be developed/iterated faster as they will not depend on nornir’s release schedule

This has two main issues; overhead of having to package everything and discoverability.

To solve those issues we will:

1. Maintain a directory where people can submit information about their plugins. This information will be rendered alongside the docs.
2. Maintain a cookiecutter repo contributors can leverage to get all the boilerplate fixed for them.

As the “core maintained” plugins will be separated from the “core” nornir repo those plugins will leverage (1) and (2) as well ensuring there are no first and second class plugins here.

[ktbyers]

Yes, I think it is a good idea.

Looking at both NAPALM and Ansible, the burden of contributed code starts to become a huge issue and creates big conflicts on either side (i.e. maintainers are frustrated that they can’t manage the amount of work and contributors are frustrated that fixes/improvements don’t happen).

I think we should probably consider some of the issues we ran into when we broke the napalm project into separate repositories. In other words, we probably need some way to declare plugin-core version so we don’t have to try to make core and plugins move in sync in some way (which basically becomes impossible). Which maybe is some meta information that both core contains and the plugin must have like nornir_plugin_version where Core lists a range of supported plugin versions and the individual plugin indicates the version it supports. With the expectation that Nornir Core would support a range.

We should also think more about plugin testing and potentially have boilerplate for testing (that is just included in the cookiecutter in some way).

I wonder if for discoverability we want a bit more. Might be interesting to research any project that has done this well. Potentially meta information on the plugin side that could be pulled in/read via tooling so that if there was a command-line tool you could list the plugins and other information. Anyways might be worth researching more.

[dbarrosop]

1. Regarding breaking changes and plugins requiring different versions of nornir. We follow semver rules so we don’t do breaking changes, ever, unless there is a major release. So as long as the plugins pin a major version everything should be good.
2. Re plugins sprawl and bad quality, this is something that can happen today. There is nothing stopping you from writing your own plugins and publishing them somewhere so what we are saying here is that we will move out of the core repo plugins that some of the core maintainers of nornir will maintain anyway. In any case, this is a valid concern and thanks for bringing it up, thinking about it we could enforce quality via the central directory:
   1. If you want to add a plugin to the central directory we will review and make sure it’s documented and tested
   2. The documentation will be pulled and merged into nornir’s own documentation as part of the documentation build process
   3. The tests will be run as part of the documentation build and add a “coverage” metric

Hopefully this will help ensuring a minimum amount of quality in the plugins that we refer to in our docs, or at least give you a sense of how much you can/want to trust it. Obviously, if they are not there we can’t do much about it but that’s no different from today.