Feature Request: Orchestrator Plugins

What follows is a collaboration between @jlengstorf and I, asked for by @DavidWells and @ehmicky, to stub out some solutions to #186, and PR #495. We Voltroned out a solution, this is just the tip of the iceberg of what we’re thinking about, so if you like this general concept, we’d love to dive deeper with you.

Netlify Build Plugins — Orchestration and Shared Data

This issue proposes solutions to open questions about Netlify Build Plugins:

1. How can developers ensure that build plugins run in a given order?
2. How can developers share data between build plugins?
3. If plugins are being composed in such a way, how can a maintainer easily tell what’s going on?

We propose adding functionality for an orchestrator plugin.

An example orchestrator plugin:

module.exports = {
  onInit: async ctx => {
    // this needs to be orchestrated in order
    // only executes the onInit lifecycle hook for these plugins
    await ctx.netlify.pluginExecute('netlify-plugin-1')
    await ctx.netlify.pluginExecute('netlify-plugin-2')
    await ctx.netlify.pluginExecute('netlify-plugin-3')
  },
  onPreBuild: async ctx => {
    // this one only has one plugin that needs it
    // only executes the onPreBuild hook for this plugin
    await ctx.netlify.pluginExecute('netlify-plugin-1')
  }
}

NOTE: Orchestrator plugins would not be allowed to do anything other than execute other plugins.

Individual Plugin Usage

The first plugin sets up and caches data required by other plugins:

module.exports = {
  name: 'netlify-plugin-one',
  onInit: ({ cache }) => {
    const thing = 'very important data';
    
    // this is what the cache would store from this
    // this cache would be unique to this plugin
    /*
      {
        owner: 'netlify-plugin-one', // auto-set by Netlify
        lifecycle: 'onInit', // auto-set by Netlify
        data: {
          thing: 'very important data'
        }
      }
     */
    cache.create({ thing })
  },
};

The second plugin runs after the first:

module.exports = {
  name: 'netlify-plugin-two',
  onInit: ({ cache }) => {
    if (!cache.data) return // basic error handling

    // or we can also be more specific to the first plugin
    if (cache.data.owner !== 'netlify-plugin-one') return
    
    const pluginOneStuff = cache.data.thing
    // do things with pluginOneStuff here

  }
};

---

More Details

The pluginExecute method

The API for pluginExecute that was shown in the orchestrator might look something like this:

pluginExecute(
  pluginName,   // string — the package name of the plugin to execute
  {
    lifecycle,  // array — which lifecycle methods to execute
                //   defaults to the lifecycle where the method is called
    config,     // object — any data to make available to the plugin
  }
)

[jlengstorf]

we would need to trigger both the orchestrator plugin and the plugins called by that orchestrator.

I think I’d argue that an orchestrator plugin is a full abstraction of the underlying plugins, so dry runs wouldn’t show underlying plugins being used — they’re an implementation detail at that point

like, I don’t expect the dry run to tell me that a plugin imports lodash or axios — an orchestrator turns plugins into underlying libs to accomplish a higher-level goal, so I feel like there doesn’t need to be visibility under the hood here

[jlengstorf]

I think the important thing here is to implement and form an opinion on the pattern. If that doesn’t require writing more code, that’s great!

However, there are some risks with telling people to handle things themselves:

1. It requires a deeper knowledge of the underlying code that powers build plugins, which increases the onboarding difficulty
2. It relies on the discipline of the developers instead of creating guard rails, which has historically resulted in large, hard to manage messes of userland code
3. It opens the door for unintended uses of plugin code, which means downstream updates will be more painful because people did things we didn’t expect them to do, but told them they could do (implicitly, at least) and will therefore need to handle all sorts of unexpected edge cases

The benefits of writing a wrapper that handles plugin execution vs. doing it in userland overcome those challenges:

1. There is a well documented API for running another plugin; no knowledge of how build plugins work to e.g. initialize with config required
2. Writing orchestrators automates discipline because we define what is and is not allowed — we build the guard rails
3. Creating an official API (even if all it does is exactly what @DavidWells wrote in the previous example) means we can make downstream changes and not worry about unsupported use cases as much — if you hack a tool, your hacks require you to make changes to support them — which should decrease our support and maintenance burden going forward

For those reasons, I think a lightweight API is a good idea. It could literally be what @DavidWells is proposing, just wrapped in a function:

function pluginExecute(pluginName) {
  const plugin = require(pluginName);

  const config = getConfigFromSomeInteralThing();
  const context = getContextFromSomeInteralThing();
  const currentLifeCycleMethod = getCurrentLifeCycleMethod();

  const instance = plugin(config);

  return instance[currentLifeCycleMethod](context);
}

This gives us an Officially Supported™ stance and makes it harder to do weird things that makes support suffer later.