Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Plugin Output Proposal

See original GitHub issue

Plugin outputs are a way for values to be passed between plugins.

These outputs will allow for more advances use cases of plugins that depend on values from each other.

For example, a mongodb-atlas-plugin might pass a connection string back as an output that a custom-rest-api-function-plugin might use to automatically scaffold a serverless function with said connection string.

To achieve a feature like this, Netlify Build requires additional configuration from the plugin author so outputs and build order can be correctly executed.

Considerations

Plugin outputs must work in these 2 scenarios:

Plugins defined in netlify config file
Plugins being programmatically invoked (“orchestrator plugins”)

Below is a proposal on how these two can work.

1. Plugins defined in netlify config file

Here we use 2 plugins. pluginOne has required inputs and pluginTwo’s output values are used in pluginOne.

// Has required config property
module.exports = function pluginOne(pluginConfig) {
  return {
    // Name of plugin
    name: 'plugin-one',
    // Config needed for plugin
    config: {
      biz: {
        type: 'string',
        required: true,
      }
    },
    // Lifecycle functions to run
    onBuild: ({ pluginConfig }) => {
      console.log(pluginConfig.biz)
    },
  }
}

// Exposes 1 output during onPreBuild phase
module.exports = function pluginTwo(pluginConfig) {
  return {
    // Name of plugin
    name: 'plugin-two',
    // Outputs returned for DAG resolution
    outputs: {
      foo: {
        type: 'string',
        // `when` is important for DAG resolution
        when: 'onPreBuild'
      }
    },
    // Lifecycle functions to run
    onPreBuild: (pluginAPI) => {
      // Output value `foo` is returned from onPreBuild phase
      return {
        foo: 'hello'
      }
    },
  }
}

# Netlify Config
build:
  publish: build
  functions: functions

plugins:
  - package: plugin-one
    config:
      biz: ${pluginTwo.outputs.foo}
      # ^ output referenced from `id`
      # given to plugin-two
  - package: plugin-two
    id: pluginTwo

As an aside, the configuration could also be ordered like so:

# Netlify Config
build:
  publish: build
  functions: functions

plugins:
  - package: plugin-two
    id: pluginTwo
  - package: plugin-one
    config:
      biz: ${pluginTwo.outputs.foo}

What happens on build:

Inputs & outputs dependancies are ordered up front via a DAG and cycles throw an error.

Required inputs validated in plugin-one. Output syntax is recognized, the when value is read & lifecycle is checked to verify ordering works.
Order works, so onPreBuild functionality from plugin-two runs
Then output foo from plugin-two is returned. ${pluginTwo.outputs.foo} then fully resolved to hello.
onBuild functionality from plugin-one runs with it’s biz config value set to hello
Then build ends

2. Plugins used programmatically

Here we use 3 plugins. The plugin-one & plugin-two are programmatically used in a third plugin called orchestratorPlugin. Because they are programmatically used, the order of lifecycle methods in plugin-one & plugin-two do not matter. Effectively they are used as normal NPM modules.

Programmatic usage from orchestrator plugin. Because these are used programmatically and are not defined in the Netlify config file, they can be called in any order the user wishes.

const pluginOne = require('plugin-one')
const pluginTwo = require('plugin-two')

module.exports = function orchestratorPlugin(config) {
  return {
    // Name of plugin
    name: 'orchestrator-plugin',
    // Config needed for plugin
    config: {
      optOne: {
        type: 'string',
        required: true,
      },
      optTwo: {
        type: 'string',
        required: true,
      }
    },
    // Lifecycle functions to run
    onInit: async ({ pluginConfig }) => {
      // initialize plugins with config
      const one = pluginOne({ biz: pluginConfig.optOne  })
      const two = pluginTwo({ zaz: pluginConfig.optTwo })

      const [outputFromOne, outputFromTwo] = await Promise.all([
        one.onPreBuild(pluginAPI),
        two.onBuild(pluginAPI),
      ])
      // Do custom stuff with outputFromOne / outputFromTwo
    },
  }
}

module.exports = function pluginOne(pluginConfig) {
  return {
    // Name of plugin
    name: 'plugin-one',
    // Config needed for plugin
    config: {
      biz: {
        type: 'string',
        required: true,
      }
    },
    // Outputs returned for DAG resolution
    outputs: {
      wow: {
        type: 'string',
        when: 'onBuild'
      }
    },
    onPreBuild: ({ pluginConfig }) => {
      console.log(pluginConfig.biz)
      return {
        wow: 'nice'
      }
    },
  }
}

// Has required config property
module.exports = function pluginTwo(pluginConfig) {
  return {
    // Name of plugin
    name: 'plugin-two',
    // Config needed for plugin
    config: {
      zaz: {
        type: 'string',
        required: true,
      }
    },
    // outputs returned for DAG resolution
    outputs: {
      wow: {
        type: 'string',
        when: 'onBuild'
      }
    },
    onBuild: ({ pluginConfig }) => {
      console.log(pluginConfig.zaz)
    },
  }
}

build:
  publish: build

plugins:
  - package: orchestrator-plugin
    config:
      optOne: hello
      optTwo: goodbye

What happens on build:

Because they are programmatically used, the order of lifecycle methods in plugin-one & plugin-two do not matter.

Required inputs validated from read config file
orchestrator-plugin loads & onInit functionality runs with config set to optOne: hello & optTwo: goodbye
plugin-one & plugin-two are initialized with config
Then plugin-one.onPreBuild & plugin-two.onBuild methods are called
Then output from plugin-one.onPreBuild & plugin-two.onBuild are referenced in the code of orchestrator-plugin.
Then the build ends

Implementation Proposal

The way values are resolved and lifecycle events are ordered depend on the inputs & outputs of each event listener.

There are probably multiple ways to implement something like this. It would be important to note how mature tools like terraform & cloudformation use DAG as their mechanism for resolving & ordering operations.

I’m proposing we use a DAG algorithm to resolve the order in which plugin lifecycle methods fire.

The DAG will tell us the order in which the build steps should happen. It will also be able to detect cycles and throw an error if plugin outputs don’t exist yet where they are being used as plugin config (inputs).

The DAG implementation really only effects plugins that are defined in the Netlify configuration file vs the plugins that are called programmatically (because they are just function calls & resolve themselves in user code)

Proposed Resolution flow:

Related issues

older outputs example https://github.com/netlify/build/pull/494
older syntax discussion https://github.com/netlify/build/issues/396
orchestrator plugins https://github.com/netlify/build/issues/603
https://gist.github.com/DavidWells/d0d0621a3f49c6a0a17aa6d6963de10b

Feedback and comments welcome!

Alternative suggestions on how to get this working would also be interesting to see.

Issue Analytics

State:
Created 4 years ago
Comments:17 (17 by maintainers)

Top GitHub Comments

2reactions

erquhartcommented, Jun 11, 2020

We can address the entire concern of plugin outputs when and if it’s determined they’re necessary. As mentioned in #1177 there are already ways to accomplish inter-plugin communication. We can close and start fresh if this is revisited in the future.

1reaction

ehmickycommented, Jun 2, 2020

There was lots of back and forth between different proposals related to this feature. We finally decided to postpone this feature for the time being, or until it was requested by users. @erquhart might also have some input on this.