Config: Support multiple source extension in any object configuration

Use case description

Currently, environment option in provider and in function level can only accept a single JSON object. It’s better to have multiple object support.

Maintainer’s note:

As we receive requests to support this notation also in other places, it’ll probably be good to solve with one generic solution, instead of hard-coding support for array notation in all needed places one by one.

Proposed solution (Strategy)

1. Universal solution (not specific to configuration file type)

Introduce a merge variable source, so single object or array can be constructed from multiple sources as follows:

custom:
  environment:
    FOO: value

provider:
  environment: ${merge(${self:custom.environment}, ${file(common-env.json)}})}

2. YAML configuration specific

Introduce !merge YAML function, which could work as follows

provider:
  environment: !merge
    - FOO: value
    - ${file(common-env.json)}

How to implement it? (Tactics)

Note: Solution for both methods should be proposed with two different PR’s

1. Universal solution (not specific to configuration file type)

1. In lib/configuration/variables/sources Configure merge source (it should be accompanied with tests that provide 100% coverage). On how to implement it, some hint could be taken from file source configuration. Still in case of merge I believe we will only need to process params argument. Important constraints to met:
   • Support constructing plain objects and arrays exclusively
   • Imply a top level merge. In plain object case it should work similarly to Object.assign({}, param1, param2, ..). In case of arrays: [].push(...param1, ...param2)
   • In case of plain object construction throw an exception if two input objects share same key (let’s not silently override them)
   • Ensure that all input params are either exclusively plain objects, or exclusively arrays (throw otherwise)
2. Add source to variables resolution in:
   • scripts/serverless.js
   • lib/configuration/variables/index.js
3. Document new variables source in docs/providers/aws/guide/variables.md

2. YAML configuration specific

1. Configure schema extension which adds !merge support to YAML. Hint on how such extension can be configured can probably be taken from @serverless/utils/cloudformation-schema.js which add support for functions as !Ref , !GetAtt etc… Let’s put definition of new YAML types to lib/configuration/yaml-merge.js. As items passed to !merge may be constructed with Serverless Variables, which cannot be resolved at the point of YAML resolution. We need to convert this notation into merge variable source (implemented at step 1). Additionally as merge source cannot accept inline object or array structures, we need to build some temporary collection attached directly to configuration, in which we can store items, and which we can address in merge sources. To achieve that I propose that:
   1. lib/configuration/yaml-merge.js when resolving the !merge constructs, builds in a memory a temporary object with all referenced items, and in merge variable sources address them via ${self:_yamlMerge.<configuration-path>}, e.g. if we list item that was configured at foo.bar[0] property, then we should address it in merge source param as ${self:_yamlMerge.foo.bar[0]} (Reason for preserving same paths, it to ensure meaningful error if e.g. variable resolution fails at those places)
   2. In lib/configuration/read.js once YAML is resolved into JSON, we should retrieve temporary object from lib/configuration/yaml-merge.js and add it to result configuration at _yamlMerge property. Additional notes:
      1. If there were no !merge tags, no _yamlMerge object should be created
      2. Once temporary object is retrieved from lib/configuration/yaml-merge.js, it should be discarded in context of lib/configuration/yaml-merge.js (so e.g. repeated configuration reads, do not build up on same temporary object)
   3. In scripts/serverless.js (before we do serverless.run()), delete the _yamlMerge property from the configuration
2. Let’s ensure YAML schema is extended with new types at: lib/configuration/read.js
3. Document this YAML extension in docs/providers/aws/guide/variables.md

[medikoo]

@glb great thanks for linking that. That looks very neat. Indeed maybe then it’ll be better to do it as follows:

environment:
  FOO: someVal
  !merge:
    - ${self:custom.obj1}
    - ${self:custom.obj2}

What do you think?

Note: I’ve put it slightly differently, as I think notation they propose (in rules: !merge example), will not really work, as it doesn’t seem as valid YAML structure (but I didn’t tested that through)

[medikoo]

@sajithneyo great thanks for proposal!

It definitely makes sense to support input from many sources there.

Still we have many places like that, and I wonder whether we can come up with some global solution that will allow us to use it everywhere, and not address each property like that individually.

It’ll probably have to be some variables resolution extension