Feature Proposal: Serverless should support being able to supply path to .yml file to use for configuration

Summary

serverless framework is currently wedded to a few concepts which make it very cumbersome to share javascript code and compiler compilation contexts between many node ‘micro-service’ stacks. Specifically the fact that serverless does not allow specifying a path to the serverless config file pushes every set of cloud-formation stacks deployed by serverless toward needing to be a separate npm package. This requirement is essentially enforced by the framework due to the framework not exposing any mechanism’s for supplying a path to the serverless config file to use for a given serverless invocation.

Problems

Forcing developers toward describing every stack as an npm package adds a lot of overhead to the development process – particularly when its desirable to share code between stacks.

What overhead is added by the need to describe each serverless stack as an npm package? How much of this overhead is incidental complexity vs actually useful modularization? I argue that for many projects/contexts, this requirement adds a large amount of incidental complexity with net negative benefit rather than encouraging useful patterns:

• every serverless deployable stack needs to setup and define its tooling environment. Modern tooling requirements for javascript codebases are extensive (compilers/transpilers, test runners, linters, ide integrations). Re-use of these processes between related projects is enormously beneficial. It seems to be true that in general – the fewer the number of packages for which you have to setup this tooling when working on related code, the better the development experience you can arrange for your team.

• every serverless deployable stack needs to describe full set of dependencies, devDependencies and needs to separately install those dependencies in order for the serverless commands to be runnable. Full-encapsulation of development environment (rather than requiring the installation of tools globally) into package.json is very nice – so if you wish to package all the required development dependencies then every stack deployment context ends up needing to install its own copy of the serverless framework. If you are a using compiler/linter, every stack is also going to need its own copy of the compiler and linter (and configuration of those). Every stack deployment context is going to need to manage versions of these things, is going to have to install these things when setting up the development context for each stack and when running automated tests for each stack.

• its very difficult to share code between stacks. Simple solutions for sharing code between stacks that rely on node’s module resolution algorithm to organize shared dependencies won’t work. Furthermore, ‘normal’ techniques for easing development of related sets of npm packages won’t work well with serverless because serverless doesn’t handle packaging of symlinked node dependencies in a good way.

  • tools like npm linklerna/yarn-workspaces which aim to ease development of related sets of npm packages also don’t work well with serverless. If lerna symlinks related-module-a into service-b/node_modules/, then when serverless is packaging service-b all of ‘related-module-a/node_modules/’ will be packaged into service-b including all the dev dependencies for related-module-a (not to mention source artificats, the set of things normally excluded by .npmignore etc) … This will generally make the package too large to deploy when packages have a lot of devDependencies. This fact basically breaks the ability to use lerna to ease the development process of related-module-a in the context of testing/deploying/packaging service-b.
  • even if we could use lerna, the whole lerna concept is something of a large hack and for many projects introduces a large amount of complexity that may not actually be needed.

Goals

For my purposes my goals are:

• easily share code generated from a single compilation and testing context between serverless deployed stack’s
• manage devDependencies and tooling configuration for as much of the codebase as possible in a single place (shared code should be in the same compilation context for ensuring smoothest path to maximum benefit from compiler and ide tooling)
• support easily creating new variations of a stack for use in automated testing or to deal with other kinds of strange business/development requirements – ideally by just copying/modifying a serverless.yml file
• get the best experience from ide’s

The pattern I’ve fallen on is to structure our repository like this:

.serverless_plugins/    <-- custom plugins -- should be shareable between stacks
package.json               <-- scripts to start compiler, run tests etc.  
jest.json
jest.debug.json
tsconfig.json
tslint.json
wallaby.js
src/    <-- typescript code
lib/     <-- compiled javascript
services/some_service/serverless.yml
services/some_service/node_modules => ../../node_modules/
services/some_service/package.json => ../../package.json/        <-- needed to expose devDependencies to serverless packaging process
services/some_service/lib/ => ../../lib/
services/some_service/.serverless_plugins/ ../../.serverless_plugins/

The basic idea of above pattern is to use symlinks to trick serverless so that running serverless command from any services/*/ directory behaves as if you had copied the serverless.yml file into the project root directory and ran serverless from there.

Solution

I really think a behavior like this should just be supported by default without having to use symlinks to hack a serverless.yml file into the execution context of another folder. I’d like an option to allow providing a path to a yml file to use for serverless config. This should interpret the config file as if the config file was located in the cwd() – paths defined in config files should be specified relative to the cwd() – everything should be executed just as if you had moved the configuration file to ./serverless.yml prior to execution.

Alternative

If adding the requested feature and exposing to users the ability to control which config file to use from the command line isn’t deemed desirable – would a pull request that modified the Service class and Serverless constructor to allow easier implementation via a custom subclass of the Serverless class be accepted?

Right now I’m monkey patching the serverless framework to allow using alternative config files in my automated testing environments … Its not pretty and involved by necessity some copy/pasting but essentially required changing Service.load method to consider options passed to Service constructor … A little bit of changes to the framework could greatly ease this customization …

    // PATCHED: attempt to get serviceFilenames from `that` rather than only supporting hard-coded paths ...
    // List of supported service filename variants.
    // The order defines the precedence.
    const serviceFilenames =
      that.serviceFilenames != null
        ? that.serviceFilenames
        : ["serverless.yaml", "serverless.yml", "serverless.json"];

And then hacking the Serverless constructor to use my patched Service subclass and pass in custom path to serverless config file …

export class JestServerlessConfig extends Serverless {
  constructor(serverlessContext: JestServerlessContext) {
    // configure servicePath ...
    super({
      // set servicePath to the directory in which you want serverless command to execute (abs path)
      servicePath: serverlessContext.executionPath,
      interactive: false
    });

    // HACK: re-create the service, this time with patched behavior ...
    this.service = new JestServerlessServicePatch(this, {
      serviceFilenames: [serverlessContext.relativeServerlessConfigPath]
    });

    // HACK: Variables class picks off this.serverless.service instance inside constructor and saves it ... --
    // make sure to update that reference to point at patched object...
    this.variables.service = this.service;

    this._jestServerlessContext = serverlessContext;
  }

I also have to monkey patch serverless/lib/utils/getServerlessConfigFile.

Other Alternative

Its possible I could add webpack to my build process and generate a bundle including all required node_modules for each services/*/ directory. This could be a good alternative but webpack is complicated (and it would be much slower). It also wouldn’t really work all that well many varieties of common testing tools. The need to do this ultimately results from a small addressable limitation of the serverless framework which I think would be better to handle with what seems a small augmentation to the framework…

[wallzero]

#4267 said it best. I use package.json scripts to run my serverless tasks; sometimes those tasks are slightly different when running locally. It’s silly I can’t point to a different serverless.yml file as a command line argument and would be trivial to implement.

[carlosjac]

+1 to this idea