Improve packaging performance

_Excerpt from: https://github.com/serverless/serverless/issues/6580_

While bigger packaging refactor is scheduled, it appears we can improve significantly our current packing logic in place with following steps:

Implementation proposal

1. (if package.excludeDevDepencies: true) Resolve all devDependencies, and coin their paths in node_modules (take into account https://github.com/serverless/serverless/issues/4358 error case)
2. Resolve all files in a service folder (if package.excludeDevDepencies: true avoid traversing into dev dependencies) - To ensure better performance construct logic with callbacks not promises.
3. Show warning if symlinks are approached (we may consider adding support for them in next PR)
4. Filter obtained files list against eventual package patterns, but intelligently, do not test to be included filenames against include patterns, and do not test excluded filenames against exclude patterns.

[BenLanglois]

I recently experienced a performance issue related to this when packaging my project. In serverless.yml, we specify:

package:
  exclude:
    '**'

This disables the default behaviour of including all files in the current directory. However, globby is only passed params.include, not params.exclude. So, globby will return all files in the project directory only for micromatch to later filter out most of them. This means that using exclude: '**' instead of include: '!**' has a big performance impact.

To fix this, globby needs to consider excluded patterns too. The fix for this should be simple: instead of

globby(['**'].concat(params.include), /* ... */)

use

globby(['**'].concat(patterns), /* ... */)

since patterns contains params.include and the negation of params.exclude.

[ryan-roemer]

Yeah @BenLanglois – I’ve got a long comment and suggestion for future packaging tweaks here that also discusses the performance problems you’ve also identified: https://github.com/serverless/serverless/issues/6007#issuecomment-520092329