BUG (monorepo): Package dependencies aren't included when root level dep also exists.

Hi! Awesome package you have here 😃

Created a minimal example repo reproducing the issue we are experiencing here

Currently, when 2 functions in our monorepo have conflicting dependencies declared, serverless-jetpack is failing to include the node_modules that includes the dependency if it is not the root node_modules. So for example:

functions/a/              # depends on diff ^4.0.1
  package.json
  src/
    index.js
  node_modules/
    diff/
functions/b/              # depends on diff @3.5.0
  package.json
  src/
    index.js
node_modules/
  diff/

Results in the following artifacts:

a/ # no node_modules present
  functions/a/src/index.js
  functions/a/package.json
  package.json
b/ # root node_modules present
  node_modules
  functions/b/src/index.js
  functions/b/package.json
  package.json

I can spend some time digging into this issue this weekend, but wanted to open up the issue with an example/description so if anyone else wants to look into it they have some reference info.

Details serverless-jetpack version: ^0.5.0 serverless version: ^1.48.0

/cc @ryan-roemer

edit:

was able to narrow down the source of the issue:

# global include
package:
  individually: true
  include:
    - package.json
    - "!**/yarn.lock"
    - "!functions/**" # this negative include comes after the function-level dependencies in the globInclude array

[ryan-roemer]

SUPER WEIRD STUFF

On a hunch, I decided to cut out all of jetpack and just focus on our globby() and nanomatch inputs and outputs. I created: tmp-glob-test.js

Here’s the weird stuff: If the working directory matches the “set working directory” internally to jetpack (which is then passed to globby() via `cwd) then different globbing behavior happens.

Here’s a basic example:

# Start in project root.
# NOTE: `functions/base/node_modules/diff/lib/index.js` is INCLUDED in fileMatch
# which is the `globby()` results.
$ node tmp-glob-test.js 
{
  "fileMatch": {
    "included": [
      "functions/base/node_modules/diff/lib/index.js",
      "functions/base/src/base.js"
    ],
    "excluded": [
      "functions/base/node_modules/another/README.md",
      "functions/base/node_modules/another/index.js",
      "functions/base/node_modules/diff/README.md",
      "functions/base/package.json",
      "functions/base/src/exclude-me.js"
    ]
  },
  "patternMatchLimited": {
    "included": [
      "functions/base/node_modules/diff/lib/index.js",
      "functions/base/src/base.js"
    ],
    "excluded": []
  },
  "patternMatch": {
    "included": [
      "functions/base/node_modules/another/README.md",
      "functions/base/node_modules/another/index.js",
      "functions/base/node_modules/diff/README.md",
      "functions/base/node_modules/diff/lib/index.js",
      "functions/base/package.json",
      "functions/base/src/base.js"
    ],
    "excluded": [
      "functions/base/src/exclude-me.js"
    ]
  }
}

# Move to our test fixture directory that **is** `cwd` in the script
$ cd test/packages/monorepo/yarn/

# NOTE: `functions/base/node_modules/diff/lib/index.js` is EXCLUDED in fileMatch
# which is the `globby()` results.
$ node ../../../../tmp-glob-test.js
{
  "fileMatch": {
    "included": [
      "functions/base/src/base.js"
    ],
    "excluded": [
      "functions/base/node_modules/another/README.md",
      "functions/base/node_modules/another/index.js",
      "functions/base/node_modules/diff/README.md",
      "functions/base/node_modules/diff/lib/index.js",
      "functions/base/package.json",
      "functions/base/src/exclude-me.js"
    ]
  },
  "patternMatchLimited": {
    "included": [
      "functions/base/src/base.js"
    ],
    "excluded": []
  },
  "patternMatch": {
    "included": [
      "functions/base/node_modules/another/README.md",
      "functions/base/node_modules/another/index.js",
      "functions/base/node_modules/diff/README.md",
      "functions/base/node_modules/diff/lib/index.js",
      "functions/base/package.json",
      "functions/base/src/base.js"
    ],
    "excluded": [
      "functions/base/src/exclude-me.js"
    ]
  }
}

Takeaways

• If we’re not process.cwd() === cwd then a simple !functions (and not !functions/**) gets us pretty much what we want for our monorepo scenario – exclude all of functions except for things added back in and the dependencies added by Jetpack.
• I’m going to look for an upstream issue in globby() to see if there’s some insight into this behavior. It seems super weird that globby(patterns, { cwd }) has that parameter but the actual current working directory affects results…

Notes

• Upgraded to https://github.com/sindresorhus/globby/releases/tag/v10.0.1 and didn’t make a difference.

[pdeona]

that sounds like a good approach! the greatest difficulty i encountered trying to fix this was maintaining the existing functionality for non-monorepos so adding a new config option may very well be required. as you said there is no equivalent sls behavior in the monorepo scenario so needing a little extra in our config to get it to work seems fine.