Lerna does not run scripts according to topological order

I have a monorepo configuration using lerna + yarn workspaces. All my packages are written in Typescript and therefore need to be compiled. There are interdependencies between packages in the monorepo.

After installing dependencies with yarn, I symlink all packages with lerna link. Then I try to build them with lerna run build.

I see the build fails because Lerna does not follow the dependency graph. It tries to build packages before their dependencies are built.

Expected Behavior

Lerna should execute package scripts following the dependency graph. I would expect the command lerna ls --toposort to output packages according to the dependency graph, dependencies before dependants.

Current Behavior

Lerna does not follow the dependency graph.

Possible Solution

Fix the algorithm.

Here is the dependency graph generated with lerna ls --graph:

{
  "@hmh-cam/plankton": [
    "@hmh-cam/base-composite-component",
    "@theia/core",
    "@types/cookie-parser",
    "atob",
    "btoa",
    "cookie-parser",
    "inversify"
  ],
  "@hmh-cam/base-composite-component": [
    "@hmh-cam/component-base",
    "@hmh-cam/nodejs-base-server",
    "@hmh-cam/schema-tool",
    "array-flatten",
    "exceljs",
    "express",
    "form-data",
    "inversify",
    "inversify-binding-decorators",
    "jsonwebtoken",
    "lit-element",
    "node-fetch",
    "reflect-metadata",
    "retry",
    "xlsx",
    "xmldom",
    "xpath"
  ],
  "@hmh-cam/content-block": [
    "@hmh-cam/base-composite-component",
    "@hmh-cam/component-base",
    "jsdom",
    "katex"
  ],
  "@hmh-cam/plankton-image": [
    "@hmh-cam/base-composite-component",
    "@hmh-cam/component-base",
    "@hmh-cam/nodejs-base-server",
    "@hmh-cam/overlay-modal",
    "@hmh-cam/plankton-structure",
    "node-fetch"
  ],
  "@hmh-cam/plankton-migration": [
    "@hmh-cam/base-composite-component",
    "@hmh-cam/component-base",
    "@hmh-cam/content-block",
    "@hmh-cam/nodejs-base-server",
    "@hmh-cam/plankton-structure",
    "@hmh-cam/plankton-xray",
    "commander",
    "inquirer",
    "loading-cli",
    "progress",
    "source-map-support",
    "xlsx"
  ],
  "@hmh-cam/plankton-structure": [
    "@hmh-cam/base-composite-component",
    "@hmh-cam/component-base",
    "@hmh-cam/content-block",
    "@hmh-cam/drop-down",
    "@hmh-cam/expand-collapse",
    "@hmh-cam/flip-reveal",
    "@hmh-cam/hint-list",
    "@hmh-cam/math-expression",
    "@hmh-cam/nodejs-base-server",
    "@hmh-cam/option-list",
    "@hmh-cam/overlay-modal",
    "@hmh-cam/plain-text",
    "@hmh-cam/plankton-video-embed",
    "@hmh-cam/response-validation",
    "@hmh-cam/rich-text",
    "@hmh-cam/standards",
    "@hmh-cam/text-input",
    "@hmh-cam/xray-trigger"
  ],
  "@hmh-cam/plankton-xray": [
    "@hmh-cam/base-composite-component",
    "@hmh-cam/component-base",
    "@hmh-cam/content-block",
    "@hmh-cam/drop-down",
    "@hmh-cam/nodejs-base-server",
    "@hmh-cam/plankton-structure",
    "commander",
    "progress",
    "source-map-support",
    "xlsx"
  ],
  "@hmh-cam/component-base": [
    "lit-element"
  ],
  "@hmh-cam/drag-drop": [
    "@hmh-cam/component-base",
    "@hmh-cam/plain-text",
    "@hmh-cam/response-validation"
  ],
  "@hmh-cam/drop-down": [
    "@hmh-cam/component-base",
    "@hmh-cam/inline-editor",
    "@hmh-cam/overlay-modal",
    "@hmh-cam/plain-text",
    "@hmh-cam/response-validation",
    "@hmh-cam/rich-text",
    "@material/select"
  ],
  "@hmh-cam/expand-collapse": [
    "@hmh-cam/component-base"
  ],
  "@hmh-cam/flip-reveal": [
    "@hmh-cam/component-base"
  ],
  "@hmh-cam/graphing-2d": [
    "@hmh-cam/component-base",
    "d3-array",
    "d3-axis",
    "d3-format",
    "d3-scale",
    "d3-selection"
  ],
  "@hmh-cam/hint-list": [
    "@hmh-cam/component-base",
    "@hmh-cam/plain-text",
    "@hmh-cam/rich-text"
  ],
  "@hmh-cam/inline-editor": [
    "@hmh-cam/component-base",
    "@hmh-cam/overlay-modal",
    "@hmh-cam/response-validation",
    "node-sass"
  ],
  "@hmh-cam/math-expression": [
    "@hmh-cam/component-base",
    "@hmh-cam/inline-editor",
    "@hmh-cam/overlay-modal",
    "@hmh-cam/plain-text",
    "@hmh-cam/response-validation",
    "@hmh-cam/rich-text",
    "katex"
  ],
  "@hmh-cam/option-list": [
    "@hmh-cam/component-base",
    "@hmh-cam/plain-text",
    "@hmh-cam/response-validation",
    "@hmh-cam/rich-text",
    "@hmh-cam/text-input",
    "@hmh-cam/theme-default"
  ],
  "@hmh-cam/overlay-modal": [
    "@hmh-cam/component-base"
  ],
  "@hmh-cam/plain-text": [
    "@hmh-cam/component-base",
    "katex"
  ],
  "@hmh-cam/response-validation": [
    "@hmh-cam/component-base",
    "@hmh-cam/plain-text",
    "@hmh-cam/rich-text"
  ],
  "@hmh-cam/rich-text": [
    "@hmh-cam/component-base",
    "prosemirror-commands",
    "prosemirror-dropcursor",
    "prosemirror-gapcursor",
    "prosemirror-history",
    "prosemirror-inputrules",
    "prosemirror-keymap",
    "prosemirror-menu",
    "prosemirror-model",
    "prosemirror-schema-list",
    "prosemirror-state",
    "prosemirror-transform",
    "prosemirror-view"
  ],
  "@hmh-cam/text-input": [
    "@hmh-cam/component-base",
    "@hmh-cam/inline-editor",
    "@hmh-cam/overlay-modal",
    "@hmh-cam/plain-text",
    "@hmh-cam/response-validation",
    "@hmh-cam/rich-text",
    "@material/button",
    "@material/textfield"
  ],
  "@hmh-cam/xray-trigger": [
    "@hmh-cam/component-base",
    "@hmh-cam/inline-editor",
    "@hmh-cam/overlay-modal",
    "@hmh-cam/plain-text",
    "@hmh-cam/response-validation",
    "@hmh-cam/rich-text"
  ],
  "@hmh-cam/schema-tool": [
    "ajv",
    "commander",
    "typescript-json-schema"
  ],
  "@hmh-cam/standards": [
    "@hmh-cam/nodejs-base-server"
  ],
  "@hmh-cam/waggle-structure": [
    "@hmh-cam/base-composite-component",
    "@hmh-cam/component-base",
    "@hmh-cam/content-block",
    "@hmh-cam/hint-list",
    "@hmh-cam/nodejs-base-server",
    "@hmh-cam/option-list",
    "@hmh-cam/plankton-image",
    "@hmh-cam/plankton-structure",
    "@hmh-cam/plankton-xray",
    "@hmh-cam/standards",
    "commander"
  ],
  "@hmh-cam/plankton-studio-core": [
    "@hmh-cam/base-composite-component",
    "@hmh-cam/content-block",
    "@hmh-cam/lumina",
    "@hmh-cam/nodejs-base-server",
    "@hmh-cam/plankton-image",
    "@hmh-cam/plankton-publisher",
    "@hmh-cam/plankton-structure",
    "@hmh-cam/plankton-xray",
    "@hmh-cam/standards",
    "@hmh-cam/waggle-structure",
    "inversify",
    "inversify-binding-decorators",
    "node-fetch",
    "reflect-metadata"
  ],
  "@hmh-cam/plankton-vsix": []
}

Now here is the result of lerna ls --toposort:

@hmh-cam/plankton
@hmh-cam/base-composite-component
@hmh-cam/content-block
@hmh-cam/plankton-image
@hmh-cam/plankton-migration
@hmh-cam/plankton-structure
@hmh-cam/plankton-xray
@hmh-cam/component-base
@hmh-cam/drag-drop
@hmh-cam/drop-down
@hmh-cam/expand-collapse
@hmh-cam/flip-reveal
@hmh-cam/graphing-2d
@hmh-cam/hint-list
@hmh-cam/inline-editor
@hmh-cam/math-expression
@hmh-cam/option-list
@hmh-cam/overlay-modal
@hmh-cam/plain-text
@hmh-cam/response-validation
@hmh-cam/rich-text
@hmh-cam/text-input
@hmh-cam/xray-trigger
@hmh-cam/schema-tool
@hmh-cam/standards
@hmh-cam/waggle-structure
@hmh-cam/plankton-studio-core
@hmh-cam/plankton-vsix

Looking at the above, I would expect @hmh-cam/base-composite-component to be built before @hmh-cam/plankton, because @hmh-cam/plankton depends on @hmh-cam/base-composite-component.

But the topological order is the opposite.

Your Environment

Executable	Version
lerna --version	3.22.1
yarn --version	1.22.5
node --version	10.19.0

OS	Version
macOS Mojave	10.14.6

[drochgenius]

Hi @atvorch ,

I’d suggest you move away from Lerna for managing your monorepo, as Lerna is largely unmaintained and there are much better monorepo tools available nowadays.

We have moved to Turborepo which does an awesome job at building your monorepo in topological order, and with local cache it makes for faster incremental builds. Turborepo is very easy to put in place with Lerna and whatever package manager you’re using: npm, yarn or pnpm.

So now we have removed the script I shared above in favor of Turborepo.

Hope this helps

[craigphicks]

Not sure why this issue was closed if it has not been addressed. The man page is clear that lerna run should sort in topo order unless ‘no-sort’ or ‘parallel’ flags are set.

I have been using lerna for a few months and this is the first time I have noticed topo sorting not working.
But it is definitely not working now. What has changed is adding more scope names and symbolic links. Could that be a problem?

Here is the new scenario -

I have two scopes: @examples, @sources

It seems that running:

lerna run --scope @sources build

will build in the correct order within @sources (although that could be a coincidence - update, it doesn’t always work).

However

lerna run --scope @examples  --scope @sources build 

tries (and fails) to build the @examples first, even though

lerna la --toposort

clearly shows the @examples at the bottom after @sources.

I vote to reopen.

Yet another workaround script. It’s only serial but allows --scope XXX --ignore XXX type args (however the XXX are regexp strings not glob strings).