Recommendation for multi-module setup? Memory leak with 18+ Rollup watchers

I have a project with a bunch (18) separate (TS) modules, each of which is compiled down to its own .js file with rollup and this plugin. I’d like to set up a single rollup process to watch and recompile the lot of them, which I’m currently doing with a rollup config file that instantiates this plugin 18 times, each time with a different include setting, because otherwise the plugin will emit .d.ts files for all the files in each project’s dist directory.

This regularly (but only after a bunch of recompile cycles) leads to node running out of memory:

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x134e879]
Security context: 0x2ed275600919 <JSObject>
    1: getDeclarationName(aka getDeclarationName) [0x3270c86f3011] [/home/marijn/src/cm6/node_modules/typescript/lib/typescript.js:~27802] [pc=0x34f390cf2ea2](this=0x2b31f95404d1 <undefined>,0x2df9bfc13a39 <NodeObject map = 0x2513288541d9>)
    2: declareSymbol(aka declareSymbol) [0x3270c86f3091] [/home/marijn/src/cm6/node_modules/typescript/lib/typescript.js:~27...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

I guess that has something to do with the TypeScript compiler building up its type/code graphs multiple times, once for every instance of the plugin. Is that correct?

Is there a way to set up the plugin once, but use it to build several projects, each of which emits .d.ts files only for its own files?

Or is there maybe some other recommended way to approach a use case like this?

[zouhangwithsweet]

I find way to bypass the problem. Use child_process to build a small amount components, then kill this process. With a recursion call, everything is ok. 😃

main.js

const buildChild = (a, b) => {
  const cp = require('child_process')
  const c1 = cp.spawn('node', ['./build/build.component.js', a, b])
  c1.stdout.on('data', function (data) {
    spinner.info(`${chalk.blue(data)}`)
  })

  c1.stderr.on('data', function (data) {
    spinner.warn(`${chalk.red(data)}`)
  })

  c1.on('close', function (code) {
    a += 5
    b += 5
    if (a > pkgs.length && b > pkgs.length) {
      spinner.succeed(`${chalk.green('Build done. Exit code ' + code)}`)
      return
    }
    buildChild(a, b)
  })
}

build.component.js

const runBuild = async () => {
  let index = 0
  const pkgs = await getPackages()
  const inputs = pkgs
    .map(pkg => pkg.name)
    .slice(process.argv[2], process.argv[3])

  build(inputs[index])

  async function build(name) {
    const inputOptions = {
      input: pathToEntry,
      plugins: [
        nodeResolve(),
        css(),
        vue({
          target: 'browser',
          css: false,
        }),
        typescript({
          tsconfigOverride: {
            compilerOptions: {
              declaration: false,
            },
            'exclude': [
              'node_modules',
              '__tests__',
            ],
          },
          abortOnError: false,
        }),
      ],
      external(id) {
        return /^vue/.test(id)
          || deps.some(k => new RegExp('^' + k).test(id))
      },
    }
    const outOptions = {
      format: 'es',
      file: [filename],
    }

    const bundle = await rollup.rollup(inputOptions)
    console.log(name, 'done')
    await bundle.write(outOptions)
    index++
    if (index < inputs.length) {
      await build(inputs[index])
    } else {
      console.log('batch done')
    }
  }
}

runBuild()

[ezolenko]

Indeed.

Looks like there is a memory leak somewhere. Rollup had a leak in watch mode before, but it seems to be fixed. (https://github.com/rollup/rollup/issues/883)

Each instance of the plugin will create it’s own instance of typescript.LanguageService and will not reset it during watch mode. This is how IDEs are operating I think, so typescript itself shouldn’t have obvious leaks, or somebody would notice.

You can try running plugin with clean: true, this disables cache (but I don’t see what could be leaking there either).