Use this module with bundled node modules (using webpack)

Hi there

I’m coming over from an issue on DataDog/dd-trace-js#827 where I figured out, the issue actually lies in this library.

We are using webpack to bundle our application and all the required node modules in a single bundle. Now we installed dd-trace-js which uses require-in-the-middle to hook up its plugins. However, for us the require-in-the-middle does not work if the node modules are bundled. Is this a know inssue? Is there a way to use require-in-the-middle with bundled node modules?

I worked around the issue using webpack-node-externals but I would prefer going back to including the node modules in the bundle.

Thanks, Michael

[knvpk]

Hi @techmunk , i did also checked with your solution but im getting “modules not defiend”,

[techmunk]

For anyone interested in this, I’ve written a Webpack Plugin that can do this WITHOUT needing to patch or change code in require-in-the-middle.

What follows is typescript code, so make changes as needed for JS.

import { relative, sep } from 'path'
import { compilation, Compiler, Template } from 'webpack'

declare class CompilationModule extends compilation.Module {
  request?: string
  resource?: string
  rawRequest?: string
  external?: boolean
}

export class WebpackRequireInTheMiddlePlugin {
  public readonly name = 'WebpackRequireInTheMiddlePlugin'
  protected readonly modulesMap: Map<number | string | null, [string, boolean, string?]>
  protected readonly modules: string[]
  protected readonly internalModuleConditions: string[]
  protected addShims = true
  protected fsModuleId?: string | number | null
  protected resolveModuleId?: string | number | null
  protected moduleIds: Map<string, number | string | null | undefined>

  public constructor(modules?: string[], internalModules?: string[]) {
    this.modulesMap = new Map()
    this.moduleIds = new Map()
    this.modules = modules ?? []
    this.internalModuleConditions = internalModules ?? []
  }

  public apply(compiler: Compiler): void {
    compiler.hooks.compilation.tap(this.name, compilation => this.compilation(compilation))
  }

  protected compilation(compilation: compilation.Compilation): void {
    compilation.hooks.afterOptimizeModuleIds.tap(this.name, modules => this.mapModuleIds(modules))
    compilation.mainTemplate.hooks.localVars.tap(this.name, (source) => this.addLocalVarSources(source))
    compilation.mainTemplate.hooks.require.tap(this.name, (source) => this.addRequireSources(source))
  }

  protected getModuleName(filename?: string): string {
    if (filename) {
      const segments = filename.split(sep)
      const index = segments.lastIndexOf('node_modules')
      if (index !== -1 && segments[index + 1]) {
        return segments[index + 1][0] === '@' ? `${segments[index + 1]}/${segments[index + 2]}` : segments[index + 1]
      }
    }

    return ''
  }

  protected canSkipShimming(module: CompilationModule): boolean {
    if (module.external && module.request) {
      return this.internalModuleConditions.includes(module.request)
    }
    return false
  }

  protected includeModule(module: CompilationModule): boolean {
    const moduleName = this.getModuleName(module.resource)
    return this.modules.length === 0 || (moduleName !== '' && this.modules.includes(moduleName))
  }

  protected mapModuleIds(modules: CompilationModule[]): void {
    for (const module of modules) {
      if (this.canSkipShimming(module)) {
        break
      }
      if (!module.external && module.resource) {
        if (this.includeModule(module)) {
          this.modulesMap.set(module.id, [relative(`${process.cwd()}/node_modules`, module.resource), false])
          if (this.getModuleName(module.resource) === module.rawRequest) {
            this.moduleIds.set(module.rawRequest, module.id)
            // eslint-disable-next-line @typescript-eslint/no-var-requires
            const { version } = require(`${module.rawRequest}/package.json`)
            this.modulesMap.set(module.id, [relative(`${process.cwd()}/node_modules`, module.resource), false, version])
          }
        }
        if (module.resource.includes('resolve/index.js')) {
          this.resolveModuleId = module.id
        }
      }
      else if (module.request) {
        if (this.modules.includes(module.request)) {
          this.modulesMap.set(module.id, [module.request, true])
        }
        if (module.request === 'fs') {
          this.fsModuleId = module.id
        }
      }
    }
  }

  protected getRequireShim(): string[] {
    return [
      'const __ritm_require__ = __ritm_Module__.prototype.require',
      'const __ritm_require_shim__ = function (id) {',
      Template.indent([
        'return modules[id] ? __webpack_require__(id) : __ritm_require__.apply(this, arguments)'
      ]),
      '}',
      '__ritm_Module__.prototype.require = __ritm_require_shim__'
    ]
  }

  protected getResolveFilenameShim(): string[] {
    return [
      'const __ritm_resolve_filename__ = __ritm_Module__._resolveFilename',
      '__ritm_Module__._resolveFilename = function (id) {',
      Template.indent([
        'if (modules[id] && __ritm_modules_map__.has(id)) {',
        Template.indent([
          'const [filename, core] = __ritm_modules_map__.get(id)',
          // eslint-disable-next-line no-template-curly-in-string
          'return core ? filename : `${process.cwd()}${sep}node_modules${sep}${filename}`'
        ]),
        '}',
        'return __ritm_resolve_filename__.apply(this, arguments)'
      ]),
      '}'
    ]
  }

  protected addLocalVarSources(source: string): string {
    return !this.addShims ? source : Template.asString([
      source,
      'const { sep } = require("path")',
      `const __ritm_modules_map__ = new Map(${JSON.stringify(Array.from(this.modulesMap.entries()), null, 2)})`,
      `const __ritm_module_ids_map__ = new Map(${JSON.stringify(Array.from(this.moduleIds.entries()), null, 2)})`,
      'const __ritm_Module__ = module.require("module")',
      ...this.getRequireShim(),
      ...this.getResolveFilenameShim(),
      'const __ritm_shimmed__ = {}'
    ])
  }

  protected getFsShim(): string[] {
    if (this.fsModuleId) {
      return [
        `const __ritm_fs_readFileSync__ = __webpack_require__(${this.fsModuleId}).readFileSync`,
        `installedModules[${this.fsModuleId}].exports.readFileSync = function(path) {`,
        Template.indent([
          'const [module, file] = path.split(sep).slice(-2)',
          'if (file === "package.json" && __ritm_module_ids_map__.has(module)) {',
          Template.indent([
            'const version = __ritm_modules_map__.get(__ritm_module_ids_map__.get(module)).slice(-1)',
            // eslint-disable-next-line no-template-curly-in-string
            'return `{"version": "${version}"}`'
          ]),
          '}',
          'return __ritm_fs_readFileSync__.apply(this, arguments)'
        ]),
        '}'
      ]
    }
    return []
  }

  protected getResolveModuleShim(): string[] {
    if (this.resolveModuleId) {
      return [
        `const __ritm_resolve_sync__ = __webpack_require__(${this.resolveModuleId})`,
        `installedModules[${this.resolveModuleId}].exports.sync = function(name) {`,
        Template.indent([
          'if (__ritm_module_ids_map__.has(name)) {',
          Template.indent([
            'const [filename, core] = __ritm_modules_map__.get(__ritm_module_ids_map__.get(name))',
            // eslint-disable-next-line no-template-curly-in-string
            'return core ? filename : `${process.cwd()}${sep}node_modules${sep}${filename}`'
          ]),
          '}',
          'return __ritm_resolve_sync__.apply(this, arguments)'
        ]),
        '}'
      ]
    }
    return []
  }

  protected getRequireResolveShim(): string[] {
    return [
      'const __ritm_require_resolve__ = require.resolve',
      'require.resolve = function(name) {',
      Template.indent([
        'if (__ritm_module_ids_map__.has(name)) {',
        Template.indent([
          'const [filename, core] = __ritm_modules_map__.get(__ritm_module_ids_map__.get(name))',
          // eslint-disable-next-line no-template-curly-in-string
          'return core ? filename : `${process.cwd()}${sep}node_modules${sep}${filename}`'
        ]),
        '}',
        'return __ritm_require_resolve__.apply(this, arguments)'
      ]),
      '}'
    ]
  }

  protected getShims(): string[] {
    return [
      ...this.getFsShim(),
      ...this.getResolveModuleShim(),
      ...this.getRequireResolveShim()
    ]
  }

  protected getResetShims(): string[] {
    let reset: string[] = []
    if (this.fsModuleId) {
      reset = [
        ...reset,
        `installedModules[${this.fsModuleId}].exports.readFileSync = __ritm_fs_readFileSync__`
      ]
    }
    if (this.resolveModuleId) {
      reset = [
        ...reset,
        `installedModules[${this.resolveModuleId}].exports.readFileSync = __ritm_resolve_sync__`
      ]
    }

    return reset
  }

  protected addRequireSources(source: string): string {
    return !this.addShims ? source : Template.asString([
      'if (__ritm_Module__.prototype.require !== __ritm_require_shim__ && !__ritm_shimmed__[moduleId]) {',
      Template.indent([
        '__ritm_shimmed__[moduleId] = true',
        'if (__ritm_modules_map__.has(moduleId)) {',
        Template.indent([
          ...this.getShims(),
          'const exports = __ritm_Module__.prototype.require(moduleId)',
          'installedModules[moduleId].exports = exports',
          ...this.getResetShims()
        ]),
        '}'
      ]),
      '}',
      source
    ])
  }
}

To use:

    const modules = [
      'apollo-server-core',
      'bluebird',
      'cassandra-driver',
      'elasticsearch',
      'express',
      'express-graphql',
      'express-queue',
      'fastify',
      'finalhandler',
      'generic-pool',
      'graphql',
      'handlebars',
      'hapi',
      '@hapi/hapi',
      'http',
      'https',
      'http2',
      'ioredis',
      'jade',
      'knex',
      'koa',
      'koa-router',
      '@koa/router',
      'memcached',
      'mimic-response',
      'mongodb-core',
      'mongodb',
      'mysql',
      'mysql2',
      'pg',
      'pug',
      'redis',
      'restify',
      'tedious',
      'ws'
    ]

this.config.plugins?.push(new WebpackRequireInTheMiddlePlugin(modules))

The list of modules above is taken straight from elastic-apm-node. If elastic-apm-node exported the modules, would probably not even need that!

This is tested against webpack 4 with an effort to get elastic-apm-node to work with a full bundled webpack build.