Respect/Merge Input Source Maps With Generated Source Map Data

Do you want to request a feature or report a bug? Feature

What is the current behavior? The transform pipeline does not respect/load/merge existing input source maps with generated source map data.

What is the expected behavior? If an input module has an existing source map file, especially if the input module includes a source map url for the existing source map file, metro should load that input source map and merge that mapping with the output mapping generated by metro.

Related: https://github.com/facebook/metro/issues/10 https://github.com/facebook/react-native/issues/12705

Please provide your exact Metro configuration and mention your Metro, node, yarn/npm version and operating system.

• Operating System: Windows 10
• Node Version: 9.2.0
• Yarn Version: 1.2.1
• Metro Version(s): Latest

Details The transform pipeline does not load existing input source maps, and thus does not merge them with the generated source map data.

• If the input module code contains a //# sourceMappingURL= reference to the existing input source map, then the existing input source map file contents should be loaded early in the process, possibly in the Module class where it can be cached with the other module data and should flow through the transform pipeline.

• If an input source map object is present it should be merged with/transform the output source map. The transformer should pass the input source map to the babel.transform(..) call via the babelConfig.inputSourceMap property. The post transform logic should merge the input source map with the generated one.

For Reference. https://github.com/babel/babel/blob/7d37017a5f85260c0c95580731585916e791265c/packages/babel-core/src/transformation/file/index.js#L290

I played around with modified version of transform(..) and postTransform(..) methods. It worked for my very basic proof of concept. I was hoping one of the maintainers familiar with the source map optimizations and recent overhaul of the transform pipeline (e.g. @davidaurelio ) could weight in on what else would be needed for an acceptable PR.

Something like this for example (Module.js):

  _readInputSourceMap(): ?string {
    if(!this._hasCheckedForInputSourceMap) {
      this._hasCheckedForInputSourceMap = true;
      this._inputSourceMapUrl = this._extractInputSourceMapUrl(this._readSourceCode());
      if(this._inputSourceMapUrl != null) {
        // perhaps any manipulations / raw mapping optimizations here ?
        this._inputSourceMap = this._optimizeInputSourceMap(fs.readFileSync(this._inputSourceMapUrl, 'utf8'));
      }
      // if there is not a source mapping url should we look for a matching filename with the `.map` suffix ?
      // if we're passing an input source map through, should we parse any inline source map from the code, at this point as well ?
    }
    return this._inputSourceMap;
  }

  _extractInputSourceMapUrl(sourceCode: string): ?string {
    // some optimized version of this logic perhaps:
    const urlIndex = sourceCode.indexOf('//# sourceMappingURL=');
    if(urlIndex ===  -1) {
        return null;
    }
    var sourceMapUrl = sourceCode.substring(urlIndex + 21);
    if(sourceMapUrl.indexOf('\n') > -1) {
        sourceMapUrl = sourceMapUrl.substring(0, sourceMapUrl.indexOf('\n'));
    }
    return sourceMapUrl;
  }

Pass the input source map data through…

  async transformFile(
    filename: string,
    localPath: LocalPath,
    code: string,
    inputSourceMap: SourceMap
    ....

For my proof of concept with the raw mappings I was doing this: src/transformer.js

   function buildBabelConfig(filename, options, inputSourceMap?: SourceMap, plugins?: BabelPlugins = []){
    ...
    if(!!inputSourceMap) {
      config.inputSourceMap = inputSourceMap;
    }
    ...
  }

   function transform(filename, options, src, inputSourceMap, plugins): Params) {
        ...
        const babelConfig = buildBabelConfig(filename, options, inputSourceMap, plugins);
        const {ast,map} = babel.transform(src, babelConfig);

        return {ast,map};
        ...
   }

src/JSTransformer/worker/index.js

    function transformCode(...) {

        ....
        return transformResult instanceof Promise
           ? transformResult.then(({ast,map})=> postTransform(...postTransformARgs, ast, map))
            : postTransform(...postTransformArgs, transformResult.ast, transformResult.map);
    }

    function postTransform(...) {
        ....
        var map;
       if(result.rawMappings && inputSourceMap) {
            map = toRawMappings(mergeSourceMaps(inputSourceMap, result.map)).map(compactMapping);
        } else if(result.rawMappings) {
            map = result.rawMappings.map(compactMapping);    
        } else if(inputSourceMap) {
             map = toRawMappings(inputSourceMap).map(compactMapping);
         } else {
             map = [];
         }
         
        return {
            result: {dependencies, code: result.code, map},
            transformFileStartLogEntry,
            transformFileEndLogEntry,
        };
    }

[aleclarson]

Bump. This deserves more attention from the Metro team.

What does Facebook do internally for this?

[thheller]

I’m working on shadow-cljs which is a build tool for ClojureScript and I’m adding proper support for react-native. This is all working fine already with the exception of source maps. If we want proper source maps we currently have to bypass metro completely and instead load the code dynamically at runtime. Even then it only works when running in Chrome. All we need really is for metro to read the input source maps we generated when translating CLJS->JS, as others have suggested in the past.

Any plans to make this a possibility?