`webpackAST` returned in `extraInfo` from custom loader appears ignored internally

Bug report

I think this is some kind of bug, but it’s a little confusing since this optimisation seems under-advertised. The docs do mention:

It can be useful to pass an abstract syntax tree (AST), like ESTree, as the fourth argument (meta) to speed up the build time if you want to share common ASTs between loaders.

but this omits the specific variable name of webpackAST (it implies the fourth argument should just be the AST itself), so I’m basing this report on the source code and checked-in testcases instead.

If I’m doing something wrong here, please let me know!

What is the current behavior?

When defining a custom loader which returns an extraInfo object containing an acorn webpackAST property, webpack appears to ignore the supplied ast.

If the current behavior is a bug, please provide the steps to reproduce.

• Create a custom loader which looks like the ast-loader test case: https://github.com/webpack/webpack/blob/9fcaa243573005d6fdece9a3f8d89a0e8b399613/test/cases/parsing/precreated-ast/ast-loader.js#L27
• Add a console.log to NormalModule.js here to log the value of this._ast and extraInfo: https://github.com/webpack/webpack/blob/293e677b355da0d5bc4ddfc97d2afec114912711/lib/NormalModule.js#L794-L799
• Notice that the logs show extraInfo as null always:

{ extraInfo: null, ast: null }

According to a log earlier on, at this spot, result always appears to be a single-item array containing only the code, despite the loader using this.callback(null, code, map, { webpackAST });.

What is the expected behavior?

Webpack and NormalModule should use the supplied extraInfo + webpackAST to avoid re-parsing files.

Other relevant information: webpack version: 5.73.0 Node.js version: 16.13.2 Operating System: Ubuntu 18.04.5 LTS Additional tools: n/a

[alexander-akait]

@jdb8

If it is worth doing, would you be open to a PR updating some of the documentation around this?

PR welcome

Anyway, there is some trick(pseudo code, just idea):

const otherLoader = require('other-loader');
const { urlToRequest } = require('loader-utils');

module.exports = function myTransformLoader(..args) {
    const filename = urlToRequest(this.resourcePath);

    const result = otherLoader.call(this, options);

    const { code, map, webpackAST } = myCustomTransformFn({ source, filename });
  

    this.callback(null, code, JSON.parse(map), {
        webpackAST,
    });
};

So you can run any loader inside your loader and do another fancy things

[jdb8]

@alexander-akait I think that was the issue: my loader above had another custom loader in front of it that was not passing in or returning the meta argument:

function anotherLoader(source) {
    ...
    return modifiedSource;
}

Removing that loader, or updating it to pass through the map and meta arguments resolved the issue (I’m now seeing the console.log in NormalModule log what I expect:

function anotherLoader(source, map, meta) {
    ...
    this.callback(null, modifiedSource, modifiedMap, modifiedMeta);
}

I’m curious though, are there any examples in the wild of webpackAST being used for this kind of performance optimisation? I’m working on a caching layer which can pre-crunch transpiled sources, and the plan was to also pre-generate acorn ASTs compatible with webpack to skip its internal parsing work.

I haven’t benchmarked things properly yet, but so far I’m not seeing that much of a speedup. I’m wondering if the lack of advertising for this optimisation means that it might not be worth aiming for?

If it is worth doing, would you be open to a PR updating some of the documentation around this? I think adding a warning that all loaders in the chain must support meta might help someone else avoid making the mistake I did. Would also look to more explicitly document the shape of the meta arg beyond what’s currently listed in the current docs.

I’ll close this issue since there doesn’t seem to be a bug here, though - thanks for taking a look!