toRequireContext does not work in all cases

Storybook v6.0.0-rc.3 accepts globs to describe the location of ‘stories’ files. It appears to attempt to convert these globs into calls to require.context. - https://github.com/storybookjs/storybook/blob/next/lib/core/src/server/preview/to-require-context.ts#L28

It currently sets the recursive flag based on whether the glob contains a ‘globstar’ / ** pattern:

      const recursive = glob.startsWith('**');

However, many other globs will also need this flag to be set. For example, ../components/*/stories/*.js; without the recursive flag webpack will never look in the stories/ directories and so no stories will be found.

I’ve not had time to think it through fully, but it seems like any glob (after being split from its base will need the recursive flag if it contains more than one /.

If my argument hold so far, then I think it’s important to point out that one of the consequences of this is that globs cannot always be fully represented as calls to require.context as require.context will need to traverse the entire base and the regexp will be used to eliminate unwanted matches afterwards. In some cases (such as when node_modules/ directories are being traversed) this can lead to webpack running out of memory.

Additionally, it seems to me that if the following code is still required;

      if (source.startsWith('^')) {
        // prepended '^' char causes webpack require.context to fail
        const match = source.substring(1);

        return { path: base, recursive, match };
      }

it will be impossible to represent wildcard (*) globs, such as the example above, that should not match across directories.

[daniel-ac-martin]

Now that PR’s been merged I wonder whether this issue should be closed or whether it is worth starting a campaign to improve require.context.

FWIW, it looks like the starting point to improving the implementation would be here: https://github.com/webpack/webpack/blob/master/lib/dependencies/RequireContextDependencyParserPlugin.js#L18

Mind you it looks like a related subject was brought up quite recently (and storybook was even mentioned): https://github.com/webpack/webpack/issues/11149 I’m not sure I understand the resolution to that one.

There’s also this that I still believe to be relevant: https://github.com/webpack/webpack/issues/2952

If nothing else, I think it’s worth noting that Storybook users are likely to encounter failures due to huge memory consumption when globbing for story files in a monorepo in which each component has it’s own package and therefore node_modules/ dir. In these cases require.context will descend into those node_modules/ directories (and perhaps into an infinite symlink loop) even though the glob would imply that it shouldn’t do so.

And those users are likely to blame Storybook. 😕

[ndelangen]

fixed in https://github.com/storybookjs/storybook/pull/11647