Use babel-parse for import/export/jsx blocks

This issue is intended to discuss bringing in @babel/parse to handle ES/JSX parsing.

I’ve discussed this a few times with @timneutkens and @ChristopherBiscardi in passing but it’s something we haven’t been prioritizing. Over the last week I’ve been thinking about it more heavily after @jescalan reported a bunch of new parse issues.

We’ve reached a point where most open issues with the bug label are related to parsing exports and JSX. So we ultimately need to be able to parse import/export/jsx nodes to handle the following scenarios:

• Parse out the default export since that maps to the layout
• Walk a JSX node to its completion (right now newlines aren’t properly handled)
• Handle all use cases where import/export/JSX blocks aren’t separated by empty lines
• Allow (eventually) for the interleaving of MDX in JSX blocks

There might be a small performance hit, but since it’s build time it’s not something I’m personally worried about. The big bummer will be the heavier bundle, but that really only comes into play for browser runtimes (which isn’t really our targeted env anyway).

• Anyone have any thoughts, insights, or FUD about this?
• Is there anyone that wants to take this on?

[jonsherrard]

Hey @johno

I wrote an mdx-to-mdx-ast package here: https://github.com/devular/mdx-to-mdx-ast

It uses the Acorn JS parser that powers babel to verify nodes are import, export, or JSX elements.

I’m not sure if it’s of any use to you, I mostly wrote it for fun and to learn a bit about the MDX spec and ASTs.

Thanks for all your work on MDX, there’s so many opportunities around content authoring, an amazing library. 🙏

[jonsherrard]

Update so far, I’m still in the playground phase over at: https://github.com/devular/mdx-to-mdx-ast

Handle all use cases where import/export/JSX blocks aren’t separated by empty lines

Remark

remark-parse will return

import Foo from './foo'
export default foo

as a single MDAST node.

Acorn

1. Acorn will parse the same example into two javascript expressions represented by the Acorn AST.
2. We now have to robust AST nodes (albeit Acorn AST nodes), that represent the two declarations.
3. Using Unist Flatmap, we can return multiple nodes to our new MDXAST tree from a single node of our MDAST tree.
4. I did think about constructing the new `import Foo from ‘./foo’, by hand using the object, but thought better of it and installed escodegen, to robustly create ES compatible code.

So now we have two solid solutions:

-> Acorn parsing of strings to a JS AST <- ESCodegen to create strings of ES2015 JavaScript for our MDXAST

You can see this handled here: https://github.com/devular/mdx-to-mdx-ast/blob/master/handle-multi-program-nodes.js

Reservations

Using https://github.com/zeit/ncc, the bundle is a whopping 287kb, now that we’ve added acorn, acorn-jsx, and escodegen. There may be optimisations here through tree-shaking, or potentially this a point where I look at babel again. It might have codegen and JS parsing built in, and most projects will have it installed already.

I’d like to use this in the browser at some point… that’s pretty huge - but I was thinking for a CMS - slightly more palatable.

There’s now a lot going in here JS parsing, and JS codegen, Markdown parsing, and remark,rehype plugin execution… it feels like a lot, but that’s maybe how robust it needs to be.