Allow syntactic rule application inside a lexical rule

As discussed on this thread in the Google Group:

Take JS’s template strings, for example:

`The answer is ${6 * 7}`

To parse this, it would be nice if you could write something like:

tstr
  = "`" tstrPart* "`"

tstrPart
  = "${" Expr "}"      -- expr
  | ~("`" | "${") any  -- char

Right now you can’t do this, because that invocation of Expr is not allowed. But it could be.

Yesterday Pat and I talked about this, and we think that it might be a good idea to allow syntactic rules to be invoked from inside lexical rules – this would probably require an operator, something like the analog of the “lexification” operator, #. For now, I’ll use @.

We could rewrite the tstrPart rule above using @ as follows:

tstrPart
  = "${" @Expr "}"      -- expr
  | ~("`" | "${") any   -- char

The semantics of @ would be the same as the semantics of the application of a start rule that is syntactic. In other words, @Expr would skip both leading and trailing whitespace. (That’s exactly what you’d want in this example.)

[pdubroy]

This has shipped in an experimental form in v16.1.0-pre.1, which can be installed via npm install ohm-js@next. It’s also available in the Ohm Editor by appending ?ohm-js=next to the URL.

Here is a sample grammar, using the tstr example that has been used in this thread:

G {
  tstr = "`" tstrPart* "`"

  tstrPart
    = "${" experimentalApplySyntactic<Expr> "}"  -- expr
    | ~("`" | "${") any  -- char
  
  Expr = number "+" number
  number = digit+
}

Note that both leading and trailing whitespace will be skipped around Expr. Here is a screenshot from the editor with Show spaces enabled:

Feedback on this is very much welcome!

[pdubroy]

@4silvertooth That’s an interesting case, but I don’t think we need to support it. You can always use ~space inside a lexical context to disallow spaces (and prevent space skipping) at that position. If we had the bang operator that I described above:

tstrPart
  = "${" ~space Expr! "}"  -- expr
  | ~("`" | "${") any      -- char

…should do the trick.