CST Creation customizations

I noticed something strange about this:

  body = this.RULE('body', () => {
    this.MANY(() => {
      this.SUBRULE(this.key);
      this.SUBRULE(this.value);
    });
  });

vs

  body = this.RULE('body', () => {
    this.MANY(() => {
      this.SUBRULE(this.keyValue);
    });
  });

  keyValue = this.RULE('keyValue', () => {
    this.SUBRULE(this.key);
    this.SUBRULE(this.value);
  });

In the first case, the output CST results in a case where the keys are grouped into their own array, and the values are then grouped in their own array. While in the second case the key and value are kept within a single key value combination, and then grouped into an array.

I had it the first way first, and tried to create an interpreter for the resulting output cst, but realised that it wasn’t going to work because of how the key and value was separated. So I tried to do it the second way instead. But I think it’s somewhat non-intuitive that:

this.MANY(() => {
  this.SUBRULE(this.k);
  this.SUBRULE(this.v);
});

Doesn’t mean how (kv)* would be interpreted to mean a sequence of kvs, and should give [kv, kv, kv]. But instead right now it’s [k,k,k,k], [v,v,v,v].

Another example is:

  value = this.RULE('value', () => {
    this.MANY(() => {
      this.OR([
        {ALT: () => { this.CONSUME(ValueSpaceT); }},
        {ALT: () => { this.CONSUME(ValueStringT); }},
        {ALT: () => { this.CONSUME(ValueQuotedStringT); }}
      ]);
    });
  });

In the output CST I get ValueSpaceT: [...], ValueStringT: [...], ValueQuotedStringT: [...]. Now I don’t know what the relative order between each token is, so I cannot concatenate the value of all these tokens.

Compare to the usage of embedded actions: https://github.com/sgarciac/bombadil/blob/master/src/parser.ts#L8-L21 which shows the ability to rely on the exact ordering to perform side effects during the parsing.

[bd82]

I got one of the methods to compile using some casing and “@ts-ignore” comments.

    cstPostTerminal(this:any, key, consumedToken) {
        // "standard" addition to the children dictionary
        // @ts-ignore
        super.cstPostTerminal(key, consumedToken)

        // Add Terminal to childList
        const previousCstNode = this.CST_STACK[this.CST_STACK.length - 1]
        previousCstNode.childList.push(consumedToken)
    }

[bd82]

Looked deeper into this. This could be implemented using the following overrides of the Parser methods:

class MyParserWithCstLists extends Parser {
    cstInvocationStateUpdate(fullRuleName, shortName) {
        // default initialization
        super.cstInvocationStateUpdate(fullRuleName, shortName)

        // Init the child List
        const currentCstNode = this.CST_STACK[this.CST_STACK.length - 1]
        currentCstNode.childList = []
    }

    cstPostNonTerminal(newCstNode, newRuleName) {
        // "standard" addition to the children dictionary
        super.cstPostNonTerminal(newCstNode, newRuleName)

        // Add nonTerminal to childList
        const previousCstNode = this.CST_STACK[this.CST_STACK.length - 1]
        previousCstNode.childList.push(newCstNode)
    }

    cstPostTerminal(key, consumedToken) {
        // "standard" addition to the children dictionary
        super.cstPostTerminal(key, consumedToken)

        // Add Terminal to childList
        const previousCstNode = this.CST_STACK[this.CST_STACK.length - 1]
        previousCstNode.childList.push(consumedToken)
    }

   // ...

Of course the CST visitor would also have to be modified to expose that “childList”. Or alternatively the “childList” itself could be placed on the children dictionary to work around that issue.

I think this example should suffice for now, I will reconsider investing more effort into this (proper interfaces / runnable examples) if and when there would be more requests for such capabilities.