Context sensitive tokens

Following the discussions on #993 and #1023, I’m wondering if it’d make sense to have an option for stateful tokens.

const stringStartAndEnd = createToken({
  effect: ({ insideString }) => ({ insideString: !insideString }),
  pattern: /'/
});

const literal = createToken({
  gate: ({ insideString }) => insideString,
  pattern: /[^']*/
});

For this to work, the lexer would be initiated with a state variable, an empty object at first. If present, the effect function of a token is invoked whenever encountered during lexing:

state = { ...state, ...token.effect(state)) };

The gate property of a token is called with the state variable before a match.

@bd82 does this make any sense?

[bd82]

Perhaps we could add a section to the lexer tutorial page that mentions it?

It seems like this feature is more popular than I thought…

Having a full guide e.g:

• https://sap.github.io/chevrotain/docs/guide/generating_syntax_diagrams.html#examples

Would be best but that it a fair bit of work to create…

Perhaps (as suggested) a small note could be added in the lexer tutorial to aid discover-ability.

[HoldYourWaffle]

Fellow noobs should help each other out right 😉

@bd82 Maybe it’s a good idea to make multi-mode lexing more prevalent in the documentation? There isn’t really a tutorial page for it (although the linked example is plenty to understand how it works) and I didn’t find the page until you linked to it from another issue. Perhaps we could add a section to the lexer tutorial page that mentions it?