Allow to extend newline codepoints

As far as I can tell from https://github.com/SAP/chevrotain/blob/b3699ec61c0b41d736bc5c6783a36afb3acbdfe2/src/scan/lexer_public.ts#L588-L604, currently only CR, LF and CR LF are treated as newlines.

While this is correct for most languages, some languages like JavaScript and derivatives have extra codepoints that must be treated as newlines for location purposes:

Table 33: Line Terminator Code Points

Code Point	Unicode Name	Abbreviation
U+000A	LINE FEED (LF)	<LF>
U+000D	CARRIAGE RETURN (CR)	<CR>
U+2028	LINE SEPARATOR	<LS>
U+2029	PARAGRAPH SEPARATOR	<PS>

Would it be possible to make line terminators a regexp, just like whitespaces are? Then default would be \r\n?|\n or similar, but user would be able to override it with custom chars. Or would it have significant performance implications?

[bd82]

Solution

We only require a subset of the regExp API to support this code block.

nlRegExpLike.lastIndex = 0
do {
nl = nlRegExpLike.exec(matchedImage)
if (nl !== null) {
      lastLTIdx = nl.index
      numOfLTsInMatch++
   }
} while (nl)

An faster optimized version can be written without using regExp that still conforms to the required API.

• Example Source
• Tested this optimization and at most it causes a small (1-2%) degradation.

This means:

• The default built-in implementation for new lines can be implemented using the optimized form.
• Users can still supply their own regExp literals to override the default but at a performance cost.
• Advanced users can also write custom implementations using the optimized form.

So virtually no regressions and increased customization, a win win.

[bd82]

Fixed and merged https://github.com/SAP/chevrotain/issues/523

Will release a new version with this later today.