Case sensitivity handling and note about in the docs

Hi! First of all, I’d like to thank You for creating such fast lexer. I’ve been using it along with nearley.js in various projects. It really changed my way of approaching any text parsing related topics.

I’m currently working on a language that requires all tokens to be case-insensitive, without exceptions. For now, following tips that I’ve found over the internet (and issues in this repo), I’ve been using some custom helpers that transform token text into case insensitive regex without the /i flag. This works, however it’s not pretty. Also, even if unreal, I have doubts about the overall performance of my parser.

Why I’m creating this issue? I would like a concise description on how to approach situations where all (or some) tokens are case-insensitive. An example would be nice as well.

Let’s say, that my lexer usage looks like this:

import moo from 'moo';

const lexer = ({

  /* This doesn't care about case sensitivity. */
  STRING: /"(?:[^\\]|\\.)*?"/,

  /* Case sensitivity doesn't apply here. */
  NUMBER: /(?:\.\d+|\d+\.?\d*)/,

  /* Case sensitivity doesn't apply here. */
  ADD: '+',

  /* Manually force case insensitivity */
  IN: ['in', 'iN', 'In', 'IN'],

  /* Use a helper */
  ABS: textToCaseInsensitiveRegex('ABS')
});

[autioch]

Hi again. After some testing and pondering about my code readability, I’ve came to a conclusion that I’ll stick to the textToCaseInsensitiveRegex helper. I’m doing some extra transformations on the lexer rules, so I’ve added “precompiler”, that outputs complete, finished definitions, that I later use in the app. Separating these two things is safer, easier to write tests, debug and finally reduces time for parsing and preparing the JS on the client side.

Helper:

const LETTER_REGEXP = /[a-zA-Z]/;
const isCharLetter= (char) => LETTER_REGEXP.test(char);

function textToCaseInsensitiveRegex(text) {
  const regexSource = text.split('').map((char) => {
    if (isCharLetter(char)) {
      return `[${char.toLowerCase()}${char.toUpperCase()}]`;
    }

    return char;
  });

  return new RegExp(regexSource.join(''));
};

As a side note, it’s cool that moo accepts array as an alternative to object. It’s easier to manipulate and there’s complete certainty about the rules order.

[tjvr]

Like the unicode flag handling in #123, I think it would be reasonable to allow the ignoreCase /i flag if all the RegExps use it. That would handle the case where everything in the language is case-insensitive.

If only some of the RegExps need to be case-insensitive, then you’ll have to generate the cases manually, using something like textToCaseInsensitiveRegex above.