Can not analyze symbols with Chinese

Can not analyze the Chinese function name, variable name, please add the analysis of Chinese function name and variable name support. Luajit can support gbk or utf8 Chinese function name and variable name.

example ` function 中文函数名(参数1,参数2) local 中文变量 = “Chinese variable name” end

[ghost]

@fstirlitz

That at least makes it simple to implement; I worried we might have to import the Unicode character database to check character properties or something.

I was able to convert a UnicodeData.txt into a lightweight 2MB string map of general categories recently. It’s something like '²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²¯ªªª¬ªªª¦§ª«ª¥ªªCCCCCCCCCCªª«««ªª ¦ª§­D­!!!!!!!!!!!!!!!!!!!!!!!!!!¦«§«²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²²¯ª¬¬¬¬®®­®!¨«²®­®«¤¤­!®ª­ [more GCs]'.

package hydroper.unicode
{
	/**
	 * The UnicodeType static class. 
	 */
	public class UnicodeType
	{
		public static const LETTER_UPPERCASE: uint = ' '.charCodeAt(0);
		public static const LETTER_LOWERCASE: uint = '!'.charCodeAt(0);
		public static const LETTER_TITLECASE: uint = '#'.charCodeAt(0);
		public static const LETTER_MODIFIER: uint = '$'.charCodeAt(0);
		public static const LETTER_OTHER: uint = '%'.charCodeAt(0);

		public static const MARK_NON_SPACING: uint = 'A'.charCodeAt(0);
		public static const MARK_SPACING_COMBINING: uint = 'B'.charCodeAt(0);
		public static const MARK_ENCLOSING: uint = 0xA3;

		public static const NUMBER_DECIMAL_DIGIT: uint = 'C'.charCodeAt(0);
		public static const NUMBER_LETTER: uint = '&'.charCodeAt(0);
		public static const NUMBER_OTHER: uint = 0xA4;

		public static const PUNCTUATION_CONNECTOR: uint = 'D'.charCodeAt(0);
		public static const PUNCTUATION_DASH: uint = 0xA5;
		public static const PUNCTUATION_OPEN: uint = 0xA6;
		public static const PUNCTUATION_CLOSE: uint = 0xA7;
		public static const PUNCTUATION_INITIAL_QUOTE: uint = 0xA8;
		public static const PUNCTUATION_FINAL_QUOTE: uint = 0xA9;
		public static const PUNCTUATION_OTHER: uint = 0xAA;

		public static const SYMBOL_MATH: uint = 0xAB;
		public static const SYMBOL_CURRENCY: uint = 0xAC;
		public static const SYMBOL_MODIFIER: uint = 0xAD;
		public static const SYMBOL_OTHER: uint = 0xAE;

		public static const SEPARATOR_SPACE: uint = 0xAF;
		public static const SEPARATOR_LINE: uint = 0xB0;
		public static const SEPARATOR_PARAGRAPH: uint = 0xB1;

		public static const OTHER_CONTROL: uint = 0xB2;
		public static const OTHER_FORMAT: uint = 0xB3;
		public static const OTHER_SURROGATE: uint = 0xB4;
		public static const OTHER_PRIVATE_USE: uint = 0xB5;
		public static const OTHER_NOT_ASSIGNED: uint = 0xB6;

		// private static const data: String = /* ... 2MB ...  */;

		[Inline]
		public static function getType(cp: uint): uint
		{
			return data.charCodeAt(cp);
		}
	}
}

There’s also the UnicodeSet tool for that, but it outputs a pattern-like set instead, with range elements. Range checking is generally slower than indexing into a string literal map.

[fstirlitz]

Implemented in 71729404772a771e588a6c0ca2c70d3db7f9f254. Consider it unstable, however. I might still revisit the encoding issue.