Consider UTF-8?

We are using UTF-16, which has some cross platform compatibility issues, starting with different type used on different platforms - wchar_t on Windows and char16_t everywhere else, with former being a fundamental type (not requiring an include), but latter - defined as an unsigned short. This is thanks to 'nix toolchains picking UTF-32 (which is excessive for most uses) as default Unicode type.

UTF-8 is more cross platform, and supported in Windows since Win10. For prior versions of Windows it should be possible to convert to UTF-16 for things that require it. A quick search shows that V8 might use UTF-8.

I am not sure whether or not UTF-8 would be easier for our text-related projects, like RegEx support. @rhuanjl do you have any thoughts?

[rhuanjl]

Moving to using UTF-8 internally would significantly reduce memory usage in many scenarios, just because you end up saving 50% of memory usage for basic latin characters, which work out to 99.99% of all strings in js.

IMO this is a really strong argument for it

How to ensure that the observable behaviour matches UTF16 for points defined in the spec is the key challenge though.

One option (if memory efficiency is the aim) would be to stick to 8 bit chars for ascii characters and use utf16 strings whenever non-ascii was needed - though this would be a complex change AND wouldn’t target the initial motivation of this discussion (getting rid of the wchar type)

[ljharb]

Also, I’m very skeptical that the very high usage of emoji in vernacular discussion hasn’t leaked into JS strings.