fix editing with "soft keyboards" (eg. Android, IMEs)

Do you want to request a feature or report a bug?

Bug.

What’s the current behavior?

The issue with using Slate on Android is complex, and due to a difference in how its OS keyboard is designed. On mobile devices, keyboards are starting to move away from the “keys” concepts in many ways…

• Autocorrect will make changes without any “key” being press.
• Autosuggest will insert words that don’t map to keys.
• Swipe-to-type will insert entire words in one go, instead of using keys.
• etc.

Because of this, it sounds like the Android team (reasonably from their point of view) decided to not reliably fire key-related events.

As soft input methods can use multiple and inventive ways of inputting text, there is no guarantee that any key press on a soft keyboard will generate a key event: this is left to the IME’s discretion, and in fact sending such events is discouraged. You should never rely on receiving KeyEvents for any key on a soft input method. —KeyEvent, Android Reference

It sounds like the behavior is:

• The keypress event is never triggered, which is fine for us.
• Text-based keys like <kbd>a</kbd>, <kbd>b</kbd>, etc. fire a keydown event but always with a key code of 229, indicating that the key is unidentifiable because the keyboard is still busy processing IME input, which may invalidate the actual key pressed.
• Pressing keys like <kbd>enter</kbd> fires a keydown event as normal, with an event.key that can be recognized? (This is unclear whether this happens or not.)
• Pressing <kbd>backspace</kbd> does not fire a proper keydown event.

A few different resources:

• Great Android postmortem writeup: https://docs.google.com/document/d/1Hex89Di-r-Wfpo1DLAtxpetoX588ziXVoNyC87Je3Xc/edit#
• Chrome “bug”: https://bugs.chromium.org/p/chromium/issues/detail?id=118639
• React issues: https://github.com/facebook/react/issues/4079 https://github.com/facebook/react/issues/6176 https://github.com/facebook/react/issues/11231
• VSCode’s IME test docs: https://github.com/Microsoft/vscode/wiki/IME-Test
• @danburzo’s input methods: https://github.com/danburzo/input-methods

What’s the expected behavior?

The fix for this is also complicated. There are a handful of different, overlapping pieces of logic that need to change, to accommodate a handful of different input types…

The first stage is to handle basic insertions, and auto-suggestions…

• Remove the preventDefault in onBeforeInput, so that the DOM is updated, and the onInput logic will trigger, diffing the insertion and then “fixing” it.
• Update the <Leaf> (and other?) components to increment their key such that React properly unmounts and reconciles the DOM, since it has changed out from under it.

This is actually the same starting steps as is required for https://github.com/ianstormtaylor/slate/issues/2060, so I’d recommend we solve that issue in its entirety first, to work from a solid base.

This fixes the actual text insertion pieces, and probably deletions as well. Splitting blocks can still be handled by <kbd>enter</kbd> because it still provide proper key codes.

• Check that all other behaviors that aren’t text insertions (eg. splitting blocks) are handled properly without access to many keydown events.

And then there’s some selection issues, which apparently can mess up Android’s IME (and potentially others) if the selection is manually changed during a composition.

• Prevent re-rendering the editor on compositionstart and compositionend.
• Prevent updating the selection while a composition is taking place.

I think this would solve the 90% case for soft keyboard input.

---

Separately, there’s still another question of how to properly handle these kinds of behaviors other plugins. For example if a plugin uses <kbd>backspace</kbd> at the start of a block to reset that block, that won’t work on Android. So after solving the input issues, we need to step back to an architectural level and solve this plugin handling problem. But that can wait.

[thesunny]

Hurray! Android support was published to NPM four days ago.

[thesunny]

Mini Update:

I posted a short update in Slack which I’ll summarize and expand on here.

Our project is prioritizing mobile which includes Android support. I will start again on Android within 2-8 weeks and I estimate 2-3 weeks of work which includes:

• Create a dev environment with native devices. Device emulation slows me down 2-3x and crashes often.
• Create an event logging tool specific to Slate. The modified Dan Burzo tool I created differs enough from Slate as to be confusing.
• Create a mobile demo for easier mobile testing
• Start from a clean version of the repo because there is too much testing and experimental cruft in my current version.
• Do a clean rewrite of the current helper functions and modularize them.
• Implement as much as I can as a plugin
• Create a pseudo compositionbegin, compositionchange and compositionfinish event and block all other events in between for the Android platform. These events roughly equate to (a) we guarantee that this is the state before the composition started (b) something might have changed within the composition so note the cursor position here © we guarantee that the composition has ended and that the results of the composition have been commited to the DOM so it is safe to update Slate State and re-render. These roughly equate to the idea of compositionstart, compositionupdate and compositionend but abstracts away the problems and inconsistencies between them. Note: We may wish to implement this across all platforms.