Memory Error & No Auto Complete when binary flict is large ~ 6mb

Referring to this feature

I generated a binary flict using the dict-tools for en where size was 6.8 mb – about 3 times the default en binary flict. Size is larger due to more words and the highest dimension for n-grams used was 2. However when file is reduced to a small size, significantly fewer tokens, word-complete works fine.

Short description

Memory runs out when flict is too large

Steps to reproduce

1. Generate large flict > 6mb
2. Type an entire paragraph as quickly as possible
3. No auto-complete will show and keyboard will quickly crash in seconds
4. See error

Environment information

• FlorisBoard Version: current main branch, commit 38baac1af92fea80a86f5fdd8850bc677a27a3d2.
• Install Source: Github
• Device: Emulator Android Studio
• Android version, ROM: Android 11

~~~ 1614445172221.stacktrace ~~~

java.lang.OutOfMemoryError: OutOfMemoryError thrown while trying to throw an exception; no stack trace available
~~~ 1614445172167.stacktrace ~~~

java.lang.OutOfMemoryError: Failed to allocate a 80 byte allocation with 8 free bytes and 8B until OOM, target footprint 201326592, growth limit 201326592
	at java.util.Arrays.copyOf(Arrays.java:3257)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
	at java.lang.StringBuilder.append(StringBuilder.java:137)
	at dev.patrickgold.florisboard.ime.core.FlorisBoard.onUpdateSelection(FlorisBoard.kt:388)
	at android.inputmethodservice.InputMethodService$InputMethodSessionImpl.updateSelection(InputMethodService.java:906)
	at android.inputmethodservice.IInputMethodSessionWrapper.executeMessage(IInputMethodSessionWrapper.java:104)
	at com.android.internal.os.HandlerCaller$MyHandler.handleMessage(HandlerCaller.java:44)
	at android.os.Handler.dispatchMessage(Handler.java:106)
	at android.os.Looper.loop(Looper.java:223)
	at android.app.ActivityThread.main(ActivityThread.java:7656)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)

@patrickgold

[patrickgold]

@tsiflimagas Good to know, thanks for linking!

[patrickgold]

Since Luminiso doesn’t support higher n-grams, is fixing the calculation error in flict.py part of phase two work? Or will the solution be more about reducing memory so that large binaries don’t cause a crash?

No the end bug is something I prefer to have included still in phase 1, because it reduces the file size and also reduces the amount of bytes read into memory in runtime when the binary file is cached.