"sqlite3.OperationalError: too many SQL variables" when trying to import large channels

Observed behavior

When trying to import significantly large amounts of contents have experienced this error -

ERROR 2020-08-17 09:20:24,790 kolibri.core.tasks.worker Job 737e2e6f419f498181e9c9cdb629102c raised an exception: Traceback (most recent call last):
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1283, in _execute_context
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/default.py", line 590, in do_execute
sqlite3.OperationalError: too many SQL variables

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/kivy/build/android/kolibri/core/tasks/worker.py", line 72, in handle_finished_future
  File "/home/kivy/.local/share/python-for-android/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/Lib/concurrent/futures/_base.py", line 432, in result
  File "/home/kivy/.local/share/python-for-android/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/Lib/concurrent/futures/_base.py", line 388, in __get_result
  File "/home/kivy/.local/share/python-for-android/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/Lib/concurrent/futures/thread.py", line 57, in run
  File "/home/kivy/build/android/kolibri/core/tasks/worker.py", line 217, in wrap
  File "/home/kivy/build/android/kolibri/core/tasks/job.py", line 194, in y
  File "/home/kivy/build/android/kolibri/dist/django/core/management/__init__.py", line 131, in call_command
  File "/home/kivy/build/android/kolibri/dist/django/core/management/base.py", line 330, in execute
  File "/home/kivy/build/android/kolibri/core/tasks/management/commands/base.py", line 113, in handle
  File "/home/kivy/build/android/kolibri/core/content/management/commands/importcontent.py", line 451, in handle_async
  File "/home/kivy/build/android/kolibri/core/content/management/commands/importcontent.py", line 180, in copy_content
  File "/home/kivy/build/android/kolibri/core/content/management/commands/importcontent.py", line 366, in _transfer
  File "/home/kivy/build/android/kolibri/core/content/utils/annotation.py", line 659, in set_content_visibility
  File "/home/kivy/build/android/kolibri/core/content/utils/annotation.py", line 389, in mark_local_files_as_available
  File "/home/kivy/build/android/kolibri/core/content/utils/annotation.py", line 409, in mark_local_files_availability
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1020, in execute
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1133, in _execute_clauseelement
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1323, in _execute_context
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1517, in _handle_dbapi_exception
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/util/compat.py", line 178, in raise_
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/base.py", line 1283, in _execute_context
  File "/home/kivy/build/android/kolibri/dist/sqlalchemy/engine/default.py", line 590, in do_execute
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) too many SQL variables
[SQL: UPDATE content_localfile SET available=? WHERE content_localfile.id IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 

“Large amount” could be resources totalling to around 100 MB+ from channels such as “Khan Academy” or “CK-12”. It is mostly the number of resources that seem to matter than the size.

The imports are reported as failed in the UI. If upon subsequent attempts we import smaller chunks (<100 MB) of resources, the imports progress much quickly and are reported as successful in the UI as well. This also indicates that the content transfer is happening properly and it is only the post processing part that seems to fail.

Have experienced this in imports irrespective of source and destination platforms. Either could be Windows, Linux or Android.

Expected behavior

Channel resource imports should not fail.

User-facing consequences

Very inconvenient to get started when large amounts of resources can’t be imported at once.

Errors and logs

Attached log contains multiple instances of sqlite3.OperationalError: too many SQL variable errors.

kolibri.txt

Steps to reproduce

Initiate an import of large number of resources i.e. at least size worth of 100 MB from channels such as KA or CK-12. For example, all Math Grade 1 content from KA.

Context

Kolibri 0.14.1 Irrespective of environment.

[jamalex]

Would you be able to give it a try using the following build from #7479?

• WHL
• PEX

Thanks!

[intelliant01]

Yes the import did succeed. I shall be watchful of reappearance of this and shall file a new issue thereafter only. Have not been able to replicate this since its occurance.