`deletecontent` command on a large topic + `force_delete` option can fail with "too many SQL variables" error
See original GitHub issue_Originally posted by @radinamatic in https://github.com/learningequality/kolibri/issues/7669#issuecomment-729255452_
Observed behavior
On Windows 7 VM running Kolibri 0.14.4-rc1, if you attempt to delete the “Khan Academy > Arithmetic” topic and also select the “Delete everywhere” option on the modal. The task will eventually fail. The traceback indicates that it might be due to the “Delete everywhere” (aka force deletion) option, which causes the SQL query to become too large.
Traceback from logs:
ERROR 2020-11-17 14:15:33,731 kolibri.core.tasks.worker Job b1ee8580831c42f6b76b2f646fb3ed74 raised an exception: Traceback (most recent call last):
File "c:\python36\lib\site-packages\kolibri\dist\django\db\backends\utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "c:\python36\lib\site-packages\kolibri\dist\django\db\backends\sqlite3\base.py", line 328, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: too many SQL variables
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "c:\python36\lib\site-packages\kolibri\core\tasks\worker.py", line 72, in handle_finished_future
result = future.result()
File "c:\python36\lib\concurrent\futures\_base.py", line 425, in result
return self.__get_result()
File "c:\python36\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
File "c:\python36\lib\concurrent\futures\thread.py", line 56, in run
result = self.fn(*self.args, **self.kwargs)
File "c:\python36\lib\site-packages\kolibri\core\tasks\worker.py", line 217, in wrap
return f(*args, **kwargs)
File "c:\python36\lib\site-packages\kolibri\core\tasks\job.py", line 194, in y
result = func(*args, **kwargs)
File "c:\python36\lib\site-packages\kolibri\dist\django\core\management\__init__.py", line 131, in call_command
return command.execute(*args, **defaults)
File "c:\python36\lib\site-packages\kolibri\dist\django\core\management\base.py", line 330, in execute
output = self.handle(*args, **options)
File "c:\python36\lib\site-packages\kolibri\core\tasks\management\commands\base.py", line 119, in handle
return self.handle_async(*args, **options)
File "c:\python36\lib\site-packages\kolibri\core\content\management\commands\deletecontent.py", line 123, in handle_async
channel, node_ids, exclude_node_ids, force_delete
File "c:\python36\lib\site-packages\kolibri\core\content\management\commands\deletecontent.py", line 49, in delete_metadata
propagate_forced_localfile_removal(unused_files)
File "c:\python36\lib\site-packages\kolibri\core\content\utils\annotation.py", line 666, in propagate_forced_localfile_removal
ContentNode.objects.filter(files__in=files).update(available=False)
File "c:\python36\lib\site-packages\kolibri\dist\django\db\models\query.py", line 650, in update
rows = query.get_compiler(self.db).execute_sql(CURSOR)
File "c:\python36\lib\site-packages\kolibri\dist\django\db\models\sql\compiler.py", line 1204, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "c:\python36\lib\site-packages\kolibri\dist\django\db\models\sql\compiler.py", line 899, in execute_sql
raise original_exception
File "c:\python36\lib\site-packages\kolibri\dist\django\db\models\sql\compiler.py", line 889, in execute_sql
cursor.execute(sql, params)
File "c:\python36\lib\site-packages\kolibri\dist\django\db\backends\utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "c:\python36\lib\site-packages\kolibri\dist\django\db\utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "c:\python36\lib\site-packages\kolibri\dist\django\utils\six.py", line 685, in reraise
raise value.with_traceback(tb)
File "c:\python36\lib\site-packages\kolibri\dist\django\db\backends\utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "c:\python36\lib\site-packages\kolibri\dist\django\db\backends\sqlite3\base.py", line 328, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: too many SQL variables
Different versions of this scenario don’t produce this result, namely
- Deleting KA > Arithmetic without the “delete everywhere option”
- Deleting the entire channel
Expected behavior
Ideally, no errors deleting large topics with the “delete everywhere” option
User-facing consequences
…
Errors and logs
Steps to reproduce
…
Context
Issue Analytics
- State:
- Created 3 years ago
- Comments:10 (6 by maintainers)
Top Results From Across the Web
Delete bulk email from many mailboxes - TechNet - Microsoft
I would now like to construct a query in EMS to delete these emails, but I'm not very familiar with PowerShell/EMS.
Read more >How to Remove Content Database in SharePoint using ...
Just Open SQL Server Management Studio, From the list of Content Databases, find the one that needs to be deleted, Right-click, and delete!...
Read more >too many SQL variables and NOT IN clause - Stack Overflow
How can i solve it? One solution I think is to insert all the ids in a temporal table and then execute: DELETE...
Read more >force delete kafka topic Code Example - Grepper
Topic remove-me is marked for deletion. ... Note: This will have no impact if delete.topic.enable is not set to true.
Read more >https://www2.sqlite.org/src/vpatch?from=e181225dc7...
+\`configure' configures sqlite 3.21.0 to adapt to many kinds of systems. ... +7005,15 @@ # The option is referenced via a variable to...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Yes, I think this is the most reliable way to reproduce it.
Need to also run the task runner using the ‘services’ command https://github.com/learningequality/kolibri/blob/release-v0.14.x/package.json#L23