Sqlite3 database is locked error

Describe the bug I receive a sqlite3.OperationalError: database is locked error when syncing with Stripe with webhook.

To Reproduce Steps to reproduce the behavior:

1. Enable stripe webhook to dj-stripe
2. Start local stripe webhook CLI
3. Run the command: stripe trigger checkout.session.completed
4. See attached error on webhook

Expected behavior I expect that django would handle the webhook without throwing a database is locked error. I want to use the webhook to update my account model.

If relevant it’s very helpful to include webhook tracebacks and content (note that these are logged in the database and are visible in django admin - eg http://127.0.0.1:8000/admin/djstripe/webhookeventtrigger/ )

Traceback (most recent call last):
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 416, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:
Exception: database is locked

Traceback (most recent call last):
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/djstripe/models/webhooks.py", line 194, in from_request
obj.process(save=False)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/djstripe/models/webhooks.py", line 285, in process
self.event = Event.process(self.json_body)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/djstripe/models/core.py", line 1524, in process
ret = cls._create_from_stripe_object(data)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/djstripe/models/base.py", line 585, in _create_from_stripe_object
instance.save(force_insert=True)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/base.py", line 743, in save
self.save_base(using=using, force_insert=force_insert,
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/base.py", line 780, in save_base
updated = self._save_table(
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/base.py", line 885, in _save_table
results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/base.py", line 923, in _do_insert
return manager._insert(
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1301, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1441, in execute_sql
cursor.execute(sql, params)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 99, in execute
return super().execute(sql, params)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute
with self.db.wrap_database_errors:
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/Users/endys/Desktop/Planly/venv/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 416, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database is locked

Environment

• dj-stripe version: 2.6.0
• Your Stripe account’s default API version: 2020-08-27
• Database: sqlite3
• Python version: 3.10
• Django version: 4.0.1

Can you reproduce the issue with the latest version of master?

Yes

[MicahLyle]

I think I’ve noticed that, even in PostgreSQL, since djstripe 2.6.0 (I could be wrong about the since which version), there seems to be more deadlocks on the Sentry logs.

Here’s where it can become a problem. I have some older (like ~3 year old code) that does some sanity check things with default sources that looks like this.

    if set_default:
        stripe_customer.default_source = stripe_source["id"]
        stripe_customer.save()
    new_payment_method = DjstripePaymentMethod.from_stripe_object(stripe_source)
    if set_default:
        customer.default_source = new_payment_method
        customer.save()

That code is part of a critical production workflow that involves paying and activating something (all in one step). It’s on the latest djstripe version.

For the first time ever, the other day, I had the production code there throw a 500 because of a deadlock with djstripe’s webhook processing. My plan is to catch the deadlock exception by checking for that type of error with "deadlock detected" in the exception string, and then retry a number of times, maybe with time.sleep in-between since it’s just so rare. I probably have to do it though because it’s a critical endpoint that really shouldn’t fail. I’m hopeful that will work fine, but, I’m not sure if I’ll have to do extra work with the transaction due to a deadlock actually occurring where I’ll have to abort the transaction and retry the entire request, etc.

[jleclanche]

Will leave it as a valid issue but to be frank, sqlite backend is meant to be used at most for loose testing; anything production-worthy will be using a different backend. Webhooks not working on it is not completely unsurprising.

Still, will look into it.