checkout.session.completed and customer.subscription.created failing

Describe the bug When a customer subscribes to our service, 2 webhooks are triggered:

1. checkout.session.completed
2. customer.subscription.created

In djstripe 2.4.4 and 2.5.1, these webhooks work perfectly fine. In 2.6.0, they both fail with the following 2 errors: django.db.utils.IntegrityError: duplicate key value violates unique constraint "djstripe_subscription_stripe_id_key" and then djstripe.models.billing.Subscription.DoesNotExist: Subscription matching query does not exist. which is likely the result of the first error. Stack traces below.

I reverted back to 2.4.4, both webhooks worked. I then tried upgrading to 2.5.1, and both webhooks work. 2.6.0 seems to be where the problems start.

To Reproduce Steps to reproduce the behavior:

e.g.:

1. Enable checkout.session.completed and customer.subscription.created stripe webhooks to dj-stripe
2. Complete a checkout session for a subscription
3. Error should show as above.

Expected behavior It is expected that the webhooks are processed, including creation of the subscription and payments.

Traceback for checkout.session.completed:

Traceback (most recent call last):
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "djstripe_subscription_stripe_id_key"
DETAIL:  Key (id)=(sub_1KINUVHJBSW1P7PeQT9mlU50) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 693, in _get_or_create_from_stripe_object
    cls._create_from_stripe_object(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 585, in _create_from_stripe_object
    instance.save(force_insert=True)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "djstripe_subscription_stripe_id_key"
DETAIL:  Key (id)=(sub_1KINUVHJBSW1P7PeQT9mlU50) already exists.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/webhooks.py", line 194, in from_request
    obj.process(save=False)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/webhooks.py", line 285, in process
    self.event = Event.process(self.json_body)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/core.py", line 1525, in process
    ret.invoke_webhook_handlers()
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/core.py", line 1537, in invoke_webhook_handlers
    webhooks.call_handlers(event=self)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/webhooks.py", line 98, in call_handlers
    handler_func(event=event)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/event_handlers.py", line 318, in other_object_webhook_handler
    _handle_crud_like_event(target_cls=target_cls, event=event)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/event_handlers.py", line 427, in _handle_crud_like_event
    obj = target_cls.sync_from_stripe_data(data)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 935, in sync_from_stripe_data
    instance, created = cls._get_or_create_from_stripe_object(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 693, in _get_or_create_from_stripe_object
    cls._create_from_stripe_object(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 575, in _create_from_stripe_object
    **cls._stripe_object_to_record(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 356, in _stripe_object_to_record
    field_data, skip, is_nulled = cls._stripe_object_field_to_foreign_key(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 469, in _stripe_object_field_to_foreign_key
    field_data, _ = field.related_model._get_or_create_from_stripe_object(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 708, in _get_or_create_from_stripe_object
    return cls.stripe_objects.get(id=id_), False
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
djstripe.models.billing.Subscription.DoesNotExist: Subscription matching query does not exist.

Traceback for customer.subscription.created:

Traceback (most recent call last):
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "djstripe_subscription_stripe_id_key"
DETAIL:  Key (id)=(sub_1KINUVHJBSW1P7PeQT9mlU50) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 693, in _get_or_create_from_stripe_object
    cls._create_from_stripe_object(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 585, in _create_from_stripe_object
    instance.save(force_insert=True)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "djstripe_subscription_stripe_id_key"
DETAIL:  Key (id)=(sub_1KINUVHJBSW1P7PeQT9mlU50) already exists.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/webhooks.py", line 194, in from_request
    obj.process(save=False)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/webhooks.py", line 285, in process
    self.event = Event.process(self.json_body)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/core.py", line 1525, in process
    ret.invoke_webhook_handlers()
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/core.py", line 1537, in invoke_webhook_handlers
    webhooks.call_handlers(event=self)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/webhooks.py", line 98, in call_handlers
    handler_func(event=event)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/event_handlers.py", line 161, in customer_subscription_webhook_handler
    _handle_crud_like_event(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/event_handlers.py", line 427, in _handle_crud_like_event
    obj = target_cls.sync_from_stripe_data(data)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 935, in sync_from_stripe_data
    instance, created = cls._get_or_create_from_stripe_object(
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/djstripe/models/base.py", line 708, in _get_or_create_from_stripe_object
    return cls.stripe_objects.get(id=id_), False
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/username/.virtualenvs/venv/lib/python3.8/site-packages/django/db/models/query.py", line 435, in get
    raise self.model.DoesNotExist(
djstripe.models.billing.Subscription.DoesNotExist: Subscription matching query does not exist.

Environment

• dj-stripe version: 2.6.0
• Your Stripe account’s default API version: 2020-08-27
• Database: PostgreSQL 11.13
• Python version: 3.8.1
• Django version: 3.2.11

Can you reproduce the issue with the latest version of master?

Yes

[ebk46]

@arnav13081994 Thanks Arnav! Hope you feel better soon (not just for my sake!). Please let me know if there’s any way I can help figure this out.

[arnav13081994]

@ebk46 I am still not able to replicate the issue and I suspect the root cause of the issue has nothing to do with latest_invoice on the Subscription model. I think it is a deeper issue with the way we handle webhooks relying on the CPU to handle the webhooks which causes them to be paused midway as we do not handle webhooks using cooperative multi-tasking yet.

So for now I have created this PR on my fork on dj-stripe so please install dj-stripe from that and check to see if you still face this issue or not. I have reverted latest_invoice. This would be a “band-aid” solution to this issue.

You can install dj-stripe like so:

pip install git+https://github.com/arnav13081994/dj-stripe.git@revert_latest_invoice