Fix for #1991 causes DoesNotExist error during row iteration

I have a rather large cross-model join query that started throwing an exception after updating from 3.9.6 to 3.13.1. This appears to be caused by the fix for #1991 .

The query is:

        for inc in (self.vulnerabilities
                        .select(fn.COALESCE(Team.guid, Vulnerability.guid).alias('guid'),
                                fn.COALESCE(Team.remedy_group, self.name).alias('name'),
                                Plugin.id,
                                fn.COUNT(Vulnerability.host_id).alias('count'),
                                fn.JSON_GROUP_ARRAY(Vulnerability.host_id).python_value(str).alias('host_ids'))
                        .join(Plugin, src=Vulnerability)
                        .join(Query, src=Vulnerability)
                        .join(Host, src=Vulnerability)
                        .join(Team, join_type=JOIN.LEFT_OUTER, on=(Team.name == team_name), src=Host)
                        .where(Query.process << INCIDENT_PROCESS_PLUGIN)
                        .where(Host.state != HOST_STATE_TERMINATED)
                        .group_by(fn.COALESCE(Team.guid, Vulnerability.guid),
                                  fn.COALESCE(Team.remedy_group, self.name),
                                  Plugin.id)
                        .order_by(fn.COALESCE(Team.guid, Vulnerability.guid),
                                  fn.COALESCE(Team.remedy_group, self.name),
                                  Plugin.id)):
            # do something

The resulting query SQL that gets logged is:

('SELECT COALESCE("t1"."guid", "t2"."guid") AS "guid", COALESCE("t1"."remedy_group", ?) AS "name", "t3"."id", COUNT("t2"."host_id") AS "count", JSON_GROUP_ARRAY("t2"."host_id") AS "host_ids" FROM "vulnerability" AS "t2" INNER JOIN "plugin" AS "t3" ON ("t2"."plugin_id" = "t3"."id") INNER JOIN "query" AS "t4" ON ("t2"."query_id" = "t4"."id") INNER JOIN "host" AS "t5" ON ("t2"."host_id" = "t5"."ip_address") LEFT OUTER JOIN "team" AS "t1" ON ("t1"."name" = (SELECT "t1"."name" FROM "team" AS "t1" WHERE (("t1"."name" = TRIM(json_extract("t5"."tags", ?))) OR ("t1"."remedy_group" = TRIM(json_extract("t5"."tags", ?)))) LIMIT ?)) WHERE ((("t2"."remediation_group_id" = ?) AND ("t4"."process" IN (?, ?))) AND ("t5"."state" != ?)) GROUP BY COALESCE("t1"."guid", "t2"."guid"), COALESCE("t1"."remedy_group", ?), "t3"."id" ORDER BY COALESCE("t1"."guid", "t2"."guid"), COALESCE("t1"."remedy_group", ?), "t3"."id"', [u'Server Operations', u'$.TechOwnerName', u'$.RemedyGroup', 1, 764, u'standard', u'compliance', u'terminated', u'Server Operations', u'Server Operations'])

The exception is:

  File "/var/task/ticketlib/remediation.py", line 232, in get_known_tickets
    Plugin.id)):
  File "/var/task/peewee.py", line 4287, in next
    self.cursor_wrapper.iterate()
  File "/var/task/peewee.py", line 4206, in iterate
    result = self.process_row(row)
  File "/var/task/peewee.py", line 7450, in process_row
    if instance not in set_keys and dest not in set_keys \
  File "/var/task/peewee.py", line 6483, in __hash__
    return hash((self.__class__, self._pk))
  File "/var/task/peewee.py", line 6374, in get_id
    return getattr(self, self._meta.primary_key.safe_name)
  File "/var/task/peewee.py", line 5382, in __get__
    for field_name in self.field_names])
  File "/var/task/peewee.py", line 4332, in __get__
    return self.get_rel_instance(instance)
  File "/var/task/peewee.py", line 4327, in get_rel_instance
    raise self.rel_model.DoesNotExist
HostDoesNotExist  

It’s failing in the iterator, without any iterations of the for loop running. I am guessing I somehow have a Vulnerability with a host_id set to a value that doesn’t have a corresponding PK in the Host table, but I can’t even try to track that down because it’s blowing up the iterator immediately.

[coleifer]

I’ll fix the tests, thanks.

[brandond]

I’ll give that change a try, but the tests have all failed?