Missing Index on diagnosis_key.submission_timestamp

Disclaimer: I assume, since this application is not complete, there will change a couple of things and maybe this was already planned. This issues it at least meant as a reminder.

Current Implementation

The current Flyway migration script db/migration/postgres/V1__createTables.sql creates only the table diagnosis_key itself.

There are already two data base accesses to the column submission_timestamp in DiagnosisKeyService:

  public List<DiagnosisKey> getDiagnosisKeys() {
    return keyRepository.findAll(Sort.by(Sort.Direction.ASC, "submissionTimestamp"));
  }

and

  public void applyRetentionPolicy(int daysToRetain) {
    long threshold = LocalDateTime
        .ofInstant(Instant.now(), UTC)
        .minusDays(daysToRetain)
        .toEpochSecond(UTC) / 3600L;
    keyRepository.deleteBySubmissionTimestampIsLessThanEqual(threshold);
  }

Suggested Enhancement

Depending on the expected number of entries in this table (I assume it will be a lot), this will result in very slow full table scans. There should be an index added to the column submission_timestamp.

Expected Benefits

• Much faster data base access on high load and high number of db entries.

[christian-kirschnick]

Yeah, the TestDataGeneration class would need to be changed accordingly.

I just ran the retention for ~250k keys and I guess we should add the index:

2020-06-22T14:27:57+0200 INFO  main a.c.s.c.p.s.DiagnosisKeyService[74261]: Deleted 244717 diagnosis key(s) with a submission timestamp older than 14 day(s) ago. 
2020-06-22T14:32:45+0200 DEBUG main a.c.s.s.d.r.RetentionPolicy[74261]: Retention policy applied successfully. 

The first logging statement is misleading, as it is actually not reporting the end of the deletion task - so the actual deletion happens between both statements. That delete took 5 minutes on my machine, which is far too long.

[michael-burwig]

Hi Jan!

I also agree that the index will most likely make sense here.

As for the anticipated performance gains wrt full table scans I am not 100% sure whether the optimizer would care much for the index. See Postgres Documentation - 11.4. Indexes and ORDER BY: …For a query that requires scanning a large fraction of the table, the explicit sort is likely to be faster because it requires less disk I/O due to a better-ordered access pattern. Indexes are more useful when only a few rows need be fetched…