Retention policy is applied 1 hour earlier

Describe the bug

As @Tho-Mat pointed out in https://github.com/corona-warn-app/cwa-server/issues/699#issuecomment-671358818, retention policy applied 1 hour earlier than it should:

I also noticed that the index for (examples) the hour-file

> 2020-07-26-hour-06.zip created 26.07.2020 09:05 (German Time)
> was removed at                 09.08.2020 08:05 (German Time)
> 
> 2020-07-26-hour-08.zip created 26.07.2020 11:05 (German Time)
> was removed at                 09.08.2020 10:05 (German Time)

shouldn’t they be removed one hour later to have them 14 Days on the server?

Expected behavior

Retention policy should apply after 14 days.

[fredrb]

Update from our side: release 1.4 is currently planned for 30.09.2020.

[fredrb]

I figured out what is going on. Your analysis is correct, the index file deletes the earliest hour one hour too early.

The problem that we’re facing happens because of the retention policy of the diagnosis keys.

To give an overview of what and why happens:

1. As previously stated in this thread, the retention policy for the object store either deletes the whole day or nothing. This means that regardless of the current hour, the files can still be fetched on the CDN. However, in the index file, we can see that entries are removed hourly. And they are removed one hour too soon.
2. The index file is created by the Assembly process, based on the keys mapped to be exported into the hourly files.
3. The mapping of keys is a simple DB read aggregating keys by their submission timestamp.

During the retention policy step, we delete all the keys from the database that match the query submission_timestamp<= threshold where threshold is now - retention_policy. This means that if we run the retention policy now (14.08 at 14:00), it will delete any entries before or equal (31.07 14:00). Meaning that keys generated for 31.07 14:00 will be deleted, which is not correct, since we are now distributing keys generated between 13:00-14:00 today.

To illustrate this better, we can assume a scenario with single day retention policy running distribution day 2 at 8 am.

Wrong behavior

Day 1	Day 2
	00:00
	01:00
	02:00
	03:00
	04:00
	05:00
	06:00
	07:00 --> Created
08:00 --> Deleted
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00

This is wrong because we are only holding 23 hours worth of data. Hour 7 is created at 8:00 but removed at 7:00 the next day.

Correct behavior

Day 1	Day 2
	00:00
	01:00
	02:00
	03:00
	04:00
	05:00
	06:00
07:00 --> Delete	07:00 --> Created
08:00
09:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00

We should delete everything before 8am (not inclusive), to assure that we will always have 24h of keys in the system.

There’s already a fix for that, will open the PR in a couple of minutes. @Tho-Mat would also appreciate your comments there. Thanks again for your help investigating and explaining this issue.