WAL-E Clone should not use last_modified for recovery target time evaluation

Hello everyone,

we’re currently facing problems with wal-e clone when copying backups from one point to another because the last modified flag is used for evaluation of the recovery target time. In our opinion, the start_time or finish_time flag should be used because it is insensitive against change of metadata because of copy or move operations. If our assumption is correct, I would create a PR for it.

def choose_backup(backup_list, recovery_target_time):
    """ pick up the latest backup file starting before time recovery_target_time"""

    match_timestamp = match = None
    for backup in backup_list:
        last_modified = parse(backup['last_modified'])
        if last_modified < recovery_target_time:
            if match is None or last_modified > match_timestamp:
                match = backup
                match_timestamp = last_modified
    if match is not None:
        return match['name']

https://github.com/zalando/spilo/blob/9cbf2094d7cf9eb95081c60db540b1d4e793a55f/postgres-appliance/bootstrap/clone_with_wale.py#L70

Thank you 😃

[CyberDem0n]

Do we agree that the last_modified field is problematic?

Only if you are moving “files” around.

I don’t mind if you add some logic that will choose the field name based on conditions. Just keep in mind that wal-g shows time in a very strange format.

[AljoschaP]

Because we simply used direct cloning without copying files before, I will no longer work on it but I think it’s a problem that should definitely be addressed independent of the cloud provider. E.g. AWS has the same behavior and I think a lot of people can get into trouble because of this.