Snapshots deleted out of order

Originally reported in the forums here: https://community.home-assistant.io/t/add-on-home-assistant-google-drive-backup/107928/511

Full text of the post is below: TL:DR - seems there is a conflict when the system initiates snapshots - caused the latest, manual, full snapshot to be purged.

Updating my system and took a full snapshot, then updated an Installed add-on (via supervisor) which triggered a partial snapshot of that component (I forgot to switch it off).

Then tried to take another full snapshot from your WebUI. This indicates it is a partial snapshot not a full snapshot though the logs seem to indicate a full snapshot (still in progress).

07-28 16:39:16 INFO [backup.ha.hasource] Requesting a new snapshot
07-28 16:39:16 DEBUG [backup.ha.harequests] Making Hassio request: http://hassio/snapshots/new/full
07-28 16:39:51 DEBUG [backup.ha.harequests] Making Hassio request: http://hassio/addons/self/info
07-28 16:39:51 DEBUG [backup.ha.harequests] Making Hassio request: http://hassio/info
07-28 16:39:51 DEBUG [backup.ha.harequests] Making Hassio request: http://hassio/core/info
07-28 16:39:51 DEBUG [backup.ha.harequests] Making Hassio request: http://hassio/supervisor/info

Once finished, it said ‘full’

However, the last full snapshot I triggered manually, seems to have been deleted! (this is the most worrying one!)

I did see it in the log file, but the log of that has gone as I have thousands of these messages

07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory modified event
07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory changed
07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory modified event
07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory changed
07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory modified event
07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory changed
07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory modified event
07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory changed
07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory modified event
07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory changed
07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory modified event
07-28 16:48:44 DEBUG [backup.worker.watcher] Backup directory changed

Great add-on though :slight_smile:

[edit] After uploading the new full snapshot, it deleted the partial snapshot!

07-28 16:56:28 DEBUG [backup.drive.drivesource] Uploading Full HASnapshot 2021-07-28 16:39:16 100.00%
07-28 16:56:28 DEBUG [backup.drive.driverequests] Making Google Drive request: https://www.googleapis.com/drive/v3/files/xxxxxxxxxxxxxxxxxxxxxx/?fields=id%2Cname%2CappProperties%2Csize%2Ctrashed%2CmimeType%2CmodifiedTime%2Ccapabilities%2Cparents%2CdriveId&supportsAllDrives=true
07-28 16:56:29 INFO [backup.drive.drivesource] Deleting 'addon_7ad98f9c_zigbee2mqtt_1.15.0' From Google Drive
07-28 16:56:29 DEBUG [backup.drive.driverequests] Making Google Drive request: https://www.googleapis.com/drive/v3/files/xxxxxxxxxxxxxxxxxxxxxxx/?supportsAllDrives=true
07-28 16:56:30 INFO [backup.worker.watcher] Backup directory changed
07-28 16:56:30 DEBUG [backup.model.syncer] Sync requested by Backup Directory Watcher
07-28 16:56:30 INFO [backup.model.coordinator] Syncing Snapshots
07-28 16:56:30 DEBUG [backup.ha.harequests] Making Hassio request: http://hassio/snapshots

[sabeechen]

Ah, the problem is generational snapshots. I should have noticed that from the screenshots. It might come as a surprise that its actually working as designed, though obviously not as expected.

Generational snapshots will keep at most 1 snapshot per day, and the one it keeps is the latest regardless of whats in it. So I think the order of events are:

1. You made a full snapshot.
2. HA made a newer partial snapshot.
3. The addon thinks that partial snapshot is the correct one for that “day” so it deletes the older, full snapshot.
4. You manually created another full snapshot named “Full HASnapshot 2021-07-28 16:39:16”
5. The addons thinks that must be the newest snapshot for the day and deletes the partial snapshot.

There is something you can do to avoid this in the future. In the settings enable either:

• Ignore other snapshots, which will make it ignore any snapshot the addon didn’t make.
• Ignore upgrade snapshots, which will make the addon ignore any snapshot it didn’t make only if it contains a single folder or addon.

Additionally, when creating a snapshot manually you can select “Keep indefinitely in Home Assistant/Google Drive” which will cause it to be ignored from the deletion schedule. The addon will never delete it automatically.

It might seem silly that the addon picked the more recent partial snapshot to delete over the older full one. In general I agree, but my hands are kind of tied in terms of what to do. The addon has to promote one of the snapshots to be represented for the day. It could instead:

• Choose the oldest snapshot, but then any snapshot you create after the automatic one for the day would get deleted immediately.
• Stop and throw an error if there are two snapshots for the same day, but then the user would have to intervene before the normal backup schedule could continue.
• Do something much more complicated and confusing to try to figure out which snapshot the user would “want” to keep for a day. I suspect this would confuse people more than the existing behavior.

Choosing the latest snapshot may not always be the thing thats wanted, but it at least will produce reliable behavior.

Those automatically created snapshots kind of mess things up for this addon. I wish there were a way to make them default-off. If I’m being honest, I kind of regret implementing generational snapshots too. You aren’t the first person who found it confusing and it has been difficult to keep its logic sane as the addon has grown in functionality.

I’m open to suggestions if you think there is some way to improve this experience.

[borpin]

• Do something much more complicated and confusing to try to figure out which snapshot the user would “want” to keep for a day. I suspect this would confuse people more than the existing behavior.

Thinking about this again, I think the answer might be to provide a distinction between the automatic/generational backups and other backups (which is effectively what the two options do of course). I suspect those options may have been added after I set it up 😃