[Bug] Snapshot removing failed. Snapshot attributes are not valid
See original GitHub issueCannot delete snapshots from the remote registry at docker hub. Pushes and pulls to registry just fine, however it fails to delete old snapshots from registry resulting in ALOT of orphan snapshots in registry. I have a multi-machine workspace consisting of 5 machines and it left around 40 orphan snapshots over time.
Reproduction Steps:
- configure che to work with a docker hub organization repository for saving snapshots
- start a multi-machine workspace
- stop the workspace, verify that it saves to registry
- start the workspace and load from snapshot
- stop the workspace without saving snapshot
OS and version:
- Che running on AWS EC2 t2.medium.
- Ubuntu 16.04
- Eclipse Che 5.4
Diagnostics: Log:
2017-03-21 18:16:26,551[aceSharedPool-3] [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529] - Couldn't remove snapshot 'snapshot7r82ivljtss3ls1n', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:525) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshotAndUpdateStatus(WorkspaceRuntimes.java:893) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshot(WorkspaceRuntimes.java:470) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceManager.lambda$stopAsync$3(WorkspaceManager.java:815) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:28) ~[che-core-commons-lang-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[na:1.8.0_111-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111-internal]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111-internal]
2017-03-21 18:16:26,551[aceSharedPool-3] [ERROR] [c.p.d.m.DockerInstanceProvider 148] - Failed to remove instance snapshot: invalid machine source: MachineSourceImpl{type='image', location='null', content='nu
ll'}
2017-03-21 18:16:26,552[aceSharedPool-3] [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529] - Couldn't remove snapshot 'snapshotyq6mieud1voj4cxf', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:525) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshotAndUpdateStatus(WorkspaceRuntimes.java:893) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshot(WorkspaceRuntimes.java:470) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceManager.lambda$stopAsync$3(WorkspaceManager.java:815) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:28) ~[che-core-commons-lang-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[na:1.8.0_111-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111-internal]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111-internal]
2017-03-21 18:16:26,553[aceSharedPool-3] [ERROR] [c.p.d.m.DockerInstanceProvider 148] - Failed to remove instance snapshot: invalid machine source: MachineSourceImpl{type='image', location='null', content='nu
ll'}
2017-03-21 18:16:26,553[aceSharedPool-3] [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 529] - Couldn't remove snapshot 'snapshot15c4j45me2k0lt2p', workspace id 'workspacem52qibob1zahcdiy'
org.eclipse.che.api.machine.server.exception.SnapshotException: Snapshot removing failed. Snapshot attributes are not valid
at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:149) ~[che-plugin-docker-machine-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:557) ~[che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:508) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeBinaries(WorkspaceRuntimes.java:525) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshotAndUpdateStatus(WorkspaceRuntimes.java:893) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.snapshot(WorkspaceRuntimes.java:470) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.api.workspace.server.WorkspaceManager.lambda$stopAsync$3(WorkspaceManager.java:815) [che-core-api-workspace-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:28) ~[che-core-commons-lang-5.4.0-SNAPSHOT.jar:5.4.0-SNAPSHOT]
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[na:1.8.0_111-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111-internal]
Issue Analytics
- State:
- Created 7 years ago
- Comments:23 (7 by maintainers)
Top Results From Across the Web
ModifyDBSnapshotAttribute says InvalidParameterValue on ...
An error occurred (InvalidParameterValue) when calling the ModifyDBSnapshotAttribute operation: The parameter awsbackup:job-xxxxxxxx-xxxx-xxxx- ...
Read more >App Service Snapshot Restore Issue, This should be fixed. I ...
The command failed with an unexpected error. Here is the traceback: 'WebAppsOperations' object has no attribute 'restore_snapshot'
Read more >Why I cannot delete snapshot with acronym - Forums - IBM
The problem is obvious that the user is not using the correct acronym of the snapshot. Please navigate to the snapshot anddetermine the...
Read more >Problem with Snapshot - NetWorker - Dell
Snapshot box checked, as this is a Snapshot based backup. ... (pid5420) Snap Provider Error : No source attributes supported for this vendor....
Read more >modify-snapshot-attribute — AWS CLI 1.27.37 Command ...
Adds or removes permission settings for the specified snapshot. ... If you have the required permissions, the error response is DryRunOperation .
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Done. Fixed bug about invalid snapshot attributes. Added automate deletion of snapshots stored in Docker Hub.
@rhughes89 thank you for the suggestion! But main goal here is keeping registry clear. When you push new image over old one, you just overwrite manifests, but blobs is still there and it is the problem. You don’t see it, but it still uses your disk space. So it solves problem of a user, but not admin. In case of CHE, often user is admin too. Of course, there are clean up scripts, but they bring some other troubles…