question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unable to mount volume after node reboot

See original GitHub issue

I recently had to shutdown the entire k8s cluster. Prior to shutting it down, all pods/volumes etc were working correctly. However following the reboot, one of the deployments fails to start as it is unable to mount the pvc volume.

The error message is MountVolume.MountDevice failed for volume "pvc-3f761acf-262b-40ba-8b06-f76f205c70a4" : rpc error: code = Internal desc = {"code":1,"stdout":"Couldn't find valid filesystem superblock.\n","stderr":"resize2fs 1.44.5 (15-Dec-2018)\nresize2fs: Superblock checksum does not match superblock while trying to open /dev/sdb\n"}

As I wrote, all I did was take the cluster down, and then restart it. All other deployments have come back as expected. Including others using PVCs on the same NAS unit.

There are no obvious errors on the NAS unit.

Do you have any idea of the cause of this, and more importantly, given that I cannot start the deployment, how can I repair this?

Thanks

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:25 (13 by maintainers)

github_iconTop GitHub Comments

1reaction
chrisbsmithcommented, Jun 8, 2022

Got it! That did it. Thank you for taking the time to review and provide some help! Appreciate it.

1reaction
isugimpycommented, Nov 27, 2021

Brilliant! Thank you!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Some of the services or volumes are not mounting after a reboot
To verify that the Gluster volumes are mounted by Linux following a reboot, run this command on all cluster nodes:.
Read more >
Pods unable to mount persistent volumes on some nodes
1 Answer ... Hello @FayaazAhmed-5969 , This happens sometimes due to container runtime not properly cleaning up the resources which are not needed ......
Read more >
Troubleshooting Kubernetes FailedAttachVolume and ...
There may be situations when Kubernetes can detach the volume but is unable to attach or mount the disk in the scheduled node....
Read more >
Unable to mount volumes for pod because "volume is already ...
A common Kubernetes error is being unable to mount volumes for pod because "volume is already exclusively attached to one node and can't...
Read more >
Persistent volumes cannot attach to a new node if previous ...
... in a hung state or failed after a long wait; Your pods that are utilizing persistent volumes are not able to attach...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found