Support for Cassandra in Kubernetes

Greetings,

This looks like a great and much-needed project.

Saw the comment from this TLP blog post regarding compatibility with Cassandra running in Docker containers (e.g. a Kubernetes stateful set using persistent volumes for storage), and want to confirm that you do not recommend testing cassandra-medusa in Kubernetes at this time.

[jsanda]

Hey @bwmills

I have spent a little bit of time thinking about this. When you do a cluster-wide restore,

$ medusa restore-cluster

medusa takes on the role of a control node or orchestrator. It does ssh into each machine and then run medusa restore-node (if I remember correctly). The restore-node command roughly does the following:

• download the backup
• stop Cassandra
• restore the backup
• start Cassandra

I would be very inclined to write a k8s operator for medusa. I would have a couple (maybe more) CRDs, something like MedusaBackupJob and MedusaRestoreJob. The CRDs would allow you to configure things like:

• access to the remote storage
• schedule for reoccurring backups
• the target cluster to backup
• the destination cluster to restore

I would not expose or use the restore-cluster command. The operator would facilitate the role of control node. Suppose we have a three-node cluster - C1, C2, C3 - that we want to restore. We create a MedusaRestoreJob object with the necessary configuration. The operator in turn creates a job that performs a restore on C1. After that job completes the operator creates a job to restore C2 and then C3. I am glossing over a lot of details, but I just want to provide a high level picture at the moment. These k8s jobs would run medusa restore-node, not restore-cluster so I don’t think SSH would be a big concern. That is not to suggest that other changes are not necessary to make medusa work in k8s.

Dealing with the statefulsets, PVs, and PVCs gets interesting. The restore process involves stopping and starting the Cassandra nodes. To shut down C1 we simply kill the pod in which it is running; however, the StatefulSet controller will immediately recreate the pod. We will have to deal with this race condition. I did some research on this and I would delete the StatefulSet(s) without cascading the deletes so that all pods remain intact. The medusa operator is then free to kill a pod, perform whatever work is necessary, recreate the pod, and then recreate the StatefulSet object. The StatefulSet controller will discover the existing pods so the StatefulSet will return to the ready state very quickly.

Now onto the PVs and PVCs. When I was brainstorming this, I first though about restoring a single C* node and I assumed ReadWriteOnce access mode for the PV. When the pod is shutdown, we have to delete the existing PVC for the data volume. A new PVC is created for the MedusaRestoreJob so that the pod in which medusa is running can mount the data volume. We might also need another PV on which to download the backup. After the restore job finishes, we recreate the PVC for the C1 pod and then recreate the pod. Then we recreate the StatefulSet object.

I did a little bit of prototyping to check that these steps work, but I still need to spend time reviewing the StatefulSet controller code to make sure I understand the behavior with this sort of workflow.

I think the steps for a cluster-wide restore might actually be a bit less convoluted. We could potentially create new PVs and a new statefulset. We could then use label selectors to bind the new PVs to the new pods in the statefulset. We would have to create new PVCs. PVCs are immutable which means that we could not update the existing ones to point to the new PVs. We would then delete the existing statefulset and create a new one that uses the restore PVs.

The operator needs to know about the cluster topology as well. For instance, is the entire C* cluster running as a single statefulset? Or is there is a statefulset per rack? Or per DC? That’s a whole other discussion 😃

[jsanda]

I want to provide a quick status update on what I have been doing and thinking about.

I have started working on creating a gRPC service for Medusa. This is being tracked under #137. The goal is to have the gRPC service running in a sidecar container to handle backup requests.

I think we will need a different Dockerfile image than what is currently in the project for the sidecar.

Those requests will come from a k8s controller for Medusa. I will probably work on that in a separate repo as it will be Go code and more k8s-specific.

We could create two CRDs - Backup and BackupJob. The former would specify that a backup should be created. It could use label selectors for instance to determine which Cassandra pods should be backed up. The BackupJob would CRD would facilitate scheduled backups.

For restores, we can use an init container in addition to the sidecar container. Init containers run to completion before the main containers in the pod start. To perform a restore then, we can delete the Cassandra pod (or just the container I suppose). The StatefulSet controller will create a new pod. Before starting the Cassandra container, the Medusa init container will run.

Things might be a bit different when integrating with a k8s operator for Cassandra. This approach though provides a path forward without requiring a Cassandra operator.