feat: have docker filesystem be read/writeable but limited in space

Obviously the overall disk space that an execution can consume has to be limited. Otherwise we would be open for easy attacks that basically just write as much data into the filesystem until the host filesystem is full.

Unfortunately, by the time we started MachineLabs our options with docker to limit the overall filesystem size of an container were rather limited.

So, what we currently do can be summed up as follows:

1. Technically there is no overall limit of disk space inside an execution. Each execution “sees” the entire available disk space of the host.

2. However, the filesystem of the container is readonly preventing the user from writing anything to disk.

3. As an exception to the second point, the filesystem can write into /run and /tmp. This is possible by mounting these directories as --tmpfs for wich we can specify constraints on the size.

Unfortunately this still prevents us from doing simply things such as installing further software with pip (unless the user figures out how to install it in either /run or /tmp which seems to be hard to achieve)

Today, the situation on the docker front looks a bit better actually. Docker allows a parameter such as '--storage-opt size=10G' to limit the overall disk space of a container.

However, if that works depends on the underlying storage driver that is used. I failed to get that working on both my local environment as well as on the staging system because the systems did not match the requirements. I think we should get that working with the overlay2 storage driver over an ext4 system but would be require to upgrade our docker version.

I don’t wanna go that rabbit whole right now so I thought I write down my findings.

[SamVerschueren]

I was able to mount a second volume, format it as xfs, change the docker directory to the second volume and this is what docker info returned.

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
...

As you can see, it’s overlay2 backed with xfs, exactly what we need! The only thing that I need to do now is figuring out what the pquota mount option is.

[SamVerschueren]

@cburgdorf These are the steps I followed on GCP. I believe it worked, but after creating an image with storage-opt, I’m not sure how I can verify that I limited my disk space.

Used Ubuntu 16.04 as OS

1. Create a VM on GCP with a boot drive and an extra drive.
2. SSH into the VM
3. Install Docker (duh) - https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-16-04 After installation, you will have Docker with the overlay2 storage driver backed with extfs. You can verify this with docker info. We need xfs so that’s where the second drive kicks in. Using a boot drive with xfs doesn’t seem to play nice.
4. Run sudo lsblk to list the disks that are attached to the instance. You should see the boot disk and the secondary disk (probably sdb).
5. Run sudo mkfs.xfs /dev/sdb to format sdb to an xfs file system.
6. Create a mount point for the new disk sudo mkdir -p /mnt/data
7. Mount the disk with the pquota option sudo mount -o discard,defaults,pquota /dev/sdb /mnt/data
8. Check that the disk is mounted with sudo lsblk
9. Add read and write permission for all users sudo chmod a+w /mnt/data
10. Edit /etc/default/docker and add a -g option DOCKER_OPTS="-dns 8.8.8.8 -dns 8.8.4.4 -g /mnt/data
11. Stop the docker service sudo service docker stop
12. Move the /var/lib/docker/ directory to the new partition sudo mv /var/lib/docker /mnt/data/docker
13. Make a symlink sudo ln -s /mnt/data/docker /var/lib/docker
14. Start docker again sudo service docker start
15. Run docker info and check you should see that it uses overlay2 storage driver with xfs.

If something is not clear, let me know and I’ll update the steps.