Support execution of compiled binaries

I really want to be able to support execution, and have been working on it. I hit a flaw, and ended up writing this essay to cover options. I put it here for reference, review and comment.

Execution

Goals

• Be able to safely execute user-supplied programs.
• (optionally) Execute the compilers in a more safe manner.

Threat model

1. Curious internet folk seeing if they can hack the site/read files they shouldn’t (e.g. /etc/passwd and the like).
2. Malicious users attempting to DoS Compiler Explorer; either taking it down or attempting to delete shared resources (e.g. /opt/compiler-explorer compilers).
3. Malicious users seeking to gain control of the site to mine bitcoin/send spam/etc.
4. As above but to attempt to steal other users’ data.
5. As above but to attempt to access AWS to spin up further instances/access other services.

Considerations

The current setup runs each AWS instance with limited privileges. Each AWS instance runs multiple docker containers (one for each sub-site). The compilers are run inside these docker images, and each is run with an LD_PRELOAD wrapper that attempts to minimise the number of files the compilers can access. Additionally some command-line flags are banned (e.g. -fplugin type functionality).

The docker containers isolate both the node.js server and the compilers it runs from the host AWS environment: most of the security comes from this. Over the years the LD_PRELOAD has become less effective: most compilers now need to look at most files in /etc and /proc etc, and so a lot of files one might otherwise want to restrict need to be whitelisted. Additionally, some compilers are statically linked and so cannot for LD_PRELOADed in this way.

A full breach via the compiler would expose just the one docker instance, and modifications would be ephemeral. If the site was taken down, AWS healthchecking would kill the instance and spawn a new one (with a fresh clean docker image), so no lasting damage would be done.

However, there’s a weakness even here: the /opt drive is a read-write mounted EFS drive (Amazon’s network file system). Changes made here would be lasting, and would be seen by the other running nodes immediately.

The /opt drive is mostly a convenience: it stores all the compilers so that each AWS node doesn’t need >20GB of compilers in its image. This allows new AWS nodes to boot quickly when load requires, and means building a new Compiler Explorer image is also pretty quick.

All data in /opt can be recovered from the S3 source.

Execution of Windows compilers is achieved via wine. This can have a long start-up time, and requires a daemon process (wineserver). In order to minimise startup time, in the gcc.godbolt.org docker image the wineserver is run as a long-lived background process during boot. Calls to wine then execute quickly by attaching to this wineserver instance.

Options

All these options rely on a restricted docker container to run the user executable in. The image has very little in it; just enough to run the user program. It runs as an unprivileged user and with (if possible) user namespaces to prevent root in that container being root on the host. It has no setuid programs installed, and will be run without network bridges, and with restricted cgroups.

Docker-in-docker

Compiler Explorer continues to run in docker, and uses a docker-in-docker approach. This is/was the current plan until issues were found.

By mounting /var/run/docker.sock from the host into the CE docker image, that docker image (with some userid/groupid gymnastics) can launch other docker images. Thus it can in principal launch the execution docker images.

The following issues have been found:

• It is not possible for the execution image (EI) to mount directories from the compiler image (CI). The non-docker implementation of the CI compiles to a random directory in /tmp, and then runs the EI with that temporary directory mounted as /home/ce-user. This doesn’t work under docker-in-docker. Solutions could include mounting a well-known host directory as the /tmp in the CI, and then having knowledge of how to specify the real host path to this directory as the mount target in the EI. Care must be taken to only mount the one /tmp directory with the user’s code in it in the EI, not the whole /tmp, else information leakage between executions is possible.
• Even when running as a restricted user, the CI must be able to talk to the docker socket. As the docker daemon runs as root on the host, this effectively gives the CI effective host root access, weakening its security considerably. The increased attack vector means targeting the compilation part to gain root on the AWS instance would be feasible.

Pros:

• Pretty much written

Cons:

• Has the above flaws and issues

Drop the CI docker

Compiler Explorer would run on the AWS host directly. It would then use the same EI isolation techniques to run the compiler as it does to run user binaries. That way the fact it’s running directly on the AWS host is “hidden”. In the EI the /opt/compiler-explorer would probably have to be mounted (read-only), to gain access to libraries etc there (gcc’s libs for example). If the compiler is run in the EI it’ll need a “non-free” mount too to cover the licenses etc.

Pros:

• Solves compilation security holes in a similar way to execution does.

Cons:

• Running CE on bare AWS host opens door to more security issues there. (mitigated by compilers being run in safe way).

Remote execution

CE/compilation as before, but executable is transmitted to a separate app to be run. That app would run on AWS directly, but would then use an EI as described above.

Pros:

• Most isolation from rest of activities of all techniques
• Could run on separate AWS nodes with even fewer permissions
• Could be scaled independently of the rest of the Compiler Explorer infrastructure

Cons:

• A new server type to administrate
• Handling serialisation of the executable and its dependencies could be tricky
• Not sure how it fits into the “compilation environment” ideas mooted with Christopher Di Bella. Though it might form the foundation of the “make an environment” process too.

Other notes

• Split /opt/compiler-explorer into sensitive and non-sensitive areas so the licenses etc can be specifically unmapped.
• wine startup time is common to all issues: we either have to accept it or have some kind of persistent wineserver that can potentially leak information. Maybe wineserver can run in its own container and somehow be shared with all wine processes? Looks like wine talks via a UDS in /tmp/.wine-$UID. It might also be the case that the pause is long only due to the particular wine setup used: a container with a “pre-warmed” wine setup might be fast enough.

[mattgodbolt]

Ok! This is about to go live 😃 Thanks everyone for their invaluable help getting this done!

[mattgodbolt]

Thanks for the link. I think probably given I’m already running things locally on my instances, and have already compiled them, and have the libraries etc locally, it would be easiest to run them locally using either firejail or isolate (like you do). Thanks for the links and the kind offer of a remote service though!