[OCP4] Che deployment fails due to permission denied in Postgres

Describe the bug

I installed a basic OCP 4 cluster on AWS. The default aws-ebs storage is used. I tried to install Che from the OperatorHub marketplace and the install failed because Postgres entered a CrashLoopBackOff state.

The Postgres container’s logs show the following error:

johns-mbp-3:.odo johncollier$ oc logs postgres-cc6b567f-fc9hj
mkdir: cannot create directory '/var/lib/pgsql/data/userdata': Permission denied

Che version

• latest
• nightly
• other: please specify

Steps to reproduce

1. Deploy OpenShift 4 on AWS
2. Make sure aws-ebs is used for default strorage (should be for new installs on AWS)
3. Install the Che operator from OperatorHub
4. Create a CheCluster custom resource to install Che
5. The Che deployment will fail because Postgres will report the following error:

Expected behavior

Runtime

• kubernetes (include output of kubectl version)
• Openshift (include output of oc version)
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

johns-mbp-3:.odo johncollier$ oc version
oc v3.11.0+0cbc58b
kubernetes v1.11.0+d4cacc0
features: Basic-Auth

Server https://<url>:6443
kubernetes v1.14.0+573d946

Screenshots

Installation method

• chectl
• che-operator 7.0.0
• minishift-addon
• I don’t know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: please specify

Additional context

[tomgeorge]

Comparing Che installed in the che namespace to Che installed in the default namespace reveals that the securityContext gets set to {} in default, and is properly set in the che namespace:

  securityContext:
    seLinuxOptions:
      level: 's0:c22,c9'
    fsGroup: 1000480000

[tomgeorge]

As I’m investigating this more, it seems like default does not respect any of the security context constraints by default that are present in openshift… If I make a new user and give them create access to pods and deployments, and they run a pod, it will run as root/the default UID present in that images Dockerfile. When I run the same pod in another namespace it is runs in a security context. default seems to have annotations regarding security contexts, but does not respect them:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/sa.scc.mcs: s0:c6,c5
    openshift.io/sa.scc.supplemental-groups: 1000040000/10000
    openshift.io/sa.scc.uid-range: 1000040000/10000
  creationTimestamp: "2019-10-28T20:25:55Z"
  name: default
  resourceVersion: "7335"
  selfLink: /api/v1/namespaces/default
  uid: 2515d6e5-f9c1-11e9-9124-028754979780
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

apiVersion: project.openshift.io/v1
kind: Project
metadata:
  annotations:
    openshift.io/sa.scc.mcs: s0:c6,c5
    openshift.io/sa.scc.supplemental-groups: 1000040000/10000
    openshift.io/sa.scc.uid-range: 1000040000/10000
  creationTimestamp: "2019-10-28T20:25:55Z"
  name: default
  resourceVersion: "7335"
  selfLink: /apis/project.openshift.io/v1/projects/default
  uid: 2515d6e5-f9c1-11e9-9124-028754979780
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

I’m starting to think this is just a documentation issue that we will need to point out as @l0rd and @davidfestal have suggested. I reached out on the aos-devel slack channel but haven’t had a reply yet.