Cloud custodian v0.9.18.0 is not able to find our policy file in the container

Describe the bug

My team has been using Cloud Custodian running in Semaphore CI for about 6 months without any issues to help cleanup our cloud resources. Recently, we noticed that the latest release of Cloud Custodian (v0.9.18.0) broke our CI jobs.

We run cloud custodian docker image in our CI/CD as follows:

docker run -it --user $(id -u):$(id -g) \
  -v $(pwd)/output:/home/custodian/output \
  -v $(pwd)/cleanup/cloud-custodian/policy.yml:/home/custodian/policy.yml \
  -e AWS_DEFAULT_REGION="us-west-2" \
  -e AWS_SECRET_ACCESS_KEY=$AWS_NUKE_KEY_SECRET \
  -e AWS_ACCESS_KEY_ID=$AWS_NUKE_KEY_ID \
  --env-file <(env | grep "^AWS\|^AZURE\|^GOOGLE") \
  --net=host \
  cloudcustodian/c7n run \
  -v -s /home/custodian/output \
  /home/custodian/policy.yml

We bind a volume mount to where our policy file is on the machine to the container at /home/custodian/policy.yml. This works on v0.9.17.0 fine but for v0.9.18.0 image we get this in the logs:

Unable to find image 'cloudcustodian/c7n:latest' locally
latest: Pulling from cloudcustodian/c7n
Status: Downloaded newer image for cloudcustodian/c7n:latest
2022-08-15 17:36:52,215: custodian.commands:ERROR policy file does not exist (/home/custodian/policy.yml)
2022-08-15 17:36:52,215: custodian.commands:ERROR Found 1 errors.  Exiting.

Our workaround is to go back to using 0.9.17.0 just doing docker run -it … cloudcustodian/c7n:0.9.17.0 run…, but we’d like to use the latest if possible. Let me know if there is anything else I can provide to help with this. Thank you!

What did you expect to happen?

I expected it to find the policy file which contains our cloud custodian cleanup policies.

Cloud Provider

Amazon Web Services (AWS)

Cloud Custodian version and dependency information

v0.9.18.0

Policy

N/A

Relevant log/traceback output

Unable to find image 'cloudcustodian/c7n:latest' locally
latest: Pulling from cloudcustodian/c7n
Status: Downloaded newer image for cloudcustodian/c7n:latest
2022-08-15 17:36:52,215: custodian.commands:ERROR policy file does not exist (/home/custodian/policy.yml)
2022-08-15 17:36:52,215: custodian.commands:ERROR Found 1 errors.  Exiting.

Extra information or context

N/A

[ajkerrigan]

Coming back to this, looks like it can be closed. But just to review what seems to have actually changed, here are the permissions on /home/custodian in 0.9.17.0:

$  stat /home/custodian
  File: /home/custodian
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: fd01h/64769d    Inode: 10872077    Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 1000/custodian)   Gid: ( 1000/custodian)

And here’s 0.9.18.0:

$ stat /home/custodian
  File: /home/custodian
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: fd01h/64769d    Inode: 10872077    Links: 2
Access: (0750/drwxr-x---)  Uid: ( 1000/custodian)   Gid: ( 1000/custodian)

So running as uid 1001 would have worked in 0.9.17.0 but not in 0.9.18.0. Running as uid 1000 (the custodian user in the container) or uid 0 (root) allows access to files under /home/custodian to continue working.

Let us know if there are any unresolved aspects of this issue. And thanks for opening it! it’ll be a helpful reference for anyone else in a similar situation.

[kapilt]

custodian doesn’t run as a root in the container, but it does run as a different non privileged user ‘custodian’ https://github.com/cloud-custodian/cloud-custodian/blob/master/docker/cli#L55

which is likely causing the permission issue your seeing wrt to accessing the policy file. since your manually mapping the runtime user to a different user then what the container expects.