Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Permission denied for nginx container after enable SELinux in docker daemon

See original GitHub issue

Description of the issue

Hi everyone! Thank you for making ERP next deployment become easier and less complicated. I tried to harden docker daemon using SELinux. Luckily, this happen to my testing server, I create /etc/docker/daemon.json then fill it with

{
        "selinux-enabled": true
}

FYI, I have enabled selinux before updating to latest version of ERP next, and it worked as it supposed to be. Then… everything has change after rebuilding container to latest version using docker-compose pull and docker-compose up -d, I got bad gateway when trying to access our sites.

OS: Rocky Linux 8.4 Docker: Docker version 20.10.10, build b485636 Compose: docker-compose version 1.29.2, build 5becea4c

Steps to reproduce the issue

  1. Create /etc/docker/daemon.json
  2. Fill it with
{
        "selinux-enabled": true
}
  1. systemctl restart docker
  2. restorecon -R -v /var/lib/docker
  3. restorecon -R -v /usr/bin
  4. cd /frappe-docker
  5. docker-compose pull
  6. docker-compose up -d
  7. Accessing out sites
  8. I got Bad Gateway in top left corner
  9. docker ps
  10. Everything is running normally except frappe/erpnext-nginx:version-13 *logs below

Observed result

Get Bad Gateway

Expected result

Can access our site normally

Stacktrace / full error message if available

rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/.package.json.UaHnNY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/.inquirer.js.tXEiEY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/objects/.choice.js.Akuf6X" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/objects/.choices.js.L0oHlX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/objects/.separator.js.usVRjY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.base.js.TjOl5V" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.checkbox.js.oXxEtX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.confirm.js.mAAO0V" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.editor.js.LkNTFZ" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.expand.js.ORZdPX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.input.js.gdysHY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.list.js.4gAO2V" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.number.js.cbaqFW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.password.js.EG0AoZ" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/prompts/.rawlist.js.GYgdWY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/ui/.baseUI.js.gKkcbX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/ui/.bottom-bar.js.5hUE6W" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/ui/.prompt.js.AkrNSY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.events.js.zceOyZ" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.incrementListIndex.js.WDbbgY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.paginator.js.fjdUMV" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.readline.js.1PBltX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.screen-manager.js.xYMupW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/lib/utils/.utils.js.cFH5lW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-escapes/.index.d.ts.2v4PlY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-escapes/.index.js.JNl6HX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-escapes/.license.RVvVJW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-escapes/.package.json.5jOkyY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-escapes/.readme.md.L9bF3X" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-regex/.index.d.ts.o7WnAV" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-regex/.index.js.yjSaRX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-regex/.license.KERgVW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-regex/.package.json.rxk7iX" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-regex/.readme.md.8jroHY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-styles/.index.d.ts.4D1biV" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-styles/.index.js.4GDc2V" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-styles/.license.Fnc18V" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-styles/.package.json.Hp6tKW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/ansi-styles/.readme.md.frsc0Y" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/.index.d.ts.wP2VXY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/.license.BW89hW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/.package.json.QaS65W" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/.readme.md.yJ8e7W" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/source/.index.js.0yUX9X" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/source/.templates.js.tEj4zY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/chalk/source/.util.js.8yDwKW" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/color-convert/.CHANGELOG.md.RME2iY" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/color-convert/.LICENSE.mxCv9W" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/color-convert/.README.md.RoZO3W" failed: Permission denied (13)
rsync: [receiver] mkstemp "/assets/frappe/node_modules/@snyk/inquirer/node_modules/color-convert/.conversions.js.1WXBzX" failed: Permission denied (13)

output of docker info

Server:
 Containers: 13
  Running: 12
  Paused: 0
  Stopped: 1
 Images: 38
 Server Version: 20.10.10
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  selinux
 Kernel Version: 4.18.0-305.19.1.el8_4.x86_64
 Operating System: Rocky Linux 8.4 (Green Obsidian)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.775GiB
 Name: 
 ID: 
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

I haven’t been able to find workaround, do you have any clue? Thanks in advance 🙂

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:10

github_iconTop GitHub Comments

1reaction
revantcommented, Nov 19, 2021

it comes from erpnext-nginx container.

https://github.com/frappe/frappe_docker/blob/abe6d670c4400f188972756da95208ab8ef6ea38/build/frappe-nginx/docker-entrypoint.sh#L7

can you try adding depends_on to erpnext-nginx service:

  erpnext-nginx:
    ...
    depends_on:
      - fix-vol-permissions
...

make sure you pull the images again after #572 is merged

1reaction
revantcommented, Nov 19, 2021

can you add 1 more container to you docker-compose that fixes the vol permission. I’m trying to get it running with podman and facing the issue. I’ll update here.

...
  fix-vol-permissions:
    image: frappe/frappe-worker:${ERPNEXT_VERSION}
    user: root
    command: chown -R 1000:1000 /sites /assets /logs
    volumes:
      - sites-vol:/sites
      - assets-vol:/assets
      - logs-vol:/logs
...

Note: I’ve not yet found a way to fix it.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Container permission denied: How to diagnose this error
Like SELinux, AppArmor could cause a permission-denied error. You can verify whether it is the problem by turning off AppArmor separation: $ ...
Read more >
How to fix docker: Got permission denied issue - Stack Overflow
If you want to run docker as non-root user then you need to add it to the docker group. Create the docker group...
Read more >
SELinux configuration troubleshooting for NGINX Controller
This issue shows as a “permission denied” message in the startup logs of the corresponding docker containers.
Read more >
SELinux cause "Permission denied" issue in using docker
I am using docker on RHEL 7. After mounting host directory into container, some interesting things happen: Although I am a root user, ......
Read more >
Using NGINX and NGINX Plus with SELinux
When Security-Enhanced Linux (SELinux) is enabled for Red Hat Enterprise Linux (RHEL) and related distros, its default settings prevent ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

[Deployment help] Installing ERPNext with frappe_docker

Next page

nginx assets location