Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

podman client hangs when starting container

See original GitHub issue

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When trying to run a container (with possibly invalid configuration) via podman-py, causes the system service to hang. Killing the client call will still leave the service spinning.

❯ python -c 'import podman;podman.PodmanClient().containers.run(detach=True, image="docker.io/library/nginx", mounts=[{"target": "tests123", "source": "tests/", "mode": "bind"}], tty=True)'
^CTraceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers_run.py", line 66, in run
    container.start()
  File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers.py", line 350, in start
    response = self.client.post(
  File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/api/client.py", line 304, in post
    return self._request(
  File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/api/client.py", line 402, in _request
    self.request(
  File "/tmp/xyz/venv/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/tmp/xyz/venv/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/tmp/xyz/venv/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/tmp/xyz/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/tmp/xyz/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/tmp/xyz/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1345, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.9/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
KeyboardInterrupt
❯ podman run --mount=type=bind,source=tests/,destination=tests123 -d registry.fedoraproject.org/fedora
Error: invalid container path "tests123", must be an absolute path
❯ podman run --mount=type=bind,source=tests/,destination=/tests123 -t -d registry.fedoraproject.org/fedora
4b2829c8e699643859b55369d2d558198dc17f04d5d7c1597a4ac791258d62c6
❯ python -c 'import podman;podman.PodmanClient().containers.run(detach=True, image="docker.io/library/nginx", mounts=[{"target": "/tests123", "source": "tests/", "mode": "bind"}], tty=True)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers_run.py", line 66, in run
    container.start()
  File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/domain/containers.py", line 353, in start
    response.raise_for_status()
  File "/tmp/xyz/venv/lib/python3.9/site-packages/podman/api/client.py", line 65, in raise_for_status
    raise APIError(cause, response=self._response, explanation=message)
podman.errors.exceptions.APIError: 500 Server Error: Internal Server Error (invalid mount type for `/tests123`: OCI runtime error)

Steps to reproduce the issue:

mkdir xyz && cd xyz/
python -m venv venv; source venv/bin/activate
podman system service -t 0 &
pip install podman
podman pull registry.fedoraproject.org/fedora
python -c 'import podman;podman.PodmanClient().containers.run(detach=True, image="registry.fedoraproject.org/fedora", mounts=[{"target": "tests123", "source": "tests/", "mode": "bind"}], tty=True)'

Describe the results you received: the podman client will hang while the podman service spins and stresses out the system.

Describe the results you expected: Either incorrect configuration or a successful start + detach

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.16.4
Git Commit:   51b8ddbc22cf5b10dd76dd9243924aa66ad7db39
Built:        Sat May 22 16:22:34 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.20.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.0.28-1
    path: /usr/bin/conmon
    version: 'conmon version 2.0.28, commit: 6b18f7e0f2e4cd7f7b016b88141e82210d370008'
  cpus: 24
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: autumn
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.12.7-arch1-1
  linkmode: dynamic
  memFree: 57026449408
  memTotal: 67362938880
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 0.19.1-1
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1
      commit: 1535fedf0b83fb898d449f9680000f729ba719f5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.10-1
    version: |-
      slirp4netns version 1.1.10
      commit: baa2bc5ff12fe6db646c1f4f3f966526c0eba5a0
      libslirp: 4.5.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 1073737728
  swapTotal: 1073737728
  uptime: 10h 29m 51.81s (Approximately 0.42 days)
registries: {}
store:
  configFile: /home/jbpratt/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/jbpratt/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 5
  runRoot: /run/user/1000/containers
  volumePath: /home/jbpratt/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.2
  Built: 1621718554
  BuiltTime: Sat May 22 16:22:34 2021
  GitCommit: 51b8ddbc22cf5b10dd76dd9243924aa66ad7db39
  GoVersion: go1.16.4
  OsArch: linux/amd64
  Version: 3.1.2

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.): physical