Running out of memory?

Describe the bug Almost every day I’m seeing a “cannot allocate memory” error, followed by Frigate stopping working (still running, but nothing happens) until I restart the container.

Version of frigate 0.8.4-5043040

Config file Include your full config file wrapped in triple back ticks.

detectors:
  cpu1:
    type: cpu
  cpu2:
    type: cpu
  # coral:
  #   type: edgetpu
  #   device: usb
mqtt:
  host: 192.168.1.253
cameras:
  front_porch:
    snapshots:
      enabled: True
      bounding_box: True
      retain:
        default: 5
    clips:
      enabled: True
      objects:
        - person
        - cat
    record:
      enabled: False
    rtmp:
      enabled: True
    ffmpeg:
      inputs:
        - path: rtsp://front:front@192.168.1.202/live
          roles:
            - detect
            - rtmp
            - clips
            # - record
    width: 1920
    height: 1080
    fps: 5
  back_porch:
    snapshots:
      enabled: True
      bounding_box: True
      retain:
        default: 5
    clips:
      enabled: True
      objects:
        - person
        - cat
    record:
      enabled: False
    rtmp:
      enabled: True
    ffmpeg:
      inputs:
        - path: rtsp://back:back@192.168.1.203/live
          roles:
            - detect
            - rtmp
            - clips
            # - record
    width: 1920
    height: 1080
    fps: 5
objects:
  track:
    - person
    # - car
    # - truck
    - cat
    # - dog
    # - bicycle
  # filters:
  # person:
  # threshold: 0.75
# detect:
#   max_disappeared: 25
clips:
  retain:
    default: 5
ffmpeg:
  output_args:
    clips: "-f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac"
    rtmp: "-c copy -an -f flv"
  # record: "-f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac"
# logger:
#   default: info
#   logs:
#     frigate.mqtt: error

Frigate container logs

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/opt/frigate/frigate/video.py", line 185, in run
    self.start_ffmpeg_detect()
  File "/opt/frigate/frigate/video.py", line 209, in start_ffmpeg_detect
    self.ffmpeg_detect_process = start_or_restart_ffmpeg(ffmpeg_cmd, self.logger, self.logpipe, self.frame_size)
  File "/opt/frigate/frigate/video.py", line 98, in start_or_restart_ffmpeg
    process = sp.Popen(ffmpeg_cmd, stdout = sp.PIPE, stderr=logpipe, stdin = sp.DEVNULL, bufsize=frame_size*10, start_new_session=True)
  File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1637, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory

Frigate stats

{
  "back_porch": {
    "camera_fps": 5.0, 
    "capture_pid": 46, 
    "detection_fps": 0.0, 
    "pid": 44, 
    "process_fps": 5.0, 
    "skipped_fps": 0.0
  }, 
  "detection_fps": 0.0, 
  "detectors": {
    "cpu1": {
      "detection_start": 0.0, 
      "inference_speed": 94.97, 
      "pid": 38
    }, 
    "cpu2": {
      "detection_start": 0.0, 
      "inference_speed": 96.29, 
      "pid": 40
    }
  }, 
  "front_porch": {
    "camera_fps": 0.0, 
    "capture_pid": 45, 
    "detection_fps": 0.0, 
    "pid": 43, 
    "process_fps": 0.0, 
    "skipped_fps": 0.0
  }, 
  "service": {
    "storage": {
      "/dev/shm": {
        "free": 63.3, 
        "mount_type": "tmpfs", 
        "total": 67.1, 
        "used": 3.8
      }, 
      "/media/frigate/clips": {
        "free": 14252.8, 
        "mount_type": "ext4", 
        "total": 30509.4, 
        "used": 16202.2
      }, 
      "/media/frigate/recordings": {
        "free": 14252.8, 
        "mount_type": "ext4", 
        "total": 30509.4, 
        "used": 16202.2
      }, 
      "/tmp/cache": {
        "free": 14252.8, 
        "mount_type": "overlay", 
        "total": 30509.4, 
        "used": 16202.2
      }
    }, 
    "uptime": 145, 
    "version": "0.8.4-5043040"
  }
}

FFprobe from your camera

root@ERROL:~# ffprobe rtsp://front:front@192.168.1.202/live
pts item:[000000:00000001:00000001]
ffprobe version 0.8, Copyright (c) 2007-2011 the FFmpeg developers
  built on May 14 2020 05:54:30 with gcc 6.3.1 20170109
  configuration: --prefix=/srv/output/amffmpeg/host/aarch64-linux-gnu/sysroot/us                                                                                                                                                             r --shlibdir=/srv/output/amffmpeg/host/aarch64-linux-gnu/sysroot/usr/lib/libplay                                                                                                                                                             er --cross-prefix=/srv/output/amffmpeg/host/bin/aarch64-linux-gnu- --arch=aarch6                                                                                                                                                             4 --extra-ldflags='-L/srv/output/amffmpeg/host/aarch64-linux-gnu/sysroot/usr/lib                                                                                                                                                             / -L/srv/output/amffmpeg/host/aarch64-linux-gnu/sysroot/usr/lib/libplayer -lamav                                                                                                                                                             utils -ldl' --disable-yasm --enable-debug --disable-ffplay --disable-ffmpeg --en                                                                                                                                                             able-cross-compile --target-os=linux --disable-librtmp --disable-static --enable                                                                                                                                                             -shared --disable-ffserver --disable-doc --cpu=cortex-a53
  libavutil    51.  9. 1 / 51.  9. 1
  libavcodec   53.  7. 0 / 53.  7. 0
  libavformat  53.  4. 0 / 53.  4. 0
  libavdevice  53.  1. 1 / 53.  1. 1
  libavfilter   2. 23. 0 /  2. 23. 0
  libswscale    2.  0. 0 /  2.  0. 0
[init-input] iformat namertsp return
[rtsp_read_header:363]
[tcp @ 0x20e4e910] tcp will get address from dns!
[tcp @ 0x20e4e910] resolved 192.168.1.202's  ipaddress
Segmentation fault

Computer Hardware

• OS: Ubuntu 20.04.2 LTS
• Running on ODROID N2
• Install method: Docker
• Coral Version: Seems to happen more often with Coral USB, but also with CPU
• Network Setup: Wired

Camera Info:

• Manufacturer: Wyze
• Model: Pan Cam
• Resolution: 720p
• FPS: 5

[blakeblackshear]

Next time it happens, stop the frigate container and see if your memory usage is still so high. It may not be a frigate issue if memory usage is that high immediately after a restart of the container. It looks like home assistant is using almost all of the system memory.

[blakeblackshear]

Definitely seems to be growing. Let’s check once more in a few hours again.