Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Support]: CPU Pegged at 100% by Frigate with TPU Installed

See original GitHub issue

Describe the problem you are having

I am running 8 cameras on Frigate on a computer with a coral module installed and I’m seeing my CPU pegged at 100% all the time. I haven’t checked in a while, but I don’t remember ever having the CPU usage so high when running frigate.

Here is a screenshot from HTOP running on the host

Screenshot 2022-07-07 10 22 53 AM

Version

0.10.1-83481AF

Frigate config file

database:
  path: /config/frigate.db

mqtt:
  host: mqtt_host.local

detectors:
  coral:
    type: edgetpu
    device: usb

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p

snapshots:
  enabled: yes

record:
  enabled: True
  retain:
    days: 2
  events:
    pre_capture: 60
    post_capture: 60
    retain:
      default: 5

cameras:
  woodshop:
    ffmpeg:
      inputs:
        - path: rtsp://1080_camera_url/s0
          roles:
            - detect
            - rtmp
    detect:
      width: 1920
      height: 1080
    record:
      events:
        objects:
          - person

    objects:
      track:
        - person

  right_backyard:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

  covered_patio:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

  2car_garage_left:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

  driveway_right:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

    motion:
      mask:
        - 2318,527,2278,407,1193,558,71,892,80,1059,548,2160,687,2160,979,1565,1525,1167,2193,1358,2664,2160,0,2160,0,1122,0,0,3840,0,3840,2160,2739,2160,2913,1838,3146,480
    objects:
      track:
        - person
        - car
      filters:
        person:
          mask:
            - 1002,0,3139,0,3151,525,2407,609,1096,904,527,1073,532,1779,539,2160,0,2160,0,0
        car:
          mask:
            - 2407,602,1104,894,558,1061,0,1249,0,0,3840,0,3840,2160,2739,2160,2913,1838,3078,894,3113,525

  main_door:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

  front_yard_left:
    ffmpeg:
      inputs:
        - path: rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64
          roles:
            # - detect
            - rtmp
    detect:
      width: 3840
      height: 2160

  doorbell:
    ffmpeg:
      inputs:
        - path: rtsp://1080_camera_url:554/cam/realmonitor?channel=1&subtype=0
          roles:
            # - detect
            - rtmp
    detect:
      width: 1920
      height: 1080

Relevant log output

[services.d] starting services
[services.d] done.
[2022-07-07 10:24:22] frigate.app                    INFO    : Starting Frigate (0.10.1-83481af)
Starting migrations
[2022-07-07 10:24:22] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-07-07 10:24:22] peewee_migrate                 INFO    : There is nothing to migrate
[2022-07-07 10:24:22] detector.coral                 INFO    : Starting detection process: 216
[2022-07-07 10:24:22] frigate.app                    INFO    : Output process started: 218
[2022-07-07 10:24:22] ws4py                          INFO    : Using epoll
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for woodshop: 225
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for right_backyard: 227
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for covered_patio: 229
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for 2car_garage_left: 230
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for driveway_right: 232
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for main_door: 234
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for front_yard_left: 235
[2022-07-07 10:24:22] frigate.app                    INFO    : Camera processor started for doorbell: 236
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for woodshop: 238
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for right_backyard: 242
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for covered_patio: 246
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for 2car_garage_left: 249
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for driveway_right: 251
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for main_door: 254
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for front_yard_left: 260
[2022-07-07 10:24:22] frigate.app                    INFO    : Capture process started for doorbell: 264
[2022-07-07 10:24:23] ws4py                          INFO    : Using epoll
[2022-07-07 10:24:22] frigate.edgetpu                INFO    : Attempting to load TPU as usb
[2022-07-07 10:24:25] frigate.edgetpu                INFO    : TPU found

FFprobe output from your camera

Most Cameras:
root@cf50164e5945:/opt/frigate# ffprobe "rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64"
ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
[rtsp @ 0x55b1019a0b80] max delay reached. need to consume packet
[rtsp @ 0x55b1019a0b80] RTP: missed 130 packets
[h264 @ 0x55b1019a67c0] error while decoding MB 85 108, bytestream -5
[h264 @ 0x55b1019a67c0] concealing 6684 DC, 6684 AC, 6684 MV errors in P frame
[rtsp @ 0x55b1019a0b80] max delay reached. need to consume packet
[rtsp @ 0x55b1019a0b80] RTP: missed 109 packets
[h264 @ 0x55b1019a67c0] error while decoding MB 195 35, bytestream -13
[h264 @ 0x55b1019a67c0] concealing 24094 DC, 24094 AC, 24094 MV errors in P frame
[rtsp @ 0x55b1019a0b80] max delay reached. need to consume packet
[rtsp @ 0x55b1019a0b80] RTP: missed 67 packets
[rtsp @ 0x55b1019a0b80] max delay reached. need to consume packet
[rtsp @ 0x55b1019a0b80] RTP: missed 1010 packets
Input #0, rtsp, from 'rtsp://4k_camera_url:554/cam/realmonitor?channel=1&subtype=0&authbasic=64':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.200000, bitrate: N/A
    Stream #0:0: Video: h264, yuv420p(progressive), 3840x2160, 100 tbr, 90k tbn, 180k tbc
root@cf50164e5945:/opt/frigate# ^C

Different Camera Model:

root@cf50164e5945:/opt/frigate# ffprobe "rtsp://camera_url/cam/realmonitor?channel=1&subtype=0"
ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, rtsp, from 'rtsp://camera_url/cam/realmonitor?channel=1&subtype=0':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.040000, bitrate: N/A
    Stream #0:0: Video: h264, yuvj420p(pc, bt709, progressive), 1920x1080, 100 tbr, 90k tbn, 180k tbc
    Stream #0:1: Audio: aac, 16000 Hz, mono, fltp

Frigate stats

{
  "2car_garage_left": {
    "camera_fps": 5.1,
    "capture_pid": 249,
    "detection_fps": 3.7,
    "pid": 230,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "covered_patio": {
    "camera_fps": 5.1,
    "capture_pid": 246,
    "detection_fps": 4.6,
    "pid": 229,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "detection_fps": 30.6,
  "detectors": {
    "coral": {
      "detection_start": 1657207629.796821,
      "inference_speed": 9,
      "pid": 216
    }
  },
  "doorbell": {
    "camera_fps": 5.1,
    "capture_pid": 264,
    "detection_fps": 0,
    "pid": 236,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "driveway_right": {
    "camera_fps": 5.1,
    "capture_pid": 251,
    "detection_fps": 7.7,
    "pid": 232,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "front_yard_left": {
    "camera_fps": 5.1,
    "capture_pid": 260,
    "detection_fps": 2.9,
    "pid": 235,
    "process_fps": 5.1,
    "skipped_fps": 0
  },
  "main_door": {
    "camera_fps": 4.9,
    "capture_pid": 254,
    "detection_fps": 0,
    "pid": 234,
    "process_fps": 4.9,
    "skipped_fps": 0
  },
  "right_backyard": {
    "camera_fps": 5,
    "capture_pid": 242,
    "detection_fps": 11.7,
    "pid": 227,
    "process_fps": 5,
    "skipped_fps": 0
  },
  "service": {
    "storage": {
      "/dev/shm": {
        "free": 976.8,
        "mount_type": "tmpfs",
        "total": 1073.7,
        "used": 96.9
      },
      "/media/frigate/clips": {
        "free": 3053872.6,
        "mount_type": "zfs",
        "total": 3964644.2,
        "used": 910771.6
      },
      "/media/frigate/recordings": {
        "free": 3053872.6,
        "mount_type": "zfs",
        "total": 3964644.2,
        "used": 910771.6
      },
      "/tmp/cache": {
        "free": 1764.9,
        "mount_type": "tmpfs",
        "total": 2000,
        "used": 235.1
      }
    },
    "temperatures": {},
    "uptime": 167,
    "version": "0.10.1-83481af"
  },
  "woodshop": {
    "camera_fps": 5,
    "capture_pid": 238,
    "detection_fps": 0,
    "pid": 225,
    "process_fps": 5,
    "skipped_fps": 0
  }
}

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

All Amcrest

Any other information that may be helpful

Docker container config (using Ansible)

 docker_container:
    name: frigate
    image: blakeblackshear/frigate:stable-amd64
    state: started
    restart_policy: unless-stopped
    shm_size: 1024M
    mounts:
      - type: tmpfs
        target: /tmp/cache
        tmpfs_size: 2000000000

    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ~/frigate/config.yml:/config/config.yml:ro
      - ~/frigate/frigate.db:/config/frigate.db
      - ~/frigate:/media/frigate
    devices:
      - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      - /dev/dri/renderD128:/dev/dri/renderD128

    ports:
      - "1935:1935" # RTMP feeds
    env:
      FRIGATE_RTSP_PASSWORD: "**********"

    labels:
      traefik.enable: "true"

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:13 (8 by maintainers)

github_iconTop GitHub Comments

2reactions
blakeblackshearcommented, Jul 8, 2022

I would suggest watching the debug view of each camera with the motion boxes overlays and seeing what is triggering motion and object detection. Your best bet is making sure you use motion masks to avoid running unnecessary detections from things like camera timestamps.

1reaction
blakeblackshearcommented, Jul 8, 2022

The best way to understand what is driving CPU usage is to turn off features to see what is driving it up. You can disable detect for each camera to get a sense of the baseline CPU to just decode the stream and look for motion.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Very high CPU usage in Frigate : r/homeassistant - Reddit
Hi All I'm running a frigate 0.8.4-5043040 with 8 cameras, ... The issue I have is I am seeing tremendous CPU usage on...
Read more >
Recommended hardware - Frigate Docs
Anything with an Intel CPU and capable of running Debian should work fine. As a bonus, you may want to look for devices...
Read more >
Frigate w/TPU, high cpu usage - Third party integrations
Using Frigate with a pcie Coral. I had it doing 4 cameras but cpu was pegged. Just doing 1 1080p camera it sits...
Read more >
Frigate NVR with Odyssey Blue and Coral USB Accelerator
Frigate NVR with Odyssey Blue and Coral USB Accelerator Description Today we wanna share a super cool open-source project to help you build...
Read more >
[SUPPORT] blakeblackshear - Frigate - Page 16
Does it cause high cpu usage? I'm planning to eventually buy a Google Coral USB TPU but was hoping to wait for prices...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

[Camera Support]:

Next page

Allow audio on JSMPEG stream and add a `camera.audio` flag