UI running but cameras frozen and image detection halted

Describe the bug Lights are on but nobody home… For the last 2 days, my frigate cameras have been showing images from 2 days ago and no detections have happened in 2 days. These pictures were taken on March 29 at 3:00pm

Version of frigate 8.3 amd64

Config file Include your full config file wrapped in triple back ticks.

web_port: 4000

mqtt:
  host: REDACTED
  topic_prefix: REDACTED
  user: REDACTED
  password: REDACTED 



cameras:

####################### synology streams ######################
mqtt:
  host: REDACTED
  user: REDACTED
  password: '{FRIGATE_MQTT_PASSWORD}'
objects:
  track:
    - person
    - cat
    - dog
    - bird
  filters:
    person:
      min_area: 4000
      min_score: 0.5
      threshold: 0.72
    cat:
      threshold: 0.72
    dog:
      threshold: 0.72
    bird:
      threshold: 0.72
clips:
  retain:
    # Required: Default retention days (default: shown below)
    default: 100

detectors:
  coral:
    type: edgetpu
    device: usb

ffmpeg:
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420px

cameras:
  right:
    width: 1080
    height: 1920
    fps: 4
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED:{FRIGATE_RTSP_PASSWORD}@REDACTED:554/Streaming/channels/1
          roles:
            - detect
            - rtmp
            - clips
    objects:
      filters:
        person:
          min_area: 9000

    snapshots:
      enabled: True
      timestamp: False
      bounding_box: True
    clips:
      enabled: True
    motion:
      mask:
        - 961,218,933,0,1080,0,1080,1527,1004,1920,1080,1920,1080,948,994,392,623,129,505,376,383,672,140,1353,0,1866,0,0,687,0,678,30

  left:
    width: 1080
    height: 1920
    fps: 4
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED:{FRIGATE_RTSP_PASSWORD}@REDACTED:554/Streaming/Channels/101?transportmode=multicast&profile=Profile_1
          roles:
            - detect
            - rtmp
            - clips
    objects:
      filters:
        person:
          min_area: 14000
    snapshots:
      enabled: True
      timestamp: False
      bounding_box: True
    clips:
      enabled: True
    motion:
      mask:
        - 653,128,654,179,886,169,872,176,905,0,0,0,0,406,335,144
        - 1080,0,1080,205,952,178,945,101,987,0
        - 1080,952,1031,917,1047,817,1014,796,1059,768,1080,667
        - 813,368,948,368,951,292,808,298

  front:
    width: 1920
    height: 1080
    fps: 4
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED:{FRIGATE_RTSP_PASSWORD}@REDACTED:554/Streaming/Channels/101?transportmode=multicast&profile=Profile_1
          roles:
            - detect
            - rtmp
            - clips
    snapshots:
      enabled: True
      timestamp: False
      bounding_box: True
    clips:
      enabled: True
    objects:
      track:
        - car
        - truck
        - person
        - cat
        - dog
        - bird
    zones:
    # Coordinates can be generated at https://www.image-map.net/

      on_property:
        coordinates: 552,0,581,60,823,172,869,207,1198,443,1231,424,1736,807,1810,679,1920,768,1917,1073,3,1080,0,0
      on_sidewalk:
        coordinates: 570,0,863,103,1206,315,1264,314,1919,802,1919,683,1318,254,923,61,721,0
      on_street:
        coordinates: 573,0,855,90,862,117,1191,303,1275,317,1919,783,1919,0
      at_gate:
        coordinates: 598,0,682,0,780,0,987,0,871,97
      inside_gate:
        coordinates: 621,380,821,212,808,139,660,85,504,119
      car_pulling_in:
        coordinates: 821,299,1138,523,1207,560,1342,704,741,949,415,531,615,391,574,277,688,222
      car_sidewalk:
        coordinates: 531,132,654,132,859,216,1211,464,1920,1012,1920,1080,1847,1080
      car_street:
        coordinates: 464,0,483,61,1167,426,1218,483,1920,1040,1919,0
    motion:
      mask:
        - 0,1080,0,0,62,0,792,1080


  back:
    width: 1920
    height: 1080
    fps: 4
    ffmpeg:
      inputs:
        - path: rtsp://REDACTED:{FRIGATE_RTSP_PASSWORD}@REDACTED:554/Streaming/Channels/101?transportmode=multicast&profile=Profile_1
          roles:
            - detect
            - rtmp
            - clips
    objects:
      filters:
        person:
          min_area: 9000
    snapshots:
      enabled: True
      timestamp: False
      bounding_box: True
    clips:
      enabled: True
    motion:
      mask:
        - 739,0,745,54,124,56,120,0

Frigate container logs

y", line 932 in _boThread 0xot00007f013e467700s (most recent call first):

t  File r"a/opp_t/ifnrniegra

te/frigate/video.py"  File , line "205 in /ruusnr

/  File l"i/ubs/rp/yltihbo/np3y.t8h/otnh3.8/threraedaidnign.pgy.p"y, line "932, line  in 890_ in b_obootosttsrtarpa_pi

n
      
nThread 0xer

00007f0148a7d700 (most recent call first):

  File   File "/usr/lib/python3.8/thre"a/doipntg/.fpryi"g, line a890t in e_/bforoitgastter/alpo

g

.Thread 0xpy", line 00007f0148a7d70071 (most recent call first):

 in   File run

  File "/usr/lib/python3".8//otphtr/efaridgiantge./pfyr"i, line g932a in t_eb/olootgs.tprya"p_inne, line r71

 in   File r"u/nu

s  File r"//luisbr//plyitbh/opny3t.h8o/nt3h.r8e/atdhirneg.apdyi"n, line g890. in p_y"b, line o932o in t_sbtoroatps

t

rThread 0xa00007f0154d02740p (most recent call first):

_  File i"n/nuesrr

/  File l"i/bu/spr/yltihbo/np3y.t8h/otnh3r.e8a/dtihng.py"reading.py", line 890 in _bootst, line r1027a in p

_

wThread 0xai00007f0154d02740t (most recent call first):

_  File f"o/ru_strs/tlaitbe/_plyotckh

on3.8/threading.  File py""/usr/lib/python3.8/thread, line i1027n in g_.wpayi"t, line _1011f in ojro_itns
      

k

  File "/opt/frigate/f  File r"i/guastre//lviibd/epoy.tphyo"n3.8/threading.py", line 1011 in j, line o245in in 

capture_camera

  File   File "/opt/frigate/frigate/video".p/yu"sr/l, line i245b in /pcyaptthuorne3_.c8a/mmeurlat

i  File processing/process.py", line 108 in ru"n

/usr/lib/python3.8/  File mu"l/tuisprr/olcibe/sspiyntgh/opnr3o.c8e/msus.lptyip"r, line o108ce in srsuinn

g/process.py", line 315 in _boot  File s"t/ruaspr

/lib/python3.8/mult  File ip"r/oucsers/sliinbg//ppyrtohcoens3s..8p/ym"u, line l315t in i_pbroooctesstsrianpg

/popen_fork.py"  File "/usr/lib/python, line 375. in 8/_mlualutnicphr

ocessing/popen_fork.py  File ""/usr/lib/python3.8/mul, line t75i in pr_olcaeunscshi

ng/popen_fork.py"  File "/usr/lib/pytho, line n193. in 8/_m_uilntiitp_r_o

cessing/popen_f  File ork.py", line 19 in __init__

"/usr/lib/python  File 3.8/multiprocessing/context.py", line 277 in "/usr/lib/python3.8/_mPuolpteinp

r  File ocessing/context.py""/usr/lib/python3.8/, line mu277l in tiprocessing/context._pPyo"p, line e224n in 

_  File Popen

  File "/usr/lib/pyth00007f014827c700on3. (most recent call first):

8/  File mu"l/tuisprr/olciebs/spiyntgh/opnro3c.e8s/tsh.rpeya"di, line n121 in gs.tpayr"t, line 

302 in   File wait

  File "/opt/frigate/frigate/a"pp/.upsyr"/lib/python3.8/multiproc, line e182s in sing/queues.py", line 227 in start_camera_captur_e_fpereodc

es  File s"e/su

sr/lib/python3.8/thread  File i"ng/.oppyt"/f, line r870i in grautne

/  File f"r/iguastre//laipbp/.ppyyt"ho, line n2283 in .s8t/atrhrte

a  File ding.py", line 932 in _bootstrap"_i/nonpetr/f

ri  File g"a/tuesr//flriib/gpayttheo/n_3._8m/atihnr_e_a.dpiyn"g.py", line 890 in _bootstrap


Current thread 0x, line 00007f0147a7b700 (most recent call first):

15  File  in "</moopdtu/lfer>i

gate/frigate/video.py"  File , line "117 in /cuasptru/rlei_bf/rpaymtehso

n3  File ."8//rouptn/pfyr.ipgya"te, line /f87r in igate/video.py"_run_code

, line 231 in ru  File n

  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap"_/iunsnre/rl

i  File b"//upsyrt/hliobn/3p.y8t/hrounn3p.y8./ptyh"r, line e194a in ding.py", line 890 in _bootstrap


Thread 0x00007f013e467700 (most recent call first):

  File "/opt/frigat_er/ufnr_imgoadtuel/ev_iadse_om.apiyn"

, line 205 in run

  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner

  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap


Thread 0x00007f0148a7d700 (most recent call first):

  File "/opt/frigate/frigate/log.py", line 71 in run

  File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner

  File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap


Thread 0x00007f0154d02740 (most recent call first):

  File "/usr/lib/python3.8/threading.py", line 1027 in _wait_for_tstate_lock

  File "/usr/lib/python3.8/threading.py", line 1011 in join

  File "/opt/frigate/frigate/video.py", line 245 in capture_camera

  File "/usr/lib/python3.8/multiprocessing/process.py", line 108 in run

  File "/usr/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap

  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75 in _launch

  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19 in __init__

  File "/usr/lib/python3.8/multiprocessing/context.py", line 277 in _Popen

  File "/usr/lib/python3.8/multiprocessing/context.py", line 224 in _Popen

  File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start

  File "/opt/frigate/frigate/app.py", line 182 in start_camera_capture_processes

  File "/opt/frigate/frigate/app.py", line 228 in start

  File "/opt/frigate/frigate/__main__.py", line 15 in <module>

  File "/usr/lib/python3.8/runpy.py", line 87 in _run_code

  File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main

"/usr/lib/python3.8/multiprocessing/context.py", line 224 in _Popen

  File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start

  File "/opt/frigate/frigate/app.py", line 182 in start_camera_capture_processes

  File "/opt/frigate/frigate/app.py", line 228 in start

  File "/opt/frigate/frigate/__main__.py", line 15 in <module>

  File "/usr/lib/python3.8/runpy.py", line 87 in _run_code

  File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main

frigate.video                  INFO    : right: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures

frigate.video                  INFO    : right: ffmpeg process is not running. exiting capture thread...

ffmpeg.right.detect            ERROR   : Unrecognised hwaccel output format: yuv420pxIncreasing reorder buffer to 11

ffmpeg.right.detect            ERROR   : [flv @ 0x559af2175840] Failed to update header with correct duration.

ffmpeg.right.detect            ERROR   : [flv @ 0x559af2175840] Failed to update header with correct filesize.

watchdog.right                 INFO    : No frames received from right in 20 seconds. Exiting ffmpeg...

watchdog.right                 INFO    : Waiting for ffmpeg to exit gracefully...

frigate.video                  INFO    : right: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures

frigate.video                  INFO    : right: ffmpeg process is not running. exiting capture thread...

ffmpeg.right.detect            ERROR   : Unrecognised hwaccel output format: yuv420pxFailed to update header with correct duration.

ffmpeg.right.detect            ERROR   : [flv @ 0x5577dfae7c40] Failed to update header with correct filesize.

Frigate stats

{
  "back": {
    "camera_fps": 2.5, 
    "capture_pid": 53, 
    "detection_fps": 0.0, 
    "pid": 41, 
    "process_fps": 2.3, 
    "skipped_fps": 0.0
  }, 
  "detection_fps": 3.7, 
  "detectors": {
    "coral": {
      "detection_start": 0.0, 
      "inference_speed": 9.66, 
      "pid": 28078
    }
  }, 
  "front": {
    "camera_fps": 2.3, 
    "capture_pid": 44, 
    "detection_fps": 0.0, 
    "pid": 40, 
    "process_fps": 2.3, 
    "skipped_fps": 0.0
  }, 
  "left": {
    "camera_fps": 2.7, 
    "capture_pid": 43, 
    "detection_fps": 0.0, 
    "pid": 39, 
    "process_fps": 2.5, 
    "skipped_fps": 0.0
  }, 
  "right": {
    "camera_fps": 4.0, 
    "capture_pid": 42, 
    "detection_fps": 3.7, 
    "pid": 38, 
    "process_fps": 4.0, 
    "skipped_fps": 0.0
  }, 
  "service": {
    "storage": {
      "/dev/shm": {
        "free": 31.2, 
        "mount_type": "tmpfs", 
        "total": 67.1, 
        "used": 35.9
      }, 
      "/media/frigate/clips": {
        "free": 2370111.5, 
        "mount_type": "btrfs", 
        "total": 11508017.2, 
        "used": 9137905.8
      }, 
      "/media/frigate/recordings": {
        "free": 2370111.5, 
        "mount_type": "btrfs", 
        "total": 11508017.2, 
        "used": 9137905.8
      }, 
      "/tmp/cache": {
        "free": 976.2, 
        "mount_type": "tmpfs", 
        "total": 1000.0, 
        "used": 23.8
      }
    }, 
    "uptime": 1380338, 
    "version": "0.8.3-d771726"
  }
}

FFprobe from your camera

Run the following command and paste output below

ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
 built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, rtsp, from 'rtsp://REDACTED:554/Streaming/Channels/101?transportmode=unicast':
  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 1.350000, bitrate: N/A
    Stream #0:0: Video: h264, yuv420p(progressive), 1080x1920, 4 fps, 25 tbr, 90k tbn, 8 tbc

Screenshots

These screenshots were taken at March 29 15:00. You can see that activity suddenly ceased 2 days prior.

Computer Hardware

• OS: [e.g. Ubuntu, Windows] Synology
• Install method: [e.g. Addon, Docker Compose, Docker Command] Docker compose
• Virtualization: [e.g. Proxmox, Virtualbox] No
• Coral Version: [e.g. USB, PCIe, None] USB
• Network Setup: [e.g. Wired, WiFi] Wired

Camera Info: Hikvision Additional context Add any other context about the problem here.

[gpete]

Have you tried calculating and checking your shared memory size (Frigate Calculating shm-size)? I was having this same issue (UI running but detection frozen) with very similar garbled log files and solved it by calculating and bumping my shm_size up for the docker container. I’ve been running over a day now without issue where before it would crash each camera as soon as it detected motion.

@scstraus your 4 1080x1920 cameras would need a shm_size of 84mb but docker defaults to 64mb. This could be your issue too since I didn’t see how your container was being started.

My current initial setup has 4 high resolution cameras (4560 x 1440) and it would freeze almost as soon as it had an object to detect. In my logs I also noticed I was getting Fatal Python error: Bus error. Some search indicated this had to do with memory. Since the machine I’m set up on has 64GB of memory I was surprised but figured it had to do with a docker allocation. Turns out this is already documented and I just missed it! (see link above)

According to the formula my cameras need about 66MB each or about 265MB for all 4. Docker defaults to only 64MB of shared memory. From what I understand the shared memory is crucial for sharing camera frames around the capture, detection, and classification processes of all images. For the 8+ cameras I’ll eventually add I’ll need 400MB+ of shared memory. Since I have ram in excess I just went for 1gb to test if it would fix it. The cameras have now run for a day without any issues so that solved it for me.

Calculate the shm-size using the documented formula (width * height * 1.5 * 7 + 270480)/1048576 = <shm size per camera in mb>. Make sure you total for all cameras you are adding, or multiply if all your cameras are the same resolution. You might want to go a little higher like I did just to make sure you have a little room for error or expansion.

For docker-compose add a shm_size line. Don’t forget to remove and recreate your container (since I’m not sure it is something that can be modified after creation). It should look something like this:

services:
  frigate:
    shm_size: '500mb'
    ...

If you are using docker run you’ll need to add --shm-size=500m to your command.

For Home Assistant setups refer to the documentation (link above) since you’ll be you’ll be modifying the default-shm-size which will affect more than just the frigate container with that change.

[tuomaz]

I have the same problem. Stack traces looks very similar. I also have Hikvision cameras.