[Support]: Frigate 0.10 deletes all event recordings during cleanup

Describe the problem you are having

Since upgrading to 0.10 Frigate keeps deleting all recordings belonging to events on next cleanup.

The event remains visible in the UI and the snapshot can be viewed but downloading yields a timeout and viewing yields a nginx 404.

Continuation of https://github.com/blakeblackshear/frigate/discussions/2447#discussioncomment-2021940

Version

0.10.0-db1255a

Frigate config file

mqtt:
  host: 192.168.x.x

detectors:
  coral:
    type: edgetpu
    device: usb:0

#logger:
#  default: debug

birdseye:
  enabled: False
  width: 1280
  height: 720
  quality: 8
  mode: objects

detect:
  max_disappeared: 30

objects:
  filters:
    person:
      min_score: 0.6
      threshold: 0.75
    car:
      min_score: 0.6
      threshold: 0.75

record:
  enabled: True
  events:
    max_seconds: 300
    pre_capture: 10
    post_capture: 10
    retain:
      default: 14
      objects:
        person: 30
        car: 14

snapshots:
  enabled: True
  timestamp: True
  retain:
    default: 30
    objects:
      person: 100
      car: 100

rtmp:
  enabled: False

live:
  height: 1080
  quality: 8

timestamp_style:
  position: "tl"
  format: "%Y-%m-%d %H:%M:%S"
  color:
    red: 255
    green: 255
    blue: 255
  thickness: 2
  effect: solid

cameras:
  grower:
    ffmpeg:
      inputs:
        - path: rtsp://x:x@192.168.x.x:554/s0
          roles:
            - rtmp
    detect:
      width: 1920
      height: 1080
      fps: 5
    rtmp:
      enabled: True
    record:
      enabled: False
    snapshots:
      enabled: False

  driveway:
    ffmpeg:
      inputs:
        - path: rtsp://x:x@192.168.x.x:554/h265Preview_01_main
          roles:
            - record
            - rtmp
        - path: rtsp://x:x@192.168.x.x:554/h264Preview_01_sub
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 4
    objects:
      track:
        - person
        - car
  garden:
    ffmpeg:
      inputs:
        - path: rtsp://x:x@192.168.x.x:554/h265Preview_01_main
          roles:
            - record
            - rtmp
        - path: rtsp://x:x@192.168.x.x:554/h264Preview_01_sub
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 4
    objects:
      track:
        - person

  garage:
    ffmpeg:
      inputs:
        - path: rtsp://x:x@192.168.x.x:554/Streaming/Channels/101?transportmode=mcast&profile=Profile_1
          roles:
            - record
            - rtmp
        - path: rtsp://x:x@192.168.x.x:554/Streaming/Channels/102?transportmode=mcast&profile=Profile_1
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 4
    objects:
      track:
        - person
        - car

  entrance:
    ffmpeg:
      inputs:
        - path: rtsp://x:x@192.168.x.x:554/Streaming/Channels/101?transportmode=mcast&profile=Profile_1
          roles:
            - record
            - rtmp
        - path: rtsp://x:x@192.168.x.x:554/Streaming/Channels/102?transportmode=mcast&profile=Profile_1
          roles:
            - detect
    detect:
      width: 640
      height: 360
      fps: 4
    objects:
      track:
        - person

Relevant log output

frigate    | [2022-01-22 11:14:50] frigate.http                   ERROR   : ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
frigate    |   built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
frigate    |   configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
frigate    |   libavutil      56. 51.100 / 56. 51.100
frigate    |   libavcodec     58. 91.100 / 58. 91.100
frigate    |   libavformat    58. 45.100 / 58. 45.100
frigate    |   libavdevice    58. 10.100 / 58. 10.100
frigate    |   libavfilter     7. 85.100 /  7. 85.100
frigate    |   libavresample   4.  0.  0 /  4.  0.  0
frigate    |   libswscale      5.  7.100 /  5.  7.100
frigate    |   libswresample   3.  7.100 /  3.  7.100
frigate    |   libpostproc    55.  7.100 / 55.  7.100
frigate    | [concat @ 0x55aacc75f740] Impossible to open '/media/frigate/recordings/2022-01/22/08/entrance/17.10.mp4'
frigate    | pipe:: No such file or directory



frigate    | [2022-01-22 11:26:21] frigate.record                 DEBUG   : Start camera: entrance.
frigate    | [2022-01-22 11:26:21] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."camera", "t1"."path", "t1"."start_time", "t1"."end_time", "t1"."duration", "t1"."motion", "t1"."objects" FROM "recordings" AS "t1" WHERE (("t1"."camera" = ?) AND ("t1"."end_time" < ?)) ORDER BY "t1"."start_time"', ['entrance', 1642846881.691337])
frigate    | [2022-01-22 11:26:21] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."label", "t1"."camera", "t1"."start_time", "t1"."end_time", "t1"."top_score", "t1"."false_positive", "t1"."zones", "t1"."thumbnail", "t1"."has_clip", "t1"."has_snapshot", "t1"."region", "t1"."box", "t1"."area" FROM "event" AS "t1" WHERE ((("t1"."camera" = ?) AND ("t1"."start_time" < ?)) AND "t1"."has_clip") ORDER BY "t1"."start_time"', ['entrance', 1642846881.691337])
frigate    | [2022-01-22 11:26:21] frigate.record                 DEBUG   : Expiring 0 recordings
frigate    | [2022-01-22 11:26:21] peewee.sqliteq                 DEBUG   : received query DELETE FROM "recordings" WHERE (0 = 1)
frigate    | [2022-01-22 11:26:21] peewee                         DEBUG   : ('DELETE FROM "recordings" WHERE (0 = 1)', [])
frigate    | [2022-01-22 11:26:21] frigate.record                 DEBUG   : End camera: entrance.
frigate    | [2022-01-22 11:26:21] frigate.record                 DEBUG   : End all cameras.
frigate    | [2022-01-22 11:26:21] frigate.record                 DEBUG   : End expire recordings (new).
frigate    | [2022-01-22 11:26:21] frigate.record                 DEBUG   : Start expire files (legacy).
frigate    | [2022-01-22 11:26:21] peewee                         DEBUG   : ('SELECT "t1"."id", "t1"."camera", "t1"."path", "t1"."start_time", "t1"."end_time", "t1"."duration", "t1"."motion", "t1"."objects" FROM "recordings" AS "t1" ORDER BY "t1"."start_time" LIMIT ? OFFSET ?', [1, 0])
frigate    | [2022-01-22 11:26:21] frigate.record                 WARNING : Unable to find file from recordings database: /media/frigate/recordings/2021-12/23/12/entrance/41.57.mp4
frigate    | [2022-01-22 11:26:21] frigate.record                 DEBUG   : Oldest recording in the db: 1642847181.728057
frigate    | [2022-01-22 11:26:21] frigate.record                 DEBUG   : End expire files (legacy).

FFprobe output from your camera

Input #0, rtsp, from 'rtsp://x:x@192.168.x.x:554/h265Preview_01_main':
  Metadata:
    title           : Session streamed by "preview"
    comment         : h265Preview_01_main
  Duration: N/A, start: 0.000375, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuv420p(tv), 3840x2160, 24.92 tbr, 90k tbn, 90k tbc
  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp

Frigate stats

{"detection_fps":1.7,"detectors":{"coral":{"detection_start":0.0,"inference_speed":7.64,"pid":225}},"driveway":{"camera_fps":4.0,"capture_pid":254,"detection_fps":0.2,"pid":246,"process_fps":4.0,"skipped_fps":0.0},"entrance":{"camera_fps":4.0,"capture_pid":267,"detection_fps":1.5,"pid":251,"process_fps":4.0,"skipped_fps":0.0},"garage":{"camera_fps":4.0,"capture_pid":262,"detection_fps":0.0,"pid":249,"process_fps":4.0,"skipped_fps":0.0},"garden":{"camera_fps":4.1,"capture_pid":256,"detection_fps":0.0,"pid":247,"process_fps":4.1,"skipped_fps":0.0},"grower":{"camera_fps":5.0,"capture_pid":252,"detection_fps":0.0,"pid":244,"process_fps":0.0,"skipped_fps":0.0},"service":{"storage":{"/dev/shm":{"free":1067.6,"mount_type":"tmpfs","total":1073.7,"used":6.2},"/media/frigate/clips":{"free":100529.1,"mount_type":"zfs","total":146261.4,"used":45732.3},"/media/frigate/recordings":{"free":100529.1,"mount_type":"zfs","total":146261.4,"used":45732.3},"/tmp/cache":{"free":1778.1,"mount_type":"tmpfs","total":2000.0,"used":221.9}},"temperatures":{},"uptime":8574,"version":"0.10.0-db1255a"}}

Operating system

Proxmox

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Mix of Hikvision, Reolink and Unifi

Any other information that may be helpful

Added the new record section for 0.10 as per the updated documentation.

[blakeblackshear]

I think I may have found the issue. When the legacy expiration runs, it doesn’t find the file for the oldest recording in the database, and it’s incorrectly assuming it should delete everything past the current time. This is an edge case that I hadn’t considered where somehow your recordings files must have been deleted when you upgraded, but the database still referenced those files. I can fix it for the next beta release. For now, the only way to fix it is to delete your database and let it get recreated.

[DomiStyle]

Closing this, works fine now with the 0.10.0 release.