[Support]: recording_cleanup exception: "sqlite3.OperationalError: too many SQL variables" (low impact)

Describe the problem you are having

Hey, firstly just wanted to say thank you once again for this project - what you’ve built here is incredible and outperforms most expensive commercial NVR systems by miles, even without the Coral (though I can’t wait till they come back in stock somewhere again… it’s been like 9 months now google and still no stock, globally? c’mon…). I hope you get some support with triaging/maintaining at some point because reading through some folks’ comments in this issues section causes me to lose a bit of faith in humanity with how ungrateful some people are 😢

Anyway, I’m not exactly sure of the functional impact of this issue because I haven’t actually noticed any issues using my frigate setup, but I assume something in my recordings directory isn’t being deleted / cleaned up correctly because I see this stack trace in my logs a bunch (see logs below).

I enabled DEBUG logging for the frigate.record module, and as you can see in my logs I appear to have 539352 recordings needing to be expired? 😅

I’m not sure how it got into this state in the first place (perhaps a bug in an older version caused a compounding effect), but I guess the deletion query on this line wasn’t written with the expectation of cleaning up 500k rows 😂

Not sure how to proceed, would be happy to try and submit a PR to make that deletion a bit more robust or at least fail a bit more gracefully with a warning in the logs explaining what to do here or something but will await your guidance first.

I guess I could probably just trash the entire recordings directory and SQLite DB but since we have a reproducible issue I’d rather use it as an opportunity to make frigate a tiny bit more robust 😄

Version

0.10.1-83481af

Frigate config file

{
  "birdseye": {
    "enabled": true,
    "height": 480,
    "mode": "objects",
    "quality": 8,
    "width": 704
  },
  "cameras": {
    "frontdoor": {
      "best_image_timeout": 60,
      "detect": {
        "enabled": true,
        "fps": 5,
        "height": 480,
        "max_disappeared": 15,
        "stationary": {
          "interval": 0,
          "max_frames": {
            "default": null,
            "objects": {}
          },
          "threshold": 50
        },
        "width": 704
      },
      "ffmpeg": {
        "global_args": [
          "-hide_banner",
          "-loglevel",
          "warning"
        ],
        "hwaccel_args": [],
        "input_args": [
          "-avoid_negative_ts",
          "make_zero",
          "-fflags",
          "+genpts+discardcorrupt",
          "-rtsp_transport",
          "tcp",
          "-stimeout",
          "5000000",
          "-use_wallclock_as_timestamps",
          "1"
        ],
        "inputs": [
          {
            "global_args": [],
            "hwaccel_args": [],
            "input_args": [],
            "path": "rtsp://admin:redacted@192.168.1.164:554/cam/realmonitor?channel=1&subtype=1",
            "roles": [
              "detect",
              "rtmp"
            ]
          },
          {
            "global_args": [],
            "hwaccel_args": [],
            "input_args": [],
            "path": "rtsp://admin:redacted@192.168.1.164:554/live",
            "roles": [
              "record"
            ]
          }
        ],
        "output_args": {
          "detect": [
            "-f",
            "rawvideo",
            "-pix_fmt",
            "yuv420p"
          ],
          "record": "-f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy",
          "rtmp": [
            "-c",
            "copy",
            "-f",
            "flv"
          ]
        }
      },
      "ffmpeg_cmds": [
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:redacted@192.168.1.164:554/cam/realmonitor?channel=1&subtype=1 -c copy -f flv rtmp://127.0.0.1/live/frontdoor -r 5 -s 704x480 -f rawvideo -pix_fmt yuv420p pipe:",
          "roles": [
            "detect",
            "rtmp"
          ]
        },
        {
          "cmd": "ffmpeg -hide_banner -loglevel warning -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:redacted@192.168.1.164:554/live -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/frontdoor-%Y%m%d%H%M%S.mp4",
          "roles": [
            "record"
          ]
        }
      ],
      "live": {
        "height": 480,
        "quality": 20
      },
      "motion": {
        "contour_area": 30,
        "delta_alpha": 0.2,
        "frame_alpha": 0.2,
        "frame_height": 50,
        "improve_contrast": false,
        "mask": [
          "704,100,0,100,0,0,704,0"
        ],
        "threshold": 25
      },
      "mqtt": {
        "bounding_box": true,
        "crop": true,
        "enabled": true,
        "height": 270,
        "quality": 70,
        "required_zones": [
          "doorsteps"
        ],
        "timestamp": true
      },
      "name": "frontdoor",
      "objects": {
        "filters": {
          "animal": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "cat": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "dog": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          },
          "person": {
            "mask": null,
            "max_area": 24000000,
            "min_area": 0,
            "min_score": 0.5,
            "threshold": 0.7
          }
        },
        "mask": "",
        "track": [
          "person",
          "cat",
          "dog",
          "animal"
        ]
      },
      "record": {
        "enabled": true,
        "events": {
          "max_seconds": 300,
          "objects": null,
          "post_capture": 5,
          "pre_capture": 5,
          "required_zones": [
            "doorsteps"
          ],
          "retain": {
            "default": 14,
            "mode": "active_objects",
            "objects": {}
          }
        },
        "expire_interval": 60,
        "retain": {
          "days": 7,
          "mode": "motion"
        },
        "retain_days": null
      },
      "rtmp": {
        "enabled": true
      },
      "snapshots": {
        "bounding_box": true,
        "clean_copy": true,
        "crop": false,
        "enabled": true,
        "height": null,
        "quality": 70,
        "required_zones": [
          "doorsteps"
        ],
        "retain": {
          "default": 10,
          "mode": "motion",
          "objects": {}
        },
        "timestamp": true
      },
      "timestamp_style": {
        "color": {
          "blue": 255,
          "green": 255,
          "red": 255
        },
        "effect": null,
        "format": "%m/%d/%Y %H:%M:%S",
        "position": "tl",
        "thickness": 2
      },
      "zones": {
        "doorsteps": {
          "coordinates": "347,480,0,480,0,281,351,224",
          "filters": {},
          "objects": []
        }
      }
    }
  },
  "database": {
    "path": "/media/frigate/frigate.db"
  },
  "detect": {
    "enabled": true,
    "fps": 5,
    "height": 480,
    "max_disappeared": 15,
    "stationary": {
      "interval": 0,
      "max_frames": {
        "default": null,
        "objects": {}
      },
      "threshold": null
    },
    "width": 704
  },
  "detectors": {
    "cpu1": {
      "device": "usb",
      "num_threads": 1,
      "type": "cpu"
    }
  },
  "environment_vars": {},
  "ffmpeg": {
    "global_args": [
      "-hide_banner",
      "-loglevel",
      "warning"
    ],
    "hwaccel_args": [],
    "input_args": [
      "-avoid_negative_ts",
      "make_zero",
      "-fflags",
      "+genpts+discardcorrupt",
      "-rtsp_transport",
      "tcp",
      "-stimeout",
      "5000000",
      "-use_wallclock_as_timestamps",
      "1"
    ],
    "output_args": {
      "detect": [
        "-f",
        "rawvideo",
        "-pix_fmt",
        "yuv420p"
      ],
      "record": "-f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy",
      "rtmp": [
        "-c",
        "copy",
        "-f",
        "flv"
      ]
    }
  },
  "live": {
    "height": 480,
    "quality": 20
  },
  "logger": {
    "default": "info",
    "logs": {
      "frigate.record": "debug"
    }
  },
  "model": {
    "height": 320,
    "labelmap": {},
    "labelmap_path": null,
    "path": null,
    "width": 320
  },
  "motion": null,
  "mqtt": {
    "client_id": "frigate",
    "host": "192.168.1.3",
    "password": "mqtt",
    "port": 1883,
    "stats_interval": 30,
    "tls_ca_certs": null,
    "tls_client_cert": null,
    "tls_client_key": null,
    "tls_insecure": null,
    "topic_prefix": "frigate",
    "user": "mqtt"
  },
  "objects": {
    "filters": null,
    "mask": "",
    "track": [
      "person",
      "cat",
      "dog",
      "animal"
    ]
  },
  "record": {
    "enabled": true,
    "events": {
      "max_seconds": 300,
      "objects": null,
      "post_capture": 5,
      "pre_capture": 5,
      "required_zones": [
        "doorsteps"
      ],
      "retain": {
        "default": 14,
        "mode": "active_objects",
        "objects": {}
      }
    },
    "expire_interval": 60,
    "retain": {
      "days": 7,
      "mode": "motion"
    },
    "retain_days": null
  },
  "rtmp": {
    "enabled": true
  },
  "snapshots": {
    "bounding_box": true,
    "clean_copy": true,
    "crop": false,
    "enabled": true,
    "height": null,
    "quality": 70,
    "required_zones": [
      "doorsteps"
    ],
    "retain": {
      "default": 10,
      "mode": "motion",
      "objects": {}
    },
    "timestamp": true
  },
  "timestamp_style": {
    "color": {
      "blue": 255,
      "green": 255,
      "red": 255
    },
    "effect": null,
    "format": "%m/%d/%Y %H:%M:%S",
    "position": "tl",
    "thickness": 2
  }
}

Relevant log output

[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-03-25 09:02:20] frigate.app                    INFO    : Starting Frigate (0.10.1-83481af)
Starting migrations
[2022-03-25 09:02:20] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-03-25 09:02:20] peewee_migrate                 INFO    : There is nothing to migrate
[2022-03-25 09:02:20] frigate.app                    INFO    : Output process started: 219
[2022-03-25 09:02:20] detector.cpu1                  INFO    : Starting detection process: 217
[2022-03-25 09:02:20] frigate.edgetpu                WARNING : CPU detectors are not recommended and should only be used for testing or for trial purposes.
[2022-03-25 09:02:20] frigate.app                    INFO    : Camera processor started for frontdoor: 222
[2022-03-25 09:02:20] ws4py                          INFO    : Using epoll
[2022-03-25 09:02:20] frigate.app                    INFO    : Capture process started for frontdoor: 224
[2022-03-25 09:02:20] ws4py                          INFO    : Using epoll
[2022-03-25 09:02:26] frigate.record                 WARNING : Discarding a corrupt recording segment: frontdoor-20220325090223.mp4
[2022-03-25 09:03:20] frigate.record                 DEBUG   : Start expire recordings (new).
[2022-03-25 09:03:20] frigate.record                 DEBUG   : Start deleted cameras.
[2022-03-25 09:03:20] frigate.record                 DEBUG   : Expiring 0 recordings
[2022-03-25 09:03:20] frigate.record                 DEBUG   : End deleted cameras.
[2022-03-25 09:03:20] frigate.record                 DEBUG   : Start all cameras.
[2022-03-25 09:03:20] frigate.record                 DEBUG   : Start camera: frontdoor.
[2022-03-25 09:03:43] frigate.record                 DEBUG   : Expiring 539352 recordings
Exception in thread recording_cleanup:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3129, in execute_sql
    cursor.execute(sql, params or ())
sqlite3.OperationalError: too many SQL variables
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/opt/frigate/frigate/record.py", line 553, in run
    self.expire_recordings()
  File "/opt/frigate/frigate/record.py", line 462, in expire_recordings
    Recordings.delete().where(Recordings.id << deleted_recordings).execute()
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 1898, in inner
    return method(self, database, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 1969, in execute
    return self._execute(database)
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 2466, in _execute
    return self.handle_result(database, cursor)
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 2477, in handle_result
    return database.rows_affected(cursor)
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3221, in rows_affected
    return cursor.rowcount
  File "/usr/local/lib/python3.8/dist-packages/playhouse/sqliteq.py", line 94, in rowcount
    self._wait()
  File "/usr/local/lib/python3.8/dist-packages/playhouse/sqliteq.py", line 63, in _wait
    raise self._exc
  File "/usr/local/lib/python3.8/dist-packages/playhouse/sqliteq.py", line 178, in execute
    cursor = self.database._execute(obj.sql, obj.params, obj.commit)
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3136, in execute_sql
    self.commit()
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 2902, in __exit__
    reraise(new_type, new_type(exc_value, *exc_args), traceback)
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 185, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/dist-packages/peewee.py", line 3129, in execute_sql
    cursor.execute(sql, params or ())
peewee.OperationalError: too many SQL variables

FFprobe output from your camera

ffprobe version 4.2.4-1ubuntu0.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, rtsp, from 'rtsp://admin:redacted@192.168.1.164:554/cam/realmonitor?channel=1&subtype=1':
  Metadata:
    title           : Media Server
  Duration: N/A, start: 0.022000, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuvj420p(pc, smpte170m, progressive), 704x480 [SAR 12855:10604 DAR 857:482], 5 fps, 10 tbr, 90k tbn, 10 tbc
    Stream #0:1: Audio: aac (LC), 48000 Hz, mono, fltp

Frigate stats

{"detection_fps":0.0,"detectors":{"cpu1":{"detection_start":0.0,"inference_speed":106.63,"pid":216}},"frontdoor":{"camera_fps":5.1,"capture_pid":225,"detection_fps":0.0,"pid":223,"process_fps":5.1,"skipped_fps":0.0},"service":{"storage":{"/dev/shm":{"free":66.0,"mount_type":"tmpfs","total":67.1,"used":1.1},"/media/frigate/clips":{"free":470748.7,"mount_type":"ext4","total":7937442.7,"used":7066599.2},"/media/frigate/recordings":{"free":470748.7,"mount_type":"ext4","total":7937442.7,"used":7066599.2},"/tmp/cache":{"free":310757.4,"mount_type":"overlay","total":502412.6,"used":171176.3}},"temperatures":{},"uptime":1936,"version":"0.10.1-83481af"}}

Operating system

Debian

Install method

Docker Compose

Coral version

CPU (no coral)

Network connection

Wired

Camera make and model

Amcrest IP8M-T2499E

Any other information that may be helpful

No response

[blakeblackshear]

I think the right approach is to implement batching of deletes. Can you see if you can find a maximum somewhere?

In the meantime, you can adjust your retention settings so less would be deleted and phase it back gradually. Or you can shut down frigate, open up the sqlite db in another tool and run a query to delete by date.

My faith in humanity has evaporated after the past 2 years. The comments here don’t phase me and it seems like there is more good than bad to me at least. Thanks for being one of the good ones.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.