Some tooling helpers to generate config based on ffmpeg paramters

Describe what you are trying to accomplish and why in non technical terms I want to be able to port my ffmpeg arguments into frigate in a reliable way

Describe the solution you’d like Clearer documentation on what the different ffmpeg config values are supposed to be used for

So basically I’m one of the lucky people who bought reolink 4k cameras which only support h265 on the main stream. So I managed to cook up this ffmpeg command:

ffmpeg -hwaccel qsv -c:v hevc_qsv -i 'rtsp://admin:password@10.20.1.13:554//h265Preview_01_main' -c:v h264_qsv -load_plugin h264_hw -b 10M -c:a copy -y output.mp4

Which uses intel quicksync to both decode and encode the stream. From h265 to h264 while just copying the sound stream. Now by using qsv I went from 100% utilization of a single core on my E-2224G to less than 10% pr stream. This is very acceptable, but my problem now is that I can’t manage to translate the above ffmpeg arguments into frigate config. I’ve tried the following:

            hwaccel_args:
              - '-hwaccel'
              - 'qsv'
              - '-c:v'
              - 'hevc_qsv'
            output_args:
              record: '-f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v h264_qsv -load_plugin h264_hw -b 10M -c:a copy'
              rtmp: '-c:v h264_qsv -load_plugin h264_hw -b 10M -c:a copy -f flv'

But it doesn’t work and I end up with a bunch of errors like so:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2021-11-14 14:08:12] frigate.app                    INFO    : Starting Frigate (0.9.4-26ae608)
[2021-11-14 14:08:12] frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
[2021-11-14 14:08:12] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2021-11-14 14:08:12] peewee_migrate                 INFO    : There is nothing to migrate
[2021-11-14 14:08:13] frigate.mqtt                   INFO    : MQTT connected
[2021-11-14 14:08:13] detector.coral                 INFO    : Starting detection process: 217
[2021-11-14 14:08:13] ws4py                          INFO    : Using epoll
[2021-11-14 14:08:13] frigate.app                    INFO    : Output process started: 218
[2021-11-14 14:08:13] frigate.app                    INFO    : Camera processor started for parkering: 221
[2021-11-14 14:08:14] frigate.app                    INFO    : Camera processor started for hagen: 223
[2021-11-14 14:08:14] frigate.app                    INFO    : Camera processor started for homelab: 226
[2021-11-14 14:08:14] frigate.app                    INFO    : Camera processor started for stua: 229
W driver/beagle/beagle_kernel_top_level_handler.cc:131] Could not set performance expectation : 66 (Inappropriate ioctl for device)
[2021-11-14 14:08:14] frigate.app                    INFO    : Camera processor started for veien: 231
[2021-11-14 14:08:13] frigate.edgetpu                INFO    : Attempting to load TPU as pci
[2021-11-14 14:08:14] frigate.edgetpu                INFO    : TPU found
[2021-11-14 14:08:14] frigate.app                    INFO    : Camera processor started for inngang: 242
[2021-11-14 14:08:14] frigate.app                    INFO    : Capture process started for parkering: 244
[2021-11-14 14:08:14] frigate.app                    INFO    : Capture process started for hagen: 247
[2021-11-14 14:08:14] frigate.app                    INFO    : Capture process started for homelab: 249
[2021-11-14 14:08:14] frigate.app                    INFO    : Capture process started for stua: 254
[2021-11-14 14:08:14] frigate.app                    INFO    : Capture process started for veien: 259
[2021-11-14 14:08:14] frigate.app                    INFO    : Capture process started for inngang: 263
[2021-11-14 14:08:14] ws4py                          INFO    : Using epoll
[2021-11-14 14:08:20] frigate.video                  INFO    : inngang: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2021-11-14 14:08:20] frigate.video                  INFO    : inngang: ffmpeg process is not running. exiting capture thread...
[2021-11-14 14:08:21] frigate.video                  INFO    : veien: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2021-11-14 14:08:21] frigate.video                  INFO    : veien: ffmpeg process is not running. exiting capture thread...
[2021-11-14 14:08:26] frigate.video                  INFO    : stua: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2021-11-14 14:08:26] frigate.video                  INFO    : stua: ffmpeg process is not running. exiting capture thread...
[2021-11-14 14:08:34] watchdog.veien                 ERROR   : FFMPEG process crashed unexpectedly for veien.
[2021-11-14 14:08:34] watchdog.veien                 ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2021-11-14 14:08:34] watchdog.veien                 ERROR   : You may have invalid args defined for this camera.
[2021-11-14 14:08:34] ffmpeg.veien.detect            ERROR   : [rtsp @ 0x55849e0aa880] Could not find codec parameters for stream 0 (Video: hevc, none): unspecified size
[2021-11-14 14:08:34] ffmpeg.veien.detect            ERROR   : Consider increasing the value for the 'analyzeduration' and 'probesize' options
[2021-11-14 14:08:34] ffmpeg.veien.detect            ERROR   : Output file #0 does not contain any stream
[2021-11-14 14:08:34] ffmpeg.veien.rtmp              ERROR   : Please use -b:a or -b:v, -b is ambiguous
[2021-11-14 14:08:34] ffmpeg.veien.rtmp              ERROR   : [h264_qsv @ 0x55e250a0c9c0] Current frame rate is unsupported
[2021-11-14 14:08:34] ffmpeg.veien.rtmp              ERROR   : [h264_qsv @ 0x55e250a0c9c0] some encoding parameters are not supported by the QSV runtime. Please double check the input parameters.
[2021-11-14 14:08:34] ffmpeg.veien.rtmp              ERROR   : Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[2021-11-14 14:08:34] watchdog.veien                 INFO    : Terminating the existing ffmpeg process...
[2021-11-14 14:08:34] watchdog.veien                 INFO    : Waiting for ffmpeg to exit gracefully...
[2021-11-14 14:08:34] ffmpeg.hagen.record_rtmp       ERROR   : av_interleaved_write_frame(): Connection reset by peer
[2021-11-14 14:08:34] ffmpeg.hagen.record_rtmp       ERROR   : [flv @ 0x563aedf233c0] Failed to update header with correct duration.
[2021-11-14 14:08:34] ffmpeg.hagen.record_rtmp       ERROR   : [flv @ 0x563aedf233c0] Failed to update header with correct filesize.
[2021-11-14 14:08:34] ffmpeg.hagen.record_rtmp       ERROR   : Error writing trailer of rtmp://127.0.0.1/live/hagen: Connection reset by peer
[2021-11-14 14:08:34] watchdog.hagen                 INFO    : Terminating the existing ffmpeg process...
[2021-11-14 14:08:34] watchdog.hagen                 INFO    : Waiting for ffmpeg to exit gracefully...
[2021-11-14 14:08:34] watchdog.stua                  ERROR   : FFMPEG process crashed unexpectedly for stua.
[2021-11-14 14:08:34] watchdog.stua                  ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2021-11-14 14:08:34] watchdog.stua                  ERROR   : You may have invalid args defined for this camera.
[2021-11-14 14:08:34] ffmpeg.stua.detect             ERROR   : [rtsp @ 0x55826b86f880] Could not find codec parameters for stream 0 (Video: hevc, none): unspecified size
[2021-11-14 14:08:34] ffmpeg.stua.detect             ERROR   : Consider increasing the value for the 'analyzeduration' and 'probesize' options
[2021-11-14 14:08:34] ffmpeg.stua.detect             ERROR   : Output file #0 does not contain any stream
[2021-11-14 14:08:34] ffmpeg.stua.rtmp               ERROR   : Please use -b:a or -b:v, -b is ambiguous
[2021-11-14 14:08:34] ffmpeg.stua.rtmp               ERROR   : [h264_qsv @ 0x5565e347ce00] Current frame rate is unsupported
[2021-11-14 14:08:34] ffmpeg.stua.rtmp               ERROR   : [h264_qsv @ 0x5565e347ce00] some encoding parameters are not supported by the QSV runtime. Please double check the input parameters.
[2021-11-14 14:08:34] ffmpeg.stua.rtmp               ERROR   : Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[2021-11-14 14:08:34] watchdog.stua                  INFO    : Terminating the existing ffmpeg process...
[2021-11-14 14:08:34] watchdog.stua                  INFO    : Waiting for ffmpeg to exit gracefully...
[2021-11-14 14:08:34] watchdog.inngang               ERROR   : FFMPEG process crashed unexpectedly for inngang.
[2021-11-14 14:08:34] watchdog.inngang               ERROR   : The following ffmpeg logs include the last 100 lines prior to exit.
[2021-11-14 14:08:34] watchdog.inngang               ERROR   : You may have invalid args defined for this camera.
[2021-11-14 14:08:34] ffmpeg.inngang.detect          ERROR   : [rtsp @ 0x55555aa06880] Could not find codec parameters for stream 0 (Video: hevc, none): unspecified size
[2021-11-14 14:08:34] ffmpeg.inngang.detect          ERROR   : Consider increasing the value for the 'analyzeduration' and 'probesize' options
[2021-11-14 14:08:34] ffmpeg.inngang.detect          ERROR   : Output file #0 does not contain any stream
[2021-11-14 14:08:34] ffmpeg.inngang.rtmp            ERROR   : Please use -b:a or -b:v, -b is ambiguous
[2021-11-14 14:08:34] ffmpeg.inngang.rtmp            ERROR   : [h264_qsv @ 0x5644d2368e00] Current frame rate is unsupported
[2021-11-14 14:08:34] ffmpeg.inngang.rtmp            ERROR   : [h264_qsv @ 0x5644d2368e00] some encoding parameters are not supported by the QSV runtime. Please double check the input parameters.
[2021-11-14 14:08:34] ffmpeg.inngang.rtmp            ERROR   : Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

From the frigate pod I can see that we end up with this command: ffmpeg -hide_banner -loglevel warning -hwaccel qsv -c:v hevc_qsv -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:password@10.20.1.13:554//h265Preview_01_main -c:v h264_qsv -load_plugin h264_hw -b 10M -c:a copy -f flv rtmp://127.0.0.1/live/stua

I tried executing my working ffmpeg command within the frigate container and that works fine. But somhow these extra flags or the order they are in is preventing this from working. So I might be asking for too much here as ffmpeg is very particular about how these arguments are ordered. But maybe it would be better to just be able to manually set the ffmpeg commands in full?

[brujoand]

I think those are general drawbacks. And with h264 the stream is also larger due to inferior compression so there is more data to shift around. Might not be a problem, but transcoding will always be a hit on image quality and would add delay. But if your simply proxying the camera streams to remove authentication or something then it’s a different story.

I’ve actually considered running two instances of frigate to reduce the load rather than have a transcoder in-front. But right now the load is fine so I’m sticking with one.

[PrplHaz4]

Yeah that approach worked pretty well, but there are a few problems. For one it’s a bit wasteful to decode and encode the same stream multiple times. I’ve got 7 cameras so it adds up. It also reduces the quality of the stream a bit.

But most importantly I want to have this setup as self contained as possible. Especially since I would like to run a service to convert the rtmp stream from frigate to webrtc for home assistant. So many arrows on my diagrams right now :p

Is that mostly due to the original stream being h265? Do you think the same downside exists for cameras outputting h264? I’ve been thinking of rtsp-simple-server for a while but trying to keep moving parts to a minimum…