0.8.0rc3 >=Gen10 Hardware support

I am trying to get the hardware acceleration running on my Gen10 QSV hardware. Issue: https://community.home-assistant.io/t/local-realtime-person-detection-for-rtsp-cameras/103107/3212?u=davidvf

Version of frigate 0.8.0 rc3

I am running the basic ffmpeg command to get started. From the OS, if i run this, i see the intel gpu start working

MAIN OS

[david@owl:~ $] ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i rtsp://192.168.0.5:7447/16gU54OAVReFohX3 -c:v h264_qsv hwaccel-test.mp4
ffmpeg version N-100411-g32586a42da Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --enable-libmfx
  libavutil      56. 62.100 / 56. 62.100
  libavcodec     58.115.102 / 58.115.102
  libavformat    58. 65.100 / 58. 65.100
  libavdevice    58. 11.103 / 58. 11.103
  libavfilter     7. 93.100 /  7. 93.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
Input #0, rtsp, from 'rtsp://192.168.0.5:7447/16gU54OAVReFohX3':
  Metadata:
    title           : B4FBE4FF7EDE_0
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac (LC), 11025 Hz, mono, fltp
    Stream #0:1: Video: h264 (Main), yuvj420p(pc, bt709, progressive), 1920x1080 [SAR 189:190 DAR 168:95], 5 fps, 5 tbr, 90k tbn, 10 tbc
File 'hwaccel-test.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:1 -> #0:0 (h264 (h264_qsv) -> h264 (h264_qsv))
  Stream #0:0 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[aac @ 0x55f5818c25c0] Too many bits 6408.707483 > 6144 per frame requested, clamping to max
Output #0, mp4, to 'hwaccel-test.mp4':
  Metadata:
    title           : B4FBE4FF7EDE_0
    encoder         : Lavf58.65.100
    Stream #0:0: Video: h264 (avc1 / 0x31637661), qsv(pc, bt709, progressive), 1920x1080 [SAR 189:190 DAR 168:95], q=2-31, 1000 kb/s, 5 fps, 10240 tbn
    Metadata:
      encoder         : Lavc58.115.102 h264_qsv
    Side data:
      cpb: bitrate max/min/avg: 0/0/1000000 buffer size: 0 vbv_delay: N/A
    Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 11025 Hz, mono, fltp, 66 kb/s
    Metadata:
      encoder         : Lavc58.115.102 aac
frame=   67 fps= 11 q=24.0 Lsize=    1566kB time=00:00:13.37 bitrate= 959.0kbits/s dup=6 drop=1 speed=2.28x
video:1466kB audio:96kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.234253%
[aac @ 0x55f5818c25c0] Qavg: 65536.000
Exiting normally, received signal 2.

[david@owl:~ $]

intel_gpu_top output

[david@owl:~ $] sudo intel_gpu_top -l
 Freq MHz      IRQ RC6 Power           RCS/0           BCS/0           VCS/0          VECS/0
 req  act       /s   %     W       %  se  wa       %  se  wa       %  se  wa       %  se  wa
  59   59       67  95  0.12    1.61   0   0    0.00   0   0    2.83   1   0    0.00   0   0
  59   59       66  95  0.12    1.59   0   0    0.00   0   0    2.76   1   0    0.00   0   0
  68   68       72  94  0.12    1.66   0   0    0.00   0   0    3.04   0   0    0.00   0   0
  71   71       68  94  0.13    1.63   0   0    0.00   0   0    2.99   0   0    0.00   0   0
  65   65       66  94  0.12    1.72   0   0    0.00   0   0    3.04   1   0    0.00   0   0
  60   60       70  94  0.12    1.77   0   0    0.00   0   0    2.97   0   0    0.00   0   0
  59   59       70  94  0.13    1.74   0   0    0.00   0   0    2.86   0   0    0.00   0   0
  69   68       69  94  0.12    1.53   0   0    0.00   0   0    3.16   0   0    0.00   0   0
  59   60       68  94  0.12    1.77   0   0    0.00   0   0    2.98   0   0    0.00   0   0
  57   57       70  95  0.13    2.26   1   0    0.00   0   0    3.16   0   0    0.00   0   0
   0    0        0 100  0.00    0.00   0   0    0.00   0   0    0.00   0   0    0.00   0   0

If I run the same command from the docker container

root@81b0d26d6fc1:/opt/frigate# ffmpeg -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i rtsp://192.168.0.5:7447/16gU54OAVReFohX3 -c:v h264_qsv hwaccel-test.mp4
ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, rtsp, from 'rtsp://192.168.0.5:7447/16gU54OAVReFohX3':
  Metadata:
    title           : B4FBE4FF7EDE_0
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac, 11025 Hz, mono, fltp
    Stream #0:1: Video: h264, yuvj420p(pc, bt709, progressive), 1920x1080 [SAR 189:190 DAR 168:95], 5 fps, 5 tbr, 90k tbn, 10 tbc
Stream mapping:
  Stream #0:1 -> #0:0 (h264 (h264_qsv) -> h264 (h264_qsv))
  Stream #0:0 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[aac @ 0x5612d4186ac0] Too many bits 6408.707483 > 6144 per frame requested, clamping to max
[h264_qsv @ 0x5612d41877c0] Selected ratecontrol mode is unsupported
[h264_qsv @ 0x5612d41877c0] Low power mode is unsupported
[h264_qsv @ 0x5612d41877c0] Current frame rate is unsupported
[h264_qsv @ 0x5612d41877c0] Current picture structure is unsupported
[h264_qsv @ 0x5612d41877c0] Current resolution is unsupported
[h264_qsv @ 0x5612d41877c0] Current pixel format is unsupported
[h264_qsv @ 0x5612d41877c0] some encoding parameters are not supported by the QSV runtime. Please double check the input parameters.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x5612d4186ac0] Qavg: 65536.000
[aac @ 0x5612d4186ac0] 2 frames left in the queue on closing
Conversion failed!
root@81b0d26d6fc1:/opt/frigate#

FFprobe from your camera

root@81b0d26d6fc1:/opt/frigate# ffprobe rtsp://192.168.0.5:7447/16gU54OAVReFohX3
ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, rtsp, from 'rtsp://192.168.0.5:7447/16gU54OAVReFohX3':
  Metadata:
    title           : B4FBE4FF7EDE_0
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Audio: aac, 11025 Hz, mono, fltp
    Stream #0:1: Video: h264, yuvj420p(pc, bt709, progressive), 1920x1080 [SAR 189:190 DAR 168:95], 5 fps, 5 tbr, 90k tbn, 10 tbc

Computer Hardware

• OS:

cat /etc/os-release
NAME="Ubuntu"
VERSION="20.10 (Groovy Gorilla)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.10"
VERSION_ID="20.10"

• Virtualization: Docker
• Coral Version: USB
• Network Setup: Wired

Camera Info:

• Manufacturer: Unifi
• Model: unifi g3
• Resolution: 1920x1080
• FPS: 5

[ekos2001]

Will this non-free driver be added?

[ekos2001]

Yes, I’m also using non-free driver. I made small docker file Dockerfile.non-free on top of frigate for amd64:

FROM blakeblackshear/frigate:0.8.0-rc3-amd64
RUN apt-get -qq update \
    && apt-get -qq remove -y intel-media-va-driver \
    && apt-get -qq install --no-install-recommends -y \
        intel-media-va-driver-non-free \
    && (apt-get autoremove -y; apt-get autoclean -y)

Then I built it docker build -f Dockerfile.non-free -t frigate-non-free .

By the way, to use HW acceleration with H.265 cameras, -c:v h264_qsv should be replaced with -c:v hevc_qsv