Performance Optimizations

This is not a bug report.

I just want to thank you for the wonderful project. It is extremely promising and very streamlined considering it is young age.

I moved to frigate from Shinobi, which has the best performance and very optimized on the hardware-side. Shinobi has excellent feature set and extensibility but it is not completely free, the UX is jarring and integration with HomeAssistant is fully manual (based on webhooks).

Frigate is much easier to run, use and integrate, but it has bigger footprint and needs more resources. Some of these extra resources is due to it being a Docker container rather than a bare-metal installation, and everyone would happily accept that. However, I have some suggestions to squeeze even more performance out of it:

HW Acceleration on Intel QSV:

The best possible performance is achieved on intel platforms using the newest drivers, which can now be easily installed directly from intel: https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-focal.html I made a Dockerfile-friendly snippet (some of it may not be needed in a production environment:

sudo apt-get install -y gpg-agent wget && \
wget -qO - https://repositories.intel.com/graphics/intel-graphics.key | sudo apt-key add - && \
sudo apt-add-repository 'deb [arch=amd64] https://repositories.intel.com/graphics/ubuntu focal main' && \
sudo apt-get update && sudo apt-get install intel-opencl-icd intel-level-zero-gpu level-zero intel-media-va-driver-non-free libmfx1 && \
sudo apt-get update && sudo apt-get install -y \
    intel-media-va-driver-non-free \
    libigfxcmrt7 \
    libmfx1 \
    libmfx-tools \
    libva-drm2 \
    libva-x11-2 \
    libxcb-shm0 \
    linux-tools-generic \
    man-db \
    pciutils \
    psmisc \
    python3 \
    python3-matplotlib \
    python3-numpy \
    python3-pip \
    tmux \
    vainfo && \
sudo setcap cap_sys_admin+ep $(readlink -f $(which perf)) && \
sudo getcap $(readlink -f $(which perf)) && \
sudo setcap cap_sys_admin+ep $(sudo find / -name intel_gpu_top) && \
sudo getcap $(sudo find / -name intel_gpu_top) && \
sudo apt-get install --no-install-recommends -y \
    asciidoc-base bison cython3 docbook-xsl flex gcc g++ git \
    libcairo-dev libdrm-dev libdw-dev libkmod-dev libmfx-dev libpciaccess-dev \
    libpixman-1-dev libprocps-dev libudev-dev libva-dev libx264-dev libx265-dev \
    libmp3lame-dev libopenjp2-7-dev librsvg2-dev libtheora-dev libtwolame-dev \
    libvidstab-dev libvorbis-dev libvpx-dev libwebp-dev libx265-dev libnuma-dev libssl-dev \
    libopus-dev librtmp-dev libxvidcore-dev libzmq3-dev ocl-icd-opencl-dev libfdk-aac-dev \
    make meson patch pkg-config \
    python3 python3-dev python3-numpy python3-setuptools python3-wheel \
    sudo xsltproc yasm && \
cd $HOME && \
git clone --depth 1 --branch v1.5.2 https://github.com/Netflix/vmaf.git && \
cd $HOME/vmaf && \
wget https://raw.githubusercontent.com/intel/media-delivery/master/patches/vmaf/0001-switch-to-setuptools.patch && \
wget https://raw.githubusercontent.com/intel/media-delivery/master/patches/vmaf/0002-Fix-vmaf-python-module-version-to-be-1.5.2.patch && \
wget https://raw.githubusercontent.com/intel/media-delivery/master/patches/vmaf/0003-Add-rev_target-to-libvmaf_rc-for-proper-building-ord.patch && \
git apply 0001-switch-to-setuptools.patch && \
git apply 0002-Fix-vmaf-python-module-version-to-be-1.5.2.patch && \
git apply 0003-Add-rev_target-to-libvmaf_rc-for-proper-building-ord.patch && \
cd $HOME/vmaf/libvmaf \
  && meson build \
  --buildtype=release \
  && ninja -j $(nproc --all) -C build \
  && sudo ninja -C build install && \
cd $HOME/vmaf/python && sudo python3 setup.py bdist_wheel && \
mkdir -p $HOME/ffmpeg_sources $HOME/bin && \
cd $HOME/ffmpeg_sources && \
wget -O ffmpeg.tar.bz2 https://ffmpeg.org/releases/ffmpeg-4.3.2.tar.bz2 && \
tar xjvf ffmpeg.tar.bz2 && \
cd $HOME/ffmpeg_sources/ffmpeg-4.3.2 && \
PATH="$HOME/bin:$PATH" ./configure \
--bindir="$HOME/bin" \
--disable-doc \
--enable-vaapi \
--enable-libmfx \
--enable-gpl \
--enable-libx264 \
--enable-libx265 \
--enable-version3 \
--enable-libvmaf \
--enable-libdrm \
--enable-libfdk-aac \
--enable-libmp3lame \
--enable-libopenjpeg \
--enable-libopus \
--enable-librsvg \
--enable-libtheora \
--enable-libtwolame \
--enable-libvidstab \
--enable-libvorbis \
--enable-libvpx \
--enable-libwebp \
--enable-libxvid \
--enable-libzmq \
--enable-nonfree \
--enable-opencl \
--enable-openssl \
--enable-librtmp \
--enable-static \
--enable-swresample \
--enable-swscale \
--extra-cflags="-I/usr/include" \
--extra-libs="-lpthread -lm" \
--prefix="$HOME/ffmpeg_build" \
--toolchain=hardened && \
export LIBVA_DRIVER_NAME=iHD &&\
PATH="$HOME/bin:$PATH" make -j $(nproc --all) && \
make install && \
hash -r && \
source $HOME/.profile && \
sudo ln -s $HOME/bin/ffmpeg /usr/local/bin/ffmpeg && \
sudo ln -s $HOME/bin/ffprobe /usr/local/bin/ffprobe && \
sudo usermod -aG video,audio,render,plugdev,dialout,voice,avahi $USER && \
sudo apt purge -y comerr-dev flite1-dev frei0r-plugins-dev gir1.2-freedesktop \
gir1.2-gdkpixbuf-2.0 gir1.2-harfbuzz-0.0 gir1.2-ibus-1.0 gir1.2-rsvg-2.0 \
icu-devtools krb5-multidev ladspa-sdk libaom-dev libasound2-dev libass-dev \
libauthen-sasl-perl libblkid-dev libbs2b-dev libcaca-dev libcairo2-dev \
libcairo-dev libcairo-script-interpreter2 libcdio-cdda-dev libcdio-dev \
libcdio-paranoia-dev libchromaprint-dev libcodec2-dev libdc1394-dev libdrm-dev \
libdw-dev libegl1-mesa-dev libegl-dev libfdk-aac-dev libfile-listing-perl \
libfindlib-ocaml libfindlib-ocaml-dev libfont-afm-perl libfontconfig1-dev \
libfreetype6-dev libfreetype-dev libfrei0r-ocaml-dev libfribidi-dev \
libgdk-pixbuf2.0-dev libgl1-mesa-dev libgl-dev libgles2-mesa-dev libgles-dev \
libglib2.0-dev libglib2.0-dev-bin libglu1-mesa-dev libglvnd-dev libglx-dev \
libgme-dev libgmp-dev libgmpxx4ldbl libgnutls28-dev libgnutls-openssl27 \
libgnutlsxx28 libgraphite2-dev libgsm1-dev libgssrpc4 libharfbuzz-dev \
libharfbuzz-gobject0 libharfbuzz-icu0 libhtml-format-perl libhtml-form-perl \
libhtml-tree-perl libhttp-cookies-perl libhttp-daemon-perl libhttp-negotiate-perl \
libibus-1.0-5 libibus-1.0-dev libice6 libice-dev libicu-dev libidn2-dev \
libio-socket-ssl-perl libjack-dev libkadm5clnt-mit11 libkadm5srv-mit11 libkdb5-9 \
libkmod-dev libkrb5-dev libladspa-ocaml libladspa-ocaml-dev liblilv-dev libltdl-dev \
liblwp-protocol-https-perl libmailtools-perl libmfx-dev libmount-dev libmp3lame-dev \
libmpg123-dev libmysofa-dev libnet-http-perl libnet-smtp-ssl-perl libnet-ssleay-perl \
libnorm-dev libnuma-dev libogg-dev libomxil-bellagio0 libomxil-bellagio-dev \
libopenal-dev libopengl-dev libopenjp2-7-dev libopenmpt-dev libopus-dev libout123-0 \
libp11-kit-dev libpciaccess-dev libpcre16-3 libpcre2-16-0 libpcre2-32-0 libpcre2-dev \
libpcre2-posix2 libpcre32-3 libpcre3-dev libpcrecpp0v5 libpgm-dev libpixman-1-dev \
libpng-dev libprocps-dev libpthread-stubs0-dev libpulse-dev libraw1394-dev \
librsvg2-dev librtmp-dev librubberband-dev libsdl2-dev libselinux1-dev libsepol1-dev \
libserd-dev libset-scalar-perl libshine-dev libslang2-dev libsm6 libsm-dev \
libsnappy-dev libsndio-dev libsodium-dev libsord-dev libsoxr-dev libspeex-dev \
libsratom-dev libssh-dev libssl-dev libtasn1-6-dev libtasn1-doc libtheora-dev \
libtry-tiny-perl libtwolame-dev libudev-dev libunbound8 libva-dev libva-glx2 \
libvdpau-dev libvidstab-dev libvorbis-dev libvpx-dev libwayland-bin libwayland-dev \
libwebp-dev libwww-perl libwww-robotrules-perl libx11-dev libx264-dev libx265-dev \
libxau-dev libxcb1-dev libxcb-render0-dev libxcb-shape0-dev libxcb-shm0-dev \
libxcb-xfixes0-dev libxcursor-dev libxdmcp-dev libxext-dev libxfixes-dev libxi-dev \
libxinerama-dev libxkbcommon-dev libxml-parser-perl libxml-sax-expat-perl \
libxrandr-dev libxrender-dev libxss-dev libxt6 libxt-dev libxv-dev libxvidcore-dev \
libxxf86vm-dev libzmq3-dev libzvbi-dev lv2-dev nettle-dev ocaml-base-nox \
ocaml-compiler-libs ocaml-findlib ocaml-interp ocaml-man ocaml-nox ocl-icd-opencl-dev \
opencl-c-headers perl-openssl-defaults uuid-dev x11proto-core-dev x11proto-dev \
x11proto-input-dev x11proto-randr-dev x11proto-scrnsaver-dev x11proto-xext-dev \
x11proto-xf86vidmode-dev x11proto-xinerama-dev xorg-sgml-doctools xtrans-dev yasm && \
sudo apt update && sudo apt full-upgrade -y && sudo apt autoremove --purge -y && sudo apt autoclean && sudo apt clean

This was an example only and should be adjusted to fit your own environment. Please note I also use LIBVA_DRIVER_NAME=iHD as default because the newest intel drivers works best with it even on older intel CPUs.

Using HLS stream instead of NGINX RTMP module for front end

I noticed that the same Frigate camera live stream used on HomeAssistant front-end uses much less CPU compared to viewing the exact same feed on Frigate’s own front-end. This is because HA uses HLS (as does Shinobi) to offload the work to the client browser. You should explore switching to HLS for your front-end. The difference in CPU use becomes bigger with increasing camera streams, so HLS is much better for scalability.

I hope that some of my points can help you make your project even better. If I have time next month, I will try and contribute directly to your code rather than commenting about it.

All the best,

[mssaleh]

Thanks for the quick response. I took another look at your Dockerfiles. They are good! You already removed some superfluous FFMPEG options related to using it for a display on the local machine. However, more work needs to be done. I will try next week to remove as many FFMPEG build options as possible for a server-only version with focus on H264 and HEVC RTSP sources hardware accelerated via Intel QuickSync. I think it will end up as a separate Docker tag because of the narrow scope, but it will be more optimized.

I will be waiting for HLS streaming feature because your web UI now is more complete but jsmpeg will not solve the biggest downside of HLS which is HEVC support. Even in HomeAssistant, HEVC livestream only works on Apple devices (if the camera is respectable enough to provide standard-compliant HEVC stream, which is unfortunately not the case with many Chinses IP cams). Shinobi project does it through the same Socket.io which you may be using for jsmpeg, but with a separate library: https://github.com/strukturag/libde265.js

Best Regards,

[paularmstrong]

I am worried that doing a bunch of heavy lifting to make drawing boxes, timestamps, etc only possible in the web UI will not address a good number of people that want to be able to stream feeds with that information to other locations (like casting to TVs, or via the Home Assistant lovelace camera view)