Increased resource usage for recordings since new version

Yesterday we upgraded from stable-5390-3 to stable-5765-1 and we noticed that when starting a recording, when looking at the resource usage the RAM keeps raising by the second and after the whole memory is exhausted, the recording stops (this usually takes about 1-2 minutes with 8 GB of RAM). In the ffmpeg logs, I can see that the speed is around 0.6x, so it’s clear that ffmpeg can’t keep up with the recording.

We also noticed this behavior with stable-5390-3 when we were tweaking the jibri settings to improve the recording quality and went overboard, at some point when the chromedriver and ffmpeg are at the limit and they can’t keep up (which is noticeable by the speed in the ffmpeg logs not being around 0.99x) the memory usage keeps rising until the recording stops. But after we found the right settings we have consistently been getting around 0.99x of speed during recordings before the update to the newer version.

I had a look at the jibri commits that were made between the two releases, however I could not find any that are of relevance that would justify the increase in resource usage except from the resolution jump from 720p to 1080p, but already with the older version we manually increased the resolution of both ffmpeg and in the xorg-video-dummy.conf to 1080p, so this couldn’t be it. This leads me to believe that it must be either the jitsi interface consuming more resources (which would result in chromedriver using up more resources, thus leaving less resources for ffmpeg) or something in the other containers taking up more resources. But I don’t know the changes of the entire jitsi good enough, you will be able to estimate that better.

This seems very surprising to me given our VM already has quite a lot of resources (8 VCPU’s and 8 GB of RAM) so it seems odd to me that that is not enough. I have to note though that we run all containers on one VM (so Jibri is not on a separate VM) using this repository and the docker-compose -f docker-compose.yml -f jibri.yml up command. Nothing else is running on the VM, which runs on Ubuntu 20.04.

Some more details about our setup / settings and configuration:

.env file is almost the same for both versions, with the only exception of leaving out JICOFO_COMPONENT_SECRET for the newer version:

JICOFO_COMPONENT_SECRET={{ JICOFO_COMPONENT_SECRET }}
JICOFO_AUTH_PASSWORD={{ JICOFO_AUTH_PASSWORD }}
JVB_AUTH_PASSWORD={{ JVB_AUTH_PASSWORD }}
JIGASI_XMPP_PASSWORD={{ JIGASI_XMPP_PASSWORD }}
JIBRI_RECORDER_PASSWORD={{ JIBRI_RECORDER_PASSWORD }}
JIBRI_XMPP_PASSWORD={{ JIBRI_XMPP_PASSWORD }}
CONFIG={{ config_path }}
HTTP_PORT=80
HTTPS_PORT=443
TZ=UTC
PUBLIC_URL=https://redacted.com
DOCKER_HOST_ADDRESS={{ ansible_host }}
ENABLE_PREJOIN_PAGE=1
ENABLE_NOISY_MIC_DETECTION=0

RESOLUTION=720
RESOLUTION_WIDTH=3840

ENABLE_LETSENCRYPT=1
LETSENCRYPT_DOMAIN={{ service_host }}
LETSENCRYPT_EMAIL={{ letsencrypt_email }}
ETHERPAD_TITLE="Video Chat"
ETHERPAD_DEFAULT_PAD_TEXT="Welcome to Web Chat!\n\n"
ETHERPAD_SKIN_NAME="colibris"
ETHERPAD_SKIN_VARIANTS="super-light-toolbar super-light-editor light-background full-width-editor"

XMPP_DOMAIN=meet.jitsi
XMPP_SERVER=xmpp.meet.jitsi
XMPP_BOSH_URL_BASE=http://xmpp.meet.jitsi:5280
XMPP_AUTH_DOMAIN=auth.meet.jitsi
XMPP_MUC_DOMAIN=muc.meet.jitsi
XMPP_INTERNAL_MUC_DOMAIN=internal-muc.meet.jitsi
XMPP_GUEST_DOMAIN=guest.meet.jitsi
XMPP_MODULES=
XMPP_MUC_MODULES=
XMPP_INTERNAL_MUC_MODULES=
JVB_BREWERY_MUC=jvbbrewery
JVB_AUTH_USER=jvb
JVB_STUN_SERVERS=meet-jit-si-turnrelay.jitsi.net:443
JVB_PORT=10000
JVB_TCP_HARVESTER_DISABLED=true
JVB_TCP_PORT=4443
JVB_TCP_MAPPED_PORT=4443
JICOFO_AUTH_USER=focus
JIGASI_XMPP_USER=jigasi
JIGASI_BREWERY_MUC=jigasibrewery
JIGASI_PORT_MIN=20000
JIGASI_PORT_MAX=20050

ENABLE_RECORDING=1
XMPP_RECORDER_DOMAIN=recorder.meet.jitsi
JIBRI_RECORDER_USER=recorder
JIBRI_RECORDING_DIR=/config/recordings
JIBRI_FINALIZE_RECORDING_SCRIPT_PATH=/config/finalize.sh
JIBRI_XMPP_USER=jibri
JIBRI_BREWERY_MUC=jibribrewery
JIBRI_PENDING_TIMEOUT=90
JIBRI_STRIP_DOMAIN_JID=muc
JIBRI_LOGS_DIR=/config/logs

ENABLE_HTTP_REDIRECT=1
RESTART_POLICY=unless-stopped

With stable-5390-3 (older version):

Changes in xorg-video-dummy.conf: Replaced: Virtual 1280 720 with Virtual 1920 1080

Replacements in the ffmpeg command arguments used by jibri:

ARGS=$(echo "$ARGS" | sed 's/-crf 25/-crf 18/' | sed 's/-preset veryfast/-preset ultrafast/' | sed 's/-y/-y -threads 0/' | sed 's/-tune zerolatency //' | sed 's/-profile:v main -level 3.1/-profile:v high -level 5.1/' | sed 's/-s 1280x720/-s 1920x1080/' )

With stable-5765-1 (newer version):

Changes in xorg-video-dummy.conf: None

Replacements in the ffmpeg command arguments used by jibri:

ARGS=$(echo "$ARGS" | sed 's/-crf 25/-crf 18/' | sed 's/-preset veryfast/-preset ultrafast/' | sed 's/-y/-y -threads 0/' | sed 's/-tune zerolatency //' | sed 's/-profile:v main -level 3.1/-profile:v high -level 5.1/' )

(difference is the removal of | sed 's/-s 1280x720/-s 1920x1080/')

Workaround

Just in case someone else also has this issue, we were able to find a “workaround” to still be able to use the new version to record. We changed the replacements in the ffmpeg command used by jibri to record losslessly:

ARGS=$(echo "$ARGS" | sed 's/-crf 25/-crf 0/' | sed 's/-preset veryfast/-preset ultrafast/' | sed 's/-y/-y -threads 0/' | sed 's/-tune zerolatency //'  | sed 's/-profile:v main -level 3.1 //' )

And in finalize.sh we run the ffmpeg command to compress the output file after the recording is done.

The disadvantages of this workaround are however, that we have to pay very close attention not to run out of storage space, considering the lossless files take up a lot more space, and the compression in finalize.sh takes around two thirds of the time of the recording to process, so it takes a while after a recording is done until it’s available.

[prayagsingh]

How about this: make a PR here and let’s take a look at how the changes look like, then we can decide.

sure. I’ll start working on it.

[saghul]

Why would it break their setup?

How about this: make a PR here and let’s take a look at how the changes look like, then we can decide.