Ffmpeg eats all the memory and crash within a minute - recording or streaming

Description

On jitsi when I start a recording or a streaming session, in less than a minute the recording/stream will stop and my whole server become slow and unresponsive.

With top, I could pin the culprit: ffmpeg. It eats away all the memory very quickly. In less than a minute my 8GB are filled.

You can find attached the log of jibri when I tried a streaming session. Nothing stands out to me. I stopped the streaming after 15 seconds and ffmpeg was already at 40% memory.

Also if I stop completely prosody, jicofo, jvb and jibri and if I log as a jibri user and starts ffmpeg by myself, using the command I found in log.0.txt, I get the same issue, the CPU shoot to 150% and the memory keeps growing. I have to kill ffmpeg before it saturates the memory.

ffmpeg -y -v info -f x11grab -draw_mouse 0 -r 30 -s 1280x720 -thread_queue_size 4096 -i :0.0+0,0 -f alsa -thread_queue_size 4096 -i plug:bsnoop -acodec aac -strict -2 -ar 44100 -c:v libx264 -preset veryfast -maxrate 2976k -bufsize 5952k -pix_fmt yuv420p -r 30 -crf 25 -g 60 -tune zerolatency -f flv rtmp://a.rtmp.youtube.com/live2/aaa

If I remove every parameters related to sound in this ffmpeg command line, so removing -f alsa -thread_queue_size 4096 -i plug:cloop -acodec aac, then the memory saturation issue goes away. Memory usage is stable. So It clearly seems to be related to the sound. How can I debug this kind of issue ?

Possible Solution

---

Steps to reproduce

---

Environment details

Ubuntu 16, followed the instruction on github

lsmod | grep snd_aloop
snd_aloop              24576  0
snd_pcm               106496  1 snd_aloop
snd                    81920  3 snd_aloop,snd_timer,snd_pcm

jibri@JibriTestSrv:/root$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: Loopback [Loopback], device 0: Loopback PCM [Loopback PCM]
  Subdevices: 8/8
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
  Subdevice #7: subdevice #7
card 0: Loopback [Loopback], device 1: Loopback PCM [Loopback PCM]
  Subdevices: 8/8
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
  Subdevice #7: subdevice #7

browser.0.txt log.0.txt ffmpeg.0.txt asoundrc.txt

[ec-blaster]

Our only real solution was to increase vCPU’s number. Increasing from 4 to 8 CPU’s was the fix we used last time. It seems as if ffmpeg begins eating memory when the CPU’s are not giving enough power to it…

[aaronkvanmeerten]

The main rationale for why we use chrome as a compositor for recording with Jibri is that it’s the best method we have to go from multiple WebRTC streams of audio and video to a single video with one audio and video stream. Any recorder will need to composite the videos, chose the active speaker, mix the audio, etc. Chrome happens to already do this, and the jitsi-meet client is custom-built for this job, so re-using it for recording has been the best method without needing to support a whole separate client. Would it be possible to do in a separate client? Absolutely, but then said client would need to be regularly updated when new features of Jitsi or Chrome dropped. So, it’s a reasonable question to ask why it’s designed this way, but the short answer is that with a small team, this was our best answer.