Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[Support]: ffmpeg crashing

See original GitHub issue

Describe the problem you are having

I’ve just upgraded one of my cameras to a 4k Hikvison model however it appears that ffmpeg is crashing out after a short period as can be seen in the logs below. This never recovers and can only be solved by restarting Frigate.

Version

0.9.4-26AE608

Frigate config file

Drive:
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 600
    ffmpeg:
      hwaccel_args:
        - -hwaccel
        - qsv
        - -qsv_device
        - /dev/dri/renderD128
        - -c:v
        - h264_qsv
      input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1
      output_args:
        detect: -vf hwdownload,format=nv12 -f rawvideo -pix_fmt yuvj420p
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
      inputs:
        - path: rtsp://***:****@10.0.0.110:554/Streaming/Channels/101/
          roles:
            - detect
            - clips
    objects:
      track:
        - person
        - car
        - cat
      filters:
        car:
          min_score: 0.2
          threshold: 0.5
          mask:
            - 2926,621,3016,867,3840,2160,3840,0,0,0,0,2160,3840,2160,2991,839,2012,649,2311,490,2308,265,2944,343
    motion:
      mask:
        - 2095,173,2938,323,3071,378,3840,858,3840,0,1701,0,0,0,0,441,1114,339,1118,217
    rtmp:
      enabled: true
    snapshots:
      required_zones:
        - entrance
        - everywhere
      enabled: true
      timestamp: true
      bounding_box: true
      crop: true
      quality: 100
      height: 600
      retain:
        default: 10
    detect:
      width: 3840
      height: 2160
    record:
      enabled: True
      retain_days: 0
      events:
        required_zones:
          - entrance
          - everywhere
        retain:
          default: 10
    zones:
      entrance:
        coordinates: 1992,642,2997,835,2941,602,2953,284,2355,209,2323,437
        objects:
          - car
      everywhere:
        coordinates: 2359,331,2953,441,2949,0,3840,543,3840,2160,0,2160,0,0,1040,0,1055,331,1158,299,1142,173,2339,209
        objects:
          - person
          - cat

Relevant log output

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-01-27 11:33:33] frigate.app INFO : Starting Frigate (0.9.4-26ae608)
Starting migrations
[2022-01-27 11:33:33] peewee_migrate INFO : Starting migrations
There is nothing to migrate
[2022-01-27 11:33:33] peewee_migrate INFO : There is nothing to migrate
[2022-01-27 11:33:33] frigate.mqtt INFO : MQTT connected
[2022-01-27 11:33:33] detector.coral_pci INFO : Starting detection process: 233
[2022-01-27 11:33:33] frigate.app INFO : Output process started: 235
[2022-01-27 11:33:33] ws4py INFO : Using epoll
[2022-01-27 11:33:33] frigate.edgetpu INFO : Attempting to load TPU as pci
[2022-01-27 11:33:33] frigate.app INFO : Camera processor started for Drive: 239
[2022-01-27 11:33:33] frigate.edgetpu INFO : TPU found
[2022-01-27 11:33:33] frigate.app INFO : Camera processor started for Garden: 245
[2022-01-27 11:33:33] frigate.app INFO : Capture process started for Drive: 253
[2022-01-27 11:33:33] frigate.app INFO : Capture process started for Garden: 254
[2022-01-27 11:33:33] ws4py INFO : Using epoll
[2022-01-27 11:33:43] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:55456]
[2022-01-27 11:34:43] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:55456]
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:40:56] frigate.video INFO : Drive: ffmpeg process is not running. exiting capture thread...
[2022-01-27 11:41:03] watchdog.Drive ERROR : FFMPEG process crashed unexpectedly for Drive.

[2022-01-27 11:41:03] watchdog.Drive ERROR : The following ffmpeg logs include the last 100 lines prior to exit.

[2022-01-27 11:41:03] watchdog.Drive ERROR : You may have invalid args defined for this camera.

[2022-01-27 11:41:03] ffmpeg.Drive.detect ERROR : [swscaler @ 0x55f3bc43f400] deprecated pixel format used, make sure you did set range correctly

[2022-01-27 11:41:03] ffmpeg.Drive.detect ERROR : av_interleaved_write_frame(): Connection reset by peer

[2022-01-27 11:41:03] ffmpeg.Drive.detect ERROR : [flv @ 0x55f3bc088a80] Failed to update header with correct duration.

[2022-01-27 11:41:03] ffmpeg.Drive.detect ERROR : [flv @ 0x55f3bc088a80] Failed to update header with correct filesize.

[2022-01-27 11:41:03] ffmpeg.Drive.detect ERROR : Error writing trailer of rtmp://127.0.0.1/live/Drive: Connection reset by peer

[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
[2022-01-27 11:41:20] frigate.video INFO : Drive: ffmpeg process is not running. exiting capture thread...
[2022-01-27 11:41:23] watchdog.Drive ERROR : FFMPEG process crashed unexpectedly for Drive.

[2022-01-27 11:41:23] watchdog.Drive ERROR : The following ffmpeg logs include the last 100 lines prior to exit.

[2022-01-27 11:41:23] watchdog.Drive ERROR : You may have invalid args defined for this camera.

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : [rtsp @ 0x55e04e9fef00] DTS discontinuity in stream 1: packet 3 with DTS 72468809565588, packet 4 with DTS 72468809572656

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : [segment @ 0x55e04ea25200] Non-monotonous DTS in output stream 0:1; previous: 14822, current: 14820; changing to 14823. This may result in incorrect timestamps in the output file.

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : [swscaler @ 0x55e04f295440] deprecated pixel format used, make sure you did set range correctly

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : av_interleaved_write_frame(): Connection reset by peer

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : [flv @ 0x55e04ea21cc0] Failed to update header with correct duration.

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : [flv @ 0x55e04ea21cc0] Failed to update header with correct filesize.

[2022-01-27 11:41:23] ffmpeg.Drive.detect ERROR : Error writing trailer of rtmp://127.0.0.1/live/Drive: Connection reset by peer

[2022-01-27 12:11:18] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:34024]
[2022-01-27 12:12:04] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:34024]
[2022-01-27 12:12:16] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:34984]
[2022-01-27 12:13:43] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:34984]
[2022-01-27 12:16:40] ws4py INFO : Managing websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:58516]
[2022-01-27 12:16:42] ws4py INFO : Terminating websocket [Local => 127.0.0.1:8082 | Remote => 127.0.0.1:58516]
[2022-01-27 13:15:28] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:46596]
[2022-01-27 13:16:43] ws4py INFO : Terminating websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:46596]
Fatal Python error: Bus error


Current thread 0x0000148991f31700 (most recent call first):
File "/opt/frigate/frigate/video.py", line 150 in capture_frames
File "/opt/frigate/frigate/video.py", line 290 in run
File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x0000148994534700 (most recent call first):
File "/usr/lib/python3.8/threading.py", line 302 in wait
File "/usr/lib/python3.8/multiprocessing/queues.py", line 227 in _feed
File "/usr/lib/python3.8/threading.py", line 870 in run
File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x0000148991d30700 (most recent call first):
File "/usr/lib/python3.8/threading.py", line 302 in wait
File "/usr/lib/python3.8/multiprocessing/queues.py", line 227 in _feed
File "/usr/lib/python3.8/threading.py", line 870 in run
File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x0000148994132700 (most recent call first):
File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 261 in wait
File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 349 in wait
File "/opt/frigate/frigate/video.py", line 215 in run
File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x0000148994333700 (most recent call first):
File "/opt/frigate/frigate/log.py", line 65 in run
File "/usr/lib/python3.8/threading.py", line 932 in _bootstrap_inner
File "/usr/lib/python3.8/threading.py", line 890 in _bootstrap

Thread 0x00001489a5550740 (most recent call first):
File "/usr/lib/python3.8/threading.py", line 1027 in _wait_for_tstate_lock
File "/usr/lib/python3.8/threading.py", line 1011 in join
File "/opt/frigate/frigate/video.py", line 321 in capture_camera
File "/usr/lib/python3.8/multiprocessing/process.py", line 108 in run
File "/usr/lib/python3.8/multiprocessing/process.py", line 315 in _bootstrap
File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 75 in _launch
File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 19 in __init__
File "/usr/lib/python3.8/multiprocessing/context.py", line 277 in _Popen
File "/usr/lib/python3.8/multiprocessing/context.py", line 224 in _Popen
File "/usr/lib/python3.8/multiprocessing/process.py", line 121 in start
File "/opt/frigate/frigate/app.py", line 255 in start_camera_capture_processes
File "/opt/frigate/frigate/app.py", line 330 in start
File "/opt/frigate/frigate/__main__.py", line 17 in <module>
File "/usr/lib/python3.8/runpy.py", line 87 in _run_code
File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main
[2022-01-27 17:24:58] ws4py INFO : Managing websocket [Local => 127.0.0.1:5002 | Remote => 127.0.0.1:58926]

FFprobe output from your camera

ffprobe version 4.3.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-17ubuntu1~20.04)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libfreetype --enable-libvidstab --enable-libmfx --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-vaapi --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, rtsp, from 'rtsp://***:****@10.0.0.110:554/Streaming/Channels/101/':
  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264, yuvj420p(pc, progressive), 3840x2160 [SAR 1:1 DAR 16:9], 20 fps, 25 tbr, 90k tbn, 40 tbc
    Stream #0:1: Audio: aac, 44100 Hz, mono, fltp

Frigate stats

{"Drive":{"camera_fps":5.1,"capture_pid":253,"detection_fps":0.5,"pid":239,"process_fps":5.0,"skipped_fps":0.0},"Garden":{"camera_fps":5.1,"capture_pid":254,"detection_fps":0.0,"pid":245,"process_fps":5.1,"skipped_fps":0.0},"detection_fps":0.5,"detectors":{"coral_pci":{"detection_start":0.0,"inference_speed":9.79,"pid":233}},"service":{"storage":{"/dev/shm":{"free":40.5,"mount_type":"tmpfs","total":67.1,"used":26.6},"/media/frigate/clips":{"free":396711.5,"mount_type":"fuse.shfs","total":480112.7,"used":79918.4},"/media/frigate/recordings":{"free":396711.5,"mount_type":"fuse.shfs","total":480112.7,"used":79918.4},"/tmp/cache":{"free":7256.0,"mount_type":"rootfs","total":8243.1,"used":987.1}},"uptime":21495,"version":"0.9.4-26ae608"}}

Operating system

UNRAID

Install method

Docker CLI

Coral version

M.2

Network connection

Wired

Camera make and model

Hikvision DS-2CD2387G2-LU

Any other information that may be helpful

No response

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:20 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
blakeblackshearcommented, Jan 28, 2022

Any time you see flv messages in the ffmpeg logs its related to rtmp. Try disabling rtmp for this camera since high bitrates are known to cause failures there.

0reactions
tungmeistercommented, Jan 29, 2022

@blakeblackshear Thanks, disabling rtmp has resolved the issue.

Read more comments on GitHub >

github_iconTop Results From Across the Web

[Support]: Ffmpeg continuously crashing · Issue #2858 - GitHub
Describe the problem you are having Hi. I am struggling to have frigate set up for 24/7 recording from two cameras on a...
Read more >
9048 (FFmpeg crashing)
This seems to be a problem with FFmpeg. 1. It shouldn't crash. 2. It probably should be able to handle the videos. I...
Read more >
3974 (ffplay crashing with some resolutions in linux terminal)
im new to linux. im trying to play a video in a terminal (not in x) with ffplay. the os is debian wheezy...
Read more >
5787 (OS X: Not possible to use videotoolbox in ffplay)
Btw using h264_vda crashes ffplay (vda is no longer supported by OS X 10.11. Oldest first ... -hwaccel videotoolbox only works in ffmpeg,...
Read more >
6768 (no png output allowed) - FFmpeg Wiki
I want to scale a png file into another png file, and ffmpeg crashes when writing the file. Even a simple copy is...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

[Support]: Coral Dev Board 1GB recommended?

Next page

[Support]: Instructions for adding a nginx reverse proxy with ssl+auth