Substreams that stretch the source aspect ratio should output clips of the source aspect ratio

Describe the bug I have a couple of cameras that are running 1920x1080 (or larger, 16:9 aspect ratio). Their substreams output 640x480 of the same view, but that’s a 4:3 aspect ratio. So when a snapshot is created due to detect rule on the substream, the snapshot still ends up stretched because the snapshot is of the substream 640x480.

Example output image:

What I would prefer to see is the image converted to the aspect ratio of the main source stream:

Because the current configuration wants you to set the width and height properties on the main part of the camera but not per ffmpeg input, there isn’t any way to ensure that this happens reliably.

Version of frigate What version are you using? 0.8.0-beta2

Config file Include your full config file wrapped in triple back ticks.

detectors:
  coral:
    type: edgetpu
    device: usb
mqtt:
  host: core-mosquitto.local.hass.io
  user: mqtt
  password: unhook-kick-flabby
ffmpeg:
  hwaccel_args:
    - '-hwaccel'
    - qsv
    - '-qsv_device'
    - /dev/dri/renderD128
cameras:
  front:
    ffmpeg:
      inputs:
        - path: rtsp://<REDACTED>/videoMain
          roles:
            - record
            - clips
        - path: rtsp://<REDACTED>/videoSub
          roles:
            - rtmp
            - detect
    fps: 5
    height: 480
    width: 640
# other cameras redacted
objects:
  track:
    - person
    - dog
    - cat
    - deer
    - bicycle

Logs n/a

Frigate debug stats

{"detection_fps":0.0,"detectors":{"coral":{"detection_start":0.0,"inference_speed":9.17,"pid":51}},"front":{"camera_fps":5.0,"capture_pid":62,"detection_fps":0.0,"pid":56,"process_fps":5.0,"skipped_fps":0.0},"kitchen":{"camera_fps":5.0,"capture_pid":65,"detection_fps":0.0,"pid":58,"process_fps":5.0,"skipped_fps":0.0},"living":{"camera_fps":5.0,"capture_pid":70,"detection_fps":0.0,"pid":60,"process_fps":5.0,"skipped_fps":0.0},"turret":{"camera_fps":5.1,"capture_pid":61,"detection_fps":0.0,"pid":54,"process_fps":5.1,"skipped_fps":0.0}}

FFprobe from your camera

Run the following command and paste output below

$ ffprobe rtsp://<REDACTED>/videoMain
Input #0, rtsp, from 'rtsp://<REDACTED>/videoMain':
  Metadata:
    title           : IP Camera Video
    comment         : videoMain
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264 (Constrained Baseline), yuvj420p(pc, progressive), 2304x1536, 90k tbr, 90k tbn, 180k tbc
    Stream #0:1: Audio: pcm_mulaw, 8000 Hz, 1 channels, s16, 64 kb/s

# -------

$ ffprobe rtsp://<REDACTED>/videoSub
Input #0, rtsp, from 'rtsp://<REDACTED>/videoSub':
  Metadata:
    title           : IP Camera Video
    comment         : videoSub
  Duration: N/A, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264 (Constrained Baseline), yuvj420p(pc, progressive), 640x480, 20.17 tbr, 90k tbn, 180k tbc
    Stream #0:1: Audio: pcm_mulaw, 8000 Hz, 1 channels, s16, 64 kb/s

Screenshots See bug

Computer Hardware

• OS: Home Assistant
• Virtualization: HA Addon (docker)
• Coral Version: USB
• Network Setup: WiFi camera

Camera Info:

• Manufacturer: Foscam, Amcrest
• Model: Foscam VZ4, Amcrest IP3M-943W
• Resolution: 640x480 for substream, 2k 3MP for main stream
• FPS: 5

Additional context Add any other context about the problem here.

[blakeblackshear]

For RTMP, I would expect the resolution to show 640x480. We are trying to get the RTMP endpoint to behave the same way it does when you connect directly to the camera with VLC. I am sure it shows 640x480 there too, but there is some extra metadata that tells it to stretch the image. If you have to resize the image, you will have to use CPU to re-encode the video again and we are trying to avoid that.

Let’s see if this config fixes the detect stream (mjpeg in frigate).

cameras:
  turret:
    ffmpeg:
      inputs:
        - path: 'rtsp://REDACTED/videoMain'
          roles:
            - record
            - clips
        - path: 'rtsp://REDACTED/videoSub'
          roles:
            - rtmp
            - detect
      output_args:
        detect: -vf scale=720:480 -f rawvideo -pix_fmt yuv420p
        rtmp: -aspect 720:480 -c copy -f flv
    fps: 5
    save_clips:
      enabled: false
    objects:
      track: []
    height: 480
    width: 720

[scottlamb]

+1. For reference, these streams are called anamorphic. All of my cameras are 16:9 and have anamorphic 704x480 sub streams. My Dahua ones declare themselves as such properly in the H.264 VUI parameters. (I don’t remember off the top of my head if there’s a way to get ffprobe to tell you that.) My older Hikvision ones don’t, and so look wrong when I view them in VLC and the like (directly, so not due to any problem with frigate). So some autodetection + knob for manual override might be best.

[edit: Looks like I was partially mistaken about my Dahua cameras doing the right thing. My Dahua IPC-HDW5231R-Z declares its aspect ratio properly (last I checked; it’s not plugged in right now). My brand-new Dahua IPC-HDW5442TP-ZE cameras don’t.]