[youtube] YouTube serving erroneous DASH Manifest VP9 formats

Checklist

• I’m reporting a broken site support
• I’ve verified that I’m running yt-dlp version 2021.07.07
• I’ve checked that all provided URLs are alive and playable in a browser (but with condition, see below)
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar issues including closed ones

Verbose log

$ yt-dlp V_h3Z40AAtw -F
[youtube] V_h3Z40AAtw: Downloading webpage
[youtube] V_h3Z40AAtw: Downloading MPD manifest
[info] Available formats for V_h3Z40AAtw:
ID  EXT  RESOLUTION FPS |  FILESIZE   TBR PROTO | VCODEC        VBR ACODEC     ABR  ASR    NOTE
--- ---- ---------- --- - --------- ----- ----- - ----------- ----- --------- ---- ------- ---------------------------------------
139 m4a  audio only     |  1.52MiB    50k dash  |                   mp4a.40.5  50k 22050Hz DASH audio, m4a_dash, 22050Hz
140 m4a  audio only     |  4.02MiB   129k https |                   mp4a.40.2 129k 44100Hz audio_quality_medium, m4a_dash, 44100Hz
160 mp4  256x144    30  |            108k dash  | avc1.4d400b  108k                        DASH video, mp4_dash
278 webm 256x144    30  |             95k dash  | vp9           95k                        DASH video, webm_dash
133 mp4  426x240    30  |            242k dash  | avc1.4d400c  242k                        DASH video, mp4_dash
242 webm 426x240    30  |            220k dash  | vp9          220k                        DASH video, webm_dash
134 mp4  640x360    30  |  19.25MiB  620k https | avc1.4d401e  620k                        360p, mp4_dash
18  mp4  640x360    30  |  22.68MiB  730k https | avc1.42001E  730k mp4a.40.2   0k 44100Hz 360p, 44100Hz
243 webm 640x360    30  |            405k dash  | vp9          405k                        DASH video, webm_dash
135 mp4  854x480    30  |           1155k dash  | avc1.4d400c 1155k                        DASH video, mp4_dash
244 webm 854x480    30  |            752k dash  | vp9          752k                        DASH video, webm_dash
136 mp4  1280x720   30  |  69.87MiB 2251k https | avc1.4d401f 2251k                        720p, mp4_dash
22  mp4  1280x720   30  |           2380k https | avc1.64001F 2380k mp4a.40.2   0k 44100Hz 720p, 44100Hz
247 webm 1280x720   30  |           1505k dash  | vp9         1505k                        DASH video, webm_dash
248 webm 1920x1080  30  |           2646k dash  | vp9         2646k                        DASH video, webm_dash

$ yt-dlp -v V_h3Z40AAtw
[debug] Command-line config: ['-v', 'V_h3Z40AAtw']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] yt-dlp version 2021.07.13.1626134551 (zip)
[debug] Python version 3.9.6 (CPython 64bit) - Linux-5.8.0-41-generic-x86_64-with-glibc2.32
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1, rtmpdump 2.4
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: V_h3Z40AAtw
[youtube] V_h3Z40AAtw: Downloading webpage
[youtube] [debug] Fetching webpage from https://www.youtube.com/watch?v=V_h3Z40AAtw&bpctr=9999999999&has_verified=1
[youtube] V_h3Z40AAtw: Downloading MPD manifest
[youtube] [debug] Fetching webpage from https://manifest.googlevideo.com/api/manifest/dash/expire/1626251414/ei/NkzuYLz-Ap-0s8IPlf2YyAo/ip/2001%3A19f0%3A7001%3A13a1%3A5400%3A3ff%3Afe11%3A205f/id/57f877678d0002dc/source/youtube/requiressl/yes/playback_host/r2---sn-oguelne7.googlevideo.com/mh/t1/mm/31%2C29/mn/sn-oguelne7%2Csn-oguesnzz/ms/au%2Crdu/mv/m/mvi/2/pl/55/tx/24027688/txs/24027687%2C24027688%2C24027689%2C24027690/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/initcwndbps/1267500/vprv/1/mt/1626229496/fvip/2/keepalive/yes/fexp/24001373%2C24007246/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Ctx%2Ctxs%2Chfr%2Cas%2Cvprv%2Citag/sig/AOq0QJ8wRAIgel_8rJx7O1ChqaQTDiBI5cysHmZ_4uCmgCWN_kPxy8cCIDfgAFVDYl5WO7a1gLifSDw6vBfjELblxSgkOodOm1am/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps/lsig/AG3C_xAwRAIgJVqT1xhW2KIhXVIj6cJRomDQ7-UOvq8yyC_J5r7ksfMCIBG0VgIJSIlNe49rl6ty6WA_DuH2AhKJvLOpq8fUBojv
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] V_h3Z40AAtw: Downloading 1 format(s): 248+140
[debug] locking youtube_V_h3Z40AAtw.lock
[debug] Invoking downloader on "https://manifest.googlevideo.com/api/manifest/dash/expire/1626251414/ei/NkzuYLz-Ap-0s8IPlf2YyAo/ip/2001%3A19f0%3A7001%3A13a1%3A5400%3A3ff%3Afe11%3A205f/id/57f877678d0002dc/source/youtube/requiressl/yes/playback_host/r2---sn-oguelne7.googlevideo.com/mh/t1/mm/31%2C29/mn/sn-oguelne7%2Csn-oguesnzz/ms/au%2Crdu/mv/m/mvi/2/pl/55/tx/24027688/txs/24027687%2C24027688%2C24027689%2C24027690/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/initcwndbps/1267500/vprv/1/mt/1626229496/fvip/2/keepalive/yes/fexp/24001373%2C24007246/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Ctx%2Ctxs%2Chfr%2Cas%2Cvprv%2Citag/sig/AOq0QJ8wRAIgel_8rJx7O1ChqaQTDiBI5cysHmZ_4uCmgCWN_kPxy8cCIDfgAFVDYl5WO7a1gLifSDw6vBfjELblxSgkOodOm1am/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps/lsig/AG3C_xAwRAIgJVqT1xhW2KIhXVIj6cJRomDQ7-UOvq8yyC_J5r7ksfMCIBG0VgIJSIlNe49rl6ty6WA_DuH2AhKJvLOpq8fUBojv"
[dashsegments] Total fragments: 50
[download] Destination: Sweet Candy ②-V_h3Z40AAtw.f248.webm
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 1 (attempt 1 of 10) ...
^C[debug] unlocking youtube_V_h3Z40AAtw.lock

ERROR: Interrupted by user

Description

link to video

The video itself plays on browser, and doesn’t have 1080p as you can see it.

But yt-dlp (and youtube-dl) reports 1080p format, which possibly doesn’t exist on the server. (format 248 on the video fails to download all segments.)

Resolutions shown in webpage here:

Edit: Tested web and android clients, some locations (JP, Vultr JP, OCI US?), with cookies or not, but all of them has this “ghosty” format

[pukkandan]

@fastily that is what --check-formats does

[coletdjnz]

I did some testing on this, though I still have a limited sample.

TL;DR: --check-formats should avoid this issue, though it can can slow things down in some cases 😕

For the failing fragments, I’ve seen so far:

• fragments failing throughout the download - these seem to be resolved fine by the retry mechanism. Re-downloading the file again, there was no failing fragments and video download was faster. I’m guessing this is probably due to server side cache, failed fragments prob due to downloading faster than backend could keep up. Nothing abnormal here afaik.

• fragment 1 failing to download every time (what the main issue is). --check-formats solves this.

• What I have yet to see: fragments (!= 1) failing to download every time. --check-formats will not solve this, if someone finds a video (VP9 DASH) where this happens, let us know

#1102 also mentioned how the video in this issue took a while for the fragment 1 retries to exhaust. I’ve noticed the following patterns:

• fragment 1 retries are slow, for every format re-download.
• fragment 1 retries are fast, for every format re-download.
• fragment 1 retries are very slow for the first time we download, then running again it is fast.