[youtube] premature download termination with --live-from-start

Checklist

• I’m reporting a bug unrelated to a specific site
• I’ve verified that I’m running yt-dlp version 2022.02.04. (update instructions)
• I’ve checked that all provided URLs are alive and playable in a browser
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I’ve read the guidelines for opening an issue

Description

(URLs are no longer playable.) Unfortunately I did not have an HLS download, but no playback issue was noticed in the browser.

Generally speaking, it seems possible for --live-from-start to prematurely terminating the download of each stream, resulting in a truncated or corrupted file.

First download, video lasted longer than the audio, 01:19:23.000 vs 00:52:58.998 (I believe the video is full-length). On playback in mpv, the following warning is logged, at the 53 minute mark:

[lavf] Too many packets in the demuxer packet queues:
[lavf]   video/0: 16054 packets, 157294896 bytes
[lavf]   audio/1: 0 packets, 0 bytes

Seeking beyond that point is considerably slower.

Second download, video and audio downloads terminated even earlier, at 00:43:38:00. The fragments after that point were never downloaded.

I suspect the reworking of --live-from-start may be at fault. It may also be another issue somewhere in the downloader as well, I do not know.

I do know, however, that based on the logs the interruption seems tied to a spontaneous webpage download in the middle of downloading the fragments. Presumably when this happens the stream download is permanently interrupted.

Verbose log

# First download
date; ytdlp -v --write-thumbnail --write-description --write-info-json  --live-from-start --wait-for-video=2-3 -N4 J5c7fJOMeXk; date
Sat Feb 26 02:00:06 AM EST 2022
[debug] Command-line config: ['-v', '--write-thumbnail', '--write-description', '--write-info-json', '--live-from-start', '--wait-for-video=2-3', '-N4', 'J5c7fJOMeXk']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.02.04 [c1653e9ef] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: f0734e119
[debug] Python version 3.10.2 (CPython 64bit) - Linux-5.15.25-1-lts-x86_64-with-glibc2.35
[debug] exe versions: ffmpeg 5.0 (setts), ffprobe 5.0, rtmpdump 2.4
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: J5c7fJOMeXk
[youtube] J5c7fJOMeXk: Downloading webpage
[youtube] J5c7fJOMeXk: Downloading android player API JSON
WARNING: [youtube] This live event will begin in 15 minutes.
[wait] Waiting for 00:00:03 - Press Ctrl+C to try now
[wait] Wait period ended; Re-extracting data                 
[debug] [youtube] Extracting URL: J5c7fJOMeXk
[youtube] J5c7fJOMeXk: Downloading webpage
[youtube] J5c7fJOMeXk: Downloading android player API JSON
WARNING: [youtube] This live event will begin in 15 minutes.
[wait] Waiting for 00:00:03 - Press Ctrl+C to try now
[wait] Wait period ended; Re-extracting data                 
<cut...>
[youtube] J5c7fJOMeXk: Downloading webpage
[youtube] J5c7fJOMeXk: Downloading android player API JSON
WARNING: [youtube] This live event will begin in a few moments.
WARNING: Release time of video is not known
[wait] Waiting for 00:00:02 - Press Ctrl+C to try now
[wait] Wait period ended; Re-extracting data                 
[debug] [youtube] Extracting URL: J5c7fJOMeXk
[youtube] J5c7fJOMeXk: Downloading webpage
[youtube] J5c7fJOMeXk: Downloading android player API JSON
WARNING: [youtube] This live event will begin in a few moments.
WARNING: Release time of video is not known
[wait] Waiting for 00:00:02 - Press Ctrl+C to try now
[wait] Wait period ended; Re-extracting data                 
[debug] [youtube] Extracting URL: J5c7fJOMeXk
[youtube] J5c7fJOMeXk: Downloading webpage
[youtube] J5c7fJOMeXk: Downloading android player API JSON
[youtube] J5c7fJOMeXk: Downloading MPD manifest
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, codec:vp9.2, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Default format spec: best/bestvideo+bestaudio
[info] J5c7fJOMeXk: Downloading 1 format(s): 299+140
[info] Writing video description to: 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].description
[info] Downloading video thumbnail 46 ...
[info] Writing video thumbnail 46 to: 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].webp
[info] Writing video metadata as JSON to: 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].info.json
[debug] Invoking downloader on "https://manifest.googlevideo.com/api/manifest/dash/expire/1645881617/ei/sdQZYtmOJeOA2LYPtcufyA8/ip/<IP_ADDR>/id/J5c7fJOMeXk.1/source/yt_live_broadcast/requiressl/yes/hfr/1/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/vprv/1/pacing/0/itag_bl/376%2C377%2C384%2C385%2C612%2C613%2C617%2C619%2C623%2C628%2C655%2C656%2C660%2C662%2C666%2C671/keepalive/yes/fexp/24001373%2C24007246/itag/0/playlist_type/LIVE/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRAIgfR6yUNDocKcpqwWm3rXlVaUPRJzTOQAGbQC4M880p44CIBEZWRSHiK-oMYi-c5d4BsaJnZd48LBQU1DZLJNPB83D", "https://manifest.googlevideo.com/api/manifest/dash/expire/1645881617/ei/sdQZYtmOJeOA2LYPtcufyA8/ip/<IP_ADDR>/id/J5c7fJOMeXk.1/source/yt_live_broadcast/requiressl/yes/hfr/1/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/vprv/1/pacing/0/itag_bl/376%2C377%2C384%2C385%2C612%2C613%2C617%2C619%2C623%2C628%2C655%2C656%2C660%2C662%2C666%2C671/keepalive/yes/fexp/24001373%2C24007246/itag/0/playlist_type/LIVE/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRAIgfR6yUNDocKcpqwWm3rXlVaUPRJzTOQAGbQC4M880p44CIBEZWRSHiK-oMYi-c5d4BsaJnZd48LBQU1DZLJNPB83D"
[dashsegments] Total fragments: unknown (live)
[download] Destination: 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f299.mp4
[dashsegments] Total fragments: unknown (live)
[download] Destination: 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f140.mp4
WARNING: The download speed shown is only of one thread. This is a known issue and patches are welcome
WARNING: The download speed shown is only of one thread. This is a known issue and patches are welcome
[download] 1.60GiB at  1.62MiB/s (52:53) (frag 3177)[youtube] J5c7fJOMeXk: Downloading webpage
[download] 1.60GiB at  2.59MiB/s (52:53) (frag 3179)
[download] 65.68MiB at 147.60KiB/s (52:53) (frag 3179)[youtube] J5c7fJOMeXk: Downloading android player API JSON
[youtube] J5c7fJOMeXk: Downloading MPD manifest
[download] 2.43GiB at  2.59MiB/s (01:19:19) (frag 4763)[download] Got server HTTP error: The read operation timed out. Retrying (attempt 1 of 10) ...
[youtube] J5c7fJOMeXk: Downloading webpage
[youtube] J5c7fJOMeXk: Downloading android player API JSON
[youtube] J5c7fJOMeXk: Downloading m3u8 information
[youtube] J5c7fJOMeXk: Downloading MPD manifest
[youtube] J5c7fJOMeXk: Downloading m3u8 information
[youtube] J5c7fJOMeXk: Downloading MPD manifest
[download] Got server HTTP error: HTTP Error 502: Bad Gateway. Retrying (attempt 2 of 10) ...
[download] Got server HTTP error: IncompleteRead(227304 bytes read). Retrying fragment 4764 (attempt 1 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 4764 (attempt 2 of 10) ...
[download] Got server HTTP error: HTTP Error 503: Service Unavailable. Retrying (attempt 1 of 10) ...
[download] 2.43GiB at  5.30KiB/s (01:20:31) (frag 4763)[download] Got server HTTP error: The read operation timed out. Retrying (attempt 2 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 4764 (attempt 3 of 10) ...
[download] 2.43GiB at  3.99KiB/s (01:21:10) (frag 4763)[download] Got server HTTP error: The read operation timed out. Retrying (attempt 1 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 4764 (attempt 4 of 10) ...
[download] 2.43GiB at  3.50KiB/s (01:21:34) (frag 4763)[download] Got server HTTP error: The read operation timed out. Retrying (attempt 1 of 10) ...
[download] Got server HTTP error: HTTP Error 503: Service Unavailable. Retrying (attempt 2 of 10) ...
[download] Got server HTTP error: IncompleteRead(1000 bytes read). Retrying fragment 4764 (attempt 5 of 10) ...
[download] Got server HTTP error: IncompleteRead(1000 bytes read). Retrying fragment 4764 (attempt 6 of 10) ...
[download] Got server HTTP error: The read operation timed out. Retrying (attempt 1 of 10) ...
[download] Got server HTTP error: HTTP Error 503: Service Unavailable. Retrying (attempt 2 of 10) ...
[download] Got server HTTP error: The read operation timed out. Retrying (attempt 3 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 4764 (attempt 7 of 10) ...
[download] Got server HTTP error: The read operation timed out. Retrying (attempt 1 of 10) ...
[download] Got server HTTP error: HTTP Error 503: Service Unavailable. Retrying (attempt 2 of 10) ...
[download] Got server HTTP error: The read operation timed out. Retrying (attempt 3 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 4764 (attempt 8 of 10) ...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 4764 (attempt 9 of 10) ...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 4764 (attempt 10 of 10) ...
[download] fragment not found; Skipping fragment 4764 ...
[download] 100% of 2.43GiB in 01:24:24

[Merger] Merging formats into "【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f299.mp4' -i 'file:【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f140.mp4' -c copy -map 0:v:0 -map 1:a:0 -movflags +faststart 'file:【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].temp.mp4'
Deleting original file 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f140.mp4 (pass -k to keep)
Deleting original file 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f299.mp4 (pass -k to keep)
Sat Feb 26 03:44:55 AM EST 2022


# download 2, started late
date; ytdlp -v --write-thumbnail --write-description --write-info-json  --live-from-start --wait-for-video=2-3 -N4 --keep-fragments J5c7fJOMeXk;date
Sat Feb 26 02:47:58 AM EST 2022
[debug] Command-line config: ['-v', '--write-thumbnail', '--write-description', '--write-info-json', '--live-from-start', '--wait-for-video=2-3', '-N4', '--keep-fragments', 'J5c7fJOMeXk']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.02.04 [c1653e9ef] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: f0734e119
[debug] Python version 3.10.2 (CPython 64bit) - Linux-5.15.25-1-lts-x86_64-with-glibc2.35
[debug] exe versions: ffmpeg 5.0 (setts), ffprobe 5.0, rtmpdump 2.4
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: J5c7fJOMeXk
[youtube] J5c7fJOMeXk: Downloading webpage
[youtube] J5c7fJOMeXk: Downloading android player API JSON
[youtube] J5c7fJOMeXk: Downloading MPD manifest
[youtube] J5c7fJOMeXk: Downloading MPD manifest
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, codec:vp9.2, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Default format spec: best/bestvideo+bestaudio
[info] J5c7fJOMeXk: Downloading 1 format(s): 299+140
[info] Writing video description to: 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].description
[info] Downloading video thumbnail 46 ...
[info] Writing video thumbnail 46 to: 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].webp
[info] Writing video metadata as JSON to: 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].info.json
[debug] Invoking downloader on "https://manifest.googlevideo.com/api/manifest/dash/expire/1645883280/ei/MNsZYs7SOM-T2LYPvYe78A4/ip/<IP_ADDR>/id/J5c7fJOMeXk.1/source/yt_live_broadcast/requiressl/yes/hfr/1/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/vprv/1/pacing/0/itag_bl/376%2C377%2C384%2C385%2C612%2C613%2C617%2C619%2C623%2C628%2C655%2C656%2C660%2C662%2C666%2C671/keepalive/yes/fexp/24001373%2C24007246/itag/0/playlist_type/LIVE/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhAKoe926iCe39_YhUkoRbQG1XMuW-Anh26o2epLvoUXLvAiBhg_iObCdPsStDTZ-KHX7IxPMGkjVzB31WSD7tNfIhpQ%3D%3D", "https://manifest.googlevideo.com/api/manifest/dash/expire/1645883280/ei/MNsZYs7SOM-T2LYPvYe78A4/ip/<IP_ADDR>/id/J5c7fJOMeXk.1/source/yt_live_broadcast/requiressl/yes/hfr/1/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/vprv/1/pacing/0/itag_bl/376%2C377%2C384%2C385%2C612%2C613%2C617%2C619%2C623%2C628%2C655%2C656%2C660%2C662%2C666%2C671/keepalive/yes/fexp/24001373%2C24007246/itag/0/playlist_type/LIVE/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhAKoe926iCe39_YhUkoRbQG1XMuW-Anh26o2epLvoUXLvAiBhg_iObCdPsStDTZ-KHX7IxPMGkjVzB31WSD7tNfIhpQ%3D%3D"
[dashsegments] Total fragments: unknown (live)
[download] Destination: 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f299.mp4
[dashsegments] Total fragments: unknown (live)
[download] Destination: 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f140.mp4
WARNING: The download speed shown is only of one thread. This is a known issue and patches are welcome
WARNING: The download speed shown is only of one thread. This is a known issue and patches are welcome
[download] 1.30GiB at 379.93KiB/s (15:48) (frag 2608)[youtube] J5c7fJOMeXk: Downloading webpage
[download] 1.30GiB at  2.67MiB/s (15:49) (frag 2610)
[download] 1.30GiB at  2.09MiB/s (15:49) (frag 2611)[youtube] J5c7fJOMeXk: Downloading MPD manifest
[youtube] J5c7fJOMeXk: Downloading MPD manifestg 2617)
[download] 100% of 1.30GiB in 15:52
[download] 100% of 54.09MiB in 15:51
[Merger] Merging formats into "【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f299.mp4' -i 'file:【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f140.mp4' -c copy -map 0:v:0 -map 1:a:0 -movflags +faststart 'file:【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].temp.mp4'
Deleting original file 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f140.mp4 (pass -k to keep)
Deleting original file 【KARAOKE】Off COLLAB(Fauna, Mumei, Ina, Kronii) IKZ [J5c7fJOMeXk].f299.mp4 (pass -k to keep)
Sat Feb 26 03:04:06 AM EST 2022

[krichbanana]

I can still reproduce this; in my case it was on a YouTube stream where the stream is lagging. I added some local debugging info to try and hunt down why, although I still lack a complete picture.

First off, the streams “die” (stop generating) on their own volition, when no_fragment_score is incremented past 30. is_live is still True. It’s possible for just one of the streams to die if the error (which caused a read timeout) was very brief.

When the fragment download fails, the MPD manifest is refetched via _extract_mpd_formats_and_subtitles. Upon doing so, the list of available formats can change:

[youtube] Format availability changed! Current: ['139', '140', '133', '134', '135', '160', '136', '137', '242', '243', '244', '278', '247', '248']; lost ['299', '298']; gained ['248', '247', '242', '244', '243', '278'] [youtube] Format availability changed! Current: ['139', '140', '133', '134', '135', '160', '136', '137']; lost ['243', '244', '248', '278', '242', '247']; gained [] (debug message is mine)

There is a brief period (up to 10 seconds?) right when a livestream actually ends where all the formats may disappear before the stream no longer is live. Due to self-throttling for the manifest download this brief period could be missed. When the formats can be refreshed, new formats will appear:

[youtube] Format availability changed! Current: ['91', '92', '93', '94', '300', '301', '139', '140', '133', '134', '135', '160', '298', '299', '136', '137']; lost []; gained ['94', '91', '301', '300', '92', '93'] (as a side note, formats 300 and 301 do not appear to be listed in the extractor; they seem to be combined 720p/1080p HLS streams that are only available until the video is converted to vp9)

---

I believe the throttling on refresh_manifest may be causing no_fragment_score to increment rapidly (beyond the 30 threshold) and causing the stream download to stop. My suggestions:

• As long as is_live is true, we do not exit the stream (don’t increment no_fragment_score). Errors should be assumed to be transient (except HTTP 403 maybe).
• We should notify if a stream has died so that we aren’t downloading just the video or audio without the user realizing.
• We should not abort downloading if the format disappears or the manifest fails to download; YouTube often throws HTTP 502/503 errors which go away after a few seconds. A retry with exponential backoff I believe would be appropriate here.
• The manifest should possibly be checked more often (with reduced/no throttle on error), but the throttle itself needs to stay. I’m not very sure about this part, since I don’t know the exact reason for the 5 second throttle; I think to avoid simultaneous requests caused by concurrency?

I likely made a few mistakes; feel free to point them out.

[pukkandan]

https://github.com/yt-dlp/yt-dlp/pull/3065 likely fixed this. Can be reopened if anyone can reproduce