[Broken] Can't get youtube comments if cookies are provided

Checklist

• I’m reporting a broken site support issue
• I’ve verified that I’m running yt-dlp version 2021.06.23
• I’ve checked that all provided URLs are alive and playable in a browser
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar bug reports including closed ones
• I’ve read bugs section in FAQ

Verbose log

yt-dlp -v --cookies newcookiefile_2.txt https://www.youtube.com/watch?v=Ff3kCgt7XQQ --get-comments
[debug] Command-line config: ['-v', '--cookies', 'newcookiefile_2.txt', 'https://www.youtube.com/watch?v=Ff3kCgt7XQQ', '--get-comments']
[debug] Encodings: locale cp65001, fs utf-8, out utf-8, pref cp65001
[debug] yt-dlp version 2021.06.23
[debug] Python version 3.9.1 (CPython 64bit) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg n4.4-19-g8d172d9409, ffprobe n4.4-19-g8d172d9409
[debug] Proxy map: {}
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=Ff3kCgt7XQQ
[youtube] Ff3kCgt7XQQ: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] Ff3kCgt7XQQ: Downloading 1 format(s): 303+251
[youtube] Downloading initial comment continuation page
WARNING: [youtube] Incomplete data received. Retrying ...
[youtube] Downloading initial comment continuation page (retry #1)
WARNING: [youtube] Incomplete data received. Retrying ...
[youtube] Downloading initial comment continuation page (retry #2)
WARNING: [youtube] Incomplete data received. Retrying ...
[youtube] Downloading initial comment continuation page (retry #3)
ERROR: Incomplete data received; please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "c:\users\sam\appdata\local\programs\python\python39\lib\site-packages\yt_dlp\YoutubeDL.py", line 1138, in wrapper
    return func(self, *args, **kwargs)
  File "c:\users\sam\appdata\local\programs\python\python39\lib\site-packages\yt_dlp\YoutubeDL.py", line 1174, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "c:\users\sam\appdata\local\programs\python\python39\lib\site-packages\yt_dlp\YoutubeDL.py", line 1211, in process_ie_result
    ie_result = self.process_video_result(ie_result, download=download)
  File "c:\users\sam\appdata\local\programs\python\python39\lib\site-packages\yt_dlp\YoutubeDL.py", line 2198, in process_video_result
    self.process_info(new_info)
  File "c:\users\sam\appdata\local\programs\python\python39\lib\site-packages\yt_dlp\YoutubeDL.py", line 2358, in process_info
    self.post_extract(info_dict)
  File "c:\users\sam\appdata\local\programs\python\python39\lib\site-packages\yt_dlp\YoutubeDL.py", line 2860, in post_extract
    actual_post_extract(info_dict or {})
  File "c:\users\sam\appdata\local\programs\python\python39\lib\site-packages\yt_dlp\YoutubeDL.py", line 2852, in actual_post_extract
    extra = post_extractor().items()
  File "c:\users\sam\appdata\local\programs\python\python39\lib\site-packages\yt_dlp\extractor\youtube.py", line 2550, in <lambda>
    info['__post_extractor'] = lambda: self._extract_comments(ytcfg, video_id, contents, webpage, xsrf_token)
  File "c:\users\sam\appdata\local\programs\python\python39\lib\site-packages\yt_dlp\extractor\youtube.py", line 1874, in _extract_comments
    for comment in comment_iter:
  File "c:\users\sam\appdata\local\programs\python\python39\lib\site-packages\yt_dlp\extractor\youtube.py", line 1797, in _comment_entries
    raise ExtractorError(last_error)
yt_dlp.utils.ExtractorError: Incomplete data received; please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.

Description

I’ve been trying to download videos with live chats and comments. It was working fine yesterday until I tried it today and kept getting “Incomplete data received” when it keeps trying to download the comment continuation page. This only happens if I use my cookie file and works fine if I don’t. I tried reverting back to the 2021.6.9 version and still got the same error.

[coletdjnz]

Thanks again for sending me the page dumps.

So it seems like YouTube is A/B testing your account with using the next API endpoint for comment continuations. This is exactly what we predicted and hoped for to happen in the future.

All continuation requests seem to be made via the next API, using the same behavior we have seen elsewhere (token as part of the payload).

It appears that the old endpoint accepts tokens designed for the first, however the location of the continuation data has changed (hence the incomplete data error OP gets).

---

From here I think we could do:

• keep using old API and modify code to extract continuation contents
• move over to new API and modify code to extract continuation contents, assuming current continuation tokens for old API are supported by new.

If the assumption for the latter proves true I think I’ll go that route - since we have robust handling of the new API.

While I’m at it I’ll use the new --extractor-args option to allow changing sorting modes 😃.

(Sidenote: since the continuation token is in the payload now, maybe YouTube will stop breaking after 80 pages or so in popular sorting mode due to producing too large continuations 😆)

[Spicadox]

I tried using cookies from another account and it works so I guess my account is being rate-limited. It’s just weird that it only started happening now considering how I’ve been using yt-dlp for so long for my daily archiving. I’ll give you a dm with my continuation page file.