Crunchyroll 404 (due to forced US beta?)

Checklist

• I’m reporting a broken site
• I’ve verified that I’m running yt-dlp version 2022.03.08.1. (update instructions)
• I’ve checked that all provided URLs are alive and playable in a browser
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I’ve read the guidelines for opening an issue
• I’ve read about sharing account credentials and I’m willing to share it if required

Region

United States

Description

Attempting to download a single video from a season, ‘https://beta.crunchyroll.com/watch/GYK53DMPR/wicked-lord-shingan-reborn’ results in a 404 Not Found error, but works as expected in a browser.

I suspect this is because ALL people in the US are forcibly redirected to the “beta” site now, which is no longer really beta, and so if the extractor still tries to point at the old site, it fails for anyone who can be detected as coming from the US. I don’t currently have a way to test this via a VPN, but that would probably be the way to verify this issue. This is not a premium video.

Since Crunchyroll now owns Funimation and has decided to remove their content from competing websites, this will probably become a larger issue going forward.

Verbose log

[debug] Command-line config: ['-vU', '-sF', 'https://beta.crunchyroll.com/watch/GYK53DMPR/wicked-lord-shingan-reborn']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.03.08.1 [c0c2c57d3] (zip)
[debug] Python version 3.9.10 (CPython 64bit) - Linux-5.16.0-4-amd64-x86_64-with-glibc2.33
[debug] exe versions: ffmpeg 4.4.1-3 (setts), ffprobe 4.4.1-3
[debug] Optional libraries: brotli, Cryptodome, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.03.08.1, Current version: 2022.03.08.1
yt-dlp is up to date (2022.03.08.1)
[debug] [crunchyroll:beta] Extracting URL: https://beta.crunchyroll.com/watch/GYK53DMPR/wicked-lord-shingan-reborn
[crunchyroll:beta] wicked-lord-shingan-reborn: Downloading webpage
[crunchyroll] None: Downloading webpage
[debug] [crunchyroll] Extracting URL: https://www.crunchyroll.com/love-chunibyo--other-delusions---heart-throb--/wicked-lord-shingan-reborn-648781
[crunchyroll] 648781: Downloading webpage
ERROR: [crunchyroll] 648781: Unable to download webpage: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on  https://github.com/yt-dlp/yt-dlp , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "/home/wiley/bin/yt-dlp/yt_dlp/extractor/common.py", line 637, in extract
    ie_result = self._real_extract(url)
  File "/home/wiley/bin/yt-dlp/yt_dlp/extractor/crunchyroll.py", line 406, in _real_extract
    webpage = self._download_webpage(
  File "/home/wiley/bin/yt-dlp/yt_dlp/extractor/crunchyroll.py", line 259, in _download_webpage
    return super(CrunchyrollBaseIE, self)._download_webpage(request, *args, **kwargs)
  File "/home/wiley/bin/yt-dlp/yt_dlp/extractor/common.py", line 928, in _download_webpage
    res = self._download_webpage_handle(
  File "/home/wiley/bin/yt-dlp/yt_dlp/extractor/common.py", line 796, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "/home/wiley/bin/yt-dlp/yt_dlp/extractor/common.py", line 781, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "/home/wiley/bin/yt-dlp/yt_dlp/extractor/common.py", line 763, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/wiley/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3588, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 555, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 747, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

[tejing1]

I’ve included a fix in #2955

[tejing1]

My PR won’t fix this. It’s an error with how the non-beta url is being calculated from the beta page info during the redirect. For now, just get the non-beta url directly from the website when downloading from this series. Or supply login credentials if you want to actually use the beta extractor; without them it just redirects to non-beta anyway.

In this case, the correct non-beta url would be https://www.crunchyroll.com/love-chunibyo-other-delusions-heart-throb-/episode-1-wicked-lord-shingan-reborn-648781

I guess the series portion of the non-beta url isn’t always equal to the series_slug_title in the beta api. Based on this example, I’d guess it needs to squash down multiple -s in a row into a single -, but I’d like some more data before making that change.