zattoo.com/tv.salt.ch - youtube_dl.utils.RegexNotFoundError: Unable to extract app token

Checklist

• I’m reporting a broken site support issue
• I’ve verified that I’m running youtube-dl version 2021.03.25
• I’ve checked that all provided URLs are alive and playable in a browser
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar bug reports including closed ones
• I’ve read bugs section in FAQ

Verbose log

First attempt on zattoo.com:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--username', 'PRIVATE', '--password', 'PRIVATE', '-v', '--verbose', 'https://zattoo.com/watch/w9suisse/212778329']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.03.25
[debug] Python version 3.9.2 (CPython) - macOS-11.2.3-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.3.2, ffprobe 4.3.2, rtmpdump 2.4
[debug] Proxy map: {}
[Zattoo] Downloading app token
ERROR: Unable to extract app token; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 806, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 827, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 533, in extract
    self.initialize()
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 437, in initialize
    self._real_initialize()
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/extractor/zattoo.py", line 57, in _real_initialize
    app_token = self._html_search_regex(
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 1021, in _html_search_regex
    res = self._search_regex(pattern, string, name, default, fatal, flags, group)
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract app token; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Second attempt on tv.salt.ch (rebranded zattoo), with the corresponding account:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--username', 'PRIVATE', '--password', 'PRIVATE', '-v', 'https://tv.salt.ch/watch/w9suisse/212741443']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.03.25
[debug] Python version 3.9.2 (CPython) - macOS-11.2.3-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.3.2, ffprobe 4.3.2, rtmpdump 2.4
[debug] Proxy map: {}
[SaltTV] Downloading app token
ERROR: Unable to extract app token; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 806, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 827, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 533, in extract
    self.initialize()
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 437, in initialize
    self._real_initialize()
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/extractor/zattoo.py", line 57, in _real_initialize
    app_token = self._html_search_regex(
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 1021, in _html_search_regex
    res = self._search_regex(pattern, string, name, default, fatal, flags, group)
  File "/usr/local/Cellar/youtube-dl/2021.3.25/libexec/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract app token; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

Retrieving Zattoo feeds doesn’t seem to work since at least 3-4 months because of this “Unable to extract app token” error. Issue https://github.com/ytdl-org/youtube-dl/issues/22258 from @Europe57 seems to be stuck, and even if there was a commit from @goggle about 1 month ago (https://github.com/goggle/youtube-dl-1/commit/49b4b838cb3b129d3adef071d3964e0c29d2eca4), it still doesn’t work (not sure if it is already in the actual release).

It would be really great if youtube-dl would work (again) with Zattoo-based feeds. Please feel free to send test/debug requests. Last try was on OS X but results were similar under Linux and the latest youtube-dl version. Thanks!

[alv991]

Also, it seems quite possible and not terribly hard at a first sight:

1. GET /login
2. in the page, search for this line: <script type="text/javascript" src="/app-64449d322222dd82e35b.js"></script>
3. GET /app-64449d322222dd82e35b.js
4. search for \"token-xxxxxxxxxx\" there
5. token = https://tv.salt.ch/token-xxxxxxx.json, which should be able to replace "/token.json" for the rest of the process (untested yet).

I’m not a python coder, so I’m not sure I can create a patch for this, but I will still try… Most probably you would be much faster here @goggle 😃

Corresponding c++ code in the pvr.zattoo source (https://github.com/rbuehlma/pvr.zattoo/blob/f7e15bebe655cb7c04324c8edc43db55ad59ece2/src/ZatData.cpp#L152)

bool ZatData::LoadAppTokenFromJson(std::string html) {
  size_t basePos = html.find("src=\"/app-") + 5;
  if (basePos < 6) {
    kodi::Log(ADDON_LOG_ERROR, "Unable to find app-*.js");
    return false;
  }
  size_t endPos = html.find("\"", basePos);
  std::string appJsPath = html.substr(basePos, endPos - basePos);
  int statusCode;
  std::string content = m_httpClient->HttpGet(m_providerUrl + appJsPath, statusCode);

  basePos = content.find("\"token-") + 1;
  if (basePos < 6) {
    kodi::Log(ADDON_LOG_ERROR, "Unable to find token-*.json in %s", appJsPath.c_str());
    return false;
  }
  endPos = content.find("\"", basePos);
  std::string tokenJsonPath = content.substr(basePos, endPos - basePos);
  return LoadAppTokenFromTokenJson(tokenJsonPath);
}

[goggle]

If you replace watch with program in the URL it should work with the version of my pull request (https://github.com/goggle/youtube-dl-1/commit/49b4b838cb3b129d3adef071d3964e0c29d2eca4).

Here is the output:

python -m youtube_dl --netrc "https://zattoo.com/program/w9suisse/212778329" -F

[Zattoo] Downloading session token
[Zattoo] Opening session
[Zattoo] Logging in
[Zattoo] 212778329: Downloading video information
[Zattoo] 212778329: Downloading DASH formats
[Zattoo] 212778329: Downloading MPD manifest
[Zattoo] 212778329: Downloading MPD manifest
[Zattoo] 212778329: Downloading HLS formats
WARNING: Unable to download JSON metadata: HTTP Error 403: Forbidden
[Zattoo] 212778329: Downloading HLS5 formats
[Zattoo] 212778329: Downloading m3u8 information
[Zattoo] 212778329: Downloading m3u8 information
[Zattoo] 212778329: Downloading HDS formats
WARNING: Unable to download JSON metadata: HTTP Error 403: Forbidden
[info] Available formats for 212778329:
format code                              extension  resolution note
hls5-5002-A-audio-mp4a.40.2-Français     mp4        audio only [fr] 
hls5-5002-A-audio-mp4a.40.2-Other__qad_  mp4        audio only [qad] 
hls5-601-A-audio-mp4a.40.2-Français      mp4        audio only [fr] 
hls5-601-A-audio-mp4a.40.2-Other__qad_   mp4        audio only [qad] 
dash-5002-A-4                            m4a        audio only [fr] DASH audio  128k , m4a_dash container, mp4a.40.2
dash-5002-A-5                            m4a        audio only [qad] DASH audio  128k , m4a_dash container, mp4a.40.2
dash-601-A-1                             m4a        audio only [fr] DASH audio  128k , m4a_dash container, mp4a.40.2
dash-601-A-2                             m4a        audio only [qad] DASH audio  128k , m4a_dash container, mp4a.40.2
hls5-5002-A-192                          mp4        400x224     192k , avc1.42c00b, 5.0fps, video only
hls5-601-A-192                           mp4        400x224     192k , avc1.42c00b, 5.0fps, video only
hls5-5002-A-400                          mp4        400x224     400k , avc1.42c00c, 12.5fps, video only
hls5-601-A-400                           mp4        400x224     400k , avc1.42c00c, 12.5fps, video only
dash-5002-A-3                            mp4        512x288    DASH video  452k , mp4_dash container, avc1.4d4015, 25fps, video only
dash-601-A-0                             mp4        512x288    DASH video  452k , mp4_dash container, avc1.4d4015, 25fps, video only
hls5-5002-A-602                          mp4        512x288     602k , avc1.4d4015, 25.0fps, video only
hls5-601-A-602                           mp4        512x288     602k , avc1.4d4015, 25.0fps, video only
dash-5002-A-2                            mp4        512x288    DASH video  700k , mp4_dash container, avc1.4d4015, 25fps, video only
hls5-5002-A-900                          mp4        512x288     900k , avc1.4d4015, 25.0fps, video only
dash-5002-A-1                            mp4        768x432    DASH video 1300k , mp4_dash container, avc1.4d401e, 25fps, video only
hls5-5002-A-1500                         mp4        768x432    1500k , avc1.4d401e, 25.0fps, video only
dash-5002-A-0                            mp4        1024x576   DASH video 2799k , mp4_dash container, avc1.4d4020, 50fps, video only
hls5-5002-A-2999                         mp4        1024x576   2999k , avc1.4d4020, 50.0fps, video only (best)

To make it work on SaltTV I need access to an account.

Edit: I have just added support for these types of urls containing watch in to my PR: https://github.com/ytdl-org/youtube-dl/pull/28253/commits/951695e01917fb59c5bb1f8b66a95149922b94c9