ITV Hub JSON error

Checklist

• I’m reporting a broken site support
• I’ve verified that I’m running youtube-dl version 2021.04.26
• I’ve checked that all provided URLs are alive and playable in a browser
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.itv.com/hub/tipping-point/2a1875a1123', '--verbose']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2021.04.26
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: none
[debug] Proxy map: {}
[debug] Using fake IP 25.161.160.85 (GB) as X-Forwarded-For.
[ITV] 2a1875a1123: Downloading webpage
[ITV] 2a1875a1123: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\extractor\common.py", line 634, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpeav8wi2f\build\youtube_dl\YoutubeDL.py", line 2279, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

Description

Several programmes on ITV Hub now fail to download. URL’s such as

https://www.itv.com/hub/the-chase/1a7842a1761 https://www.itv.com/hub/tipping-point/2a1875a1123 https://www.itv.com/hub/tenable/10a0805a0021

fail to download though some like

https://www.itv.com/hub/lorraine/1a9360a2830

continue to work

[Vangelis66]

@torbs723 wrote:

@Vangelis66 Any ideas what might be happening here?

I was tagged (in a now hidden comment) to troubleshoot this… 😄 It is now already Orthodox Easter Sunday here, I won’t be available for the most part of the next two days… However, since I, too, have a vested interest in ITVHub shows fetched via yt-dl, I took a stab at it…

But first, let’s discuss the geo-restricted label attributed to this issue… In general, it is not a good idea to aggravate the main developer(s) of yt-dl 😉 ; perhaps the label was just put there to highlight the fact that for anyone to even start troubleshooting this, an ITV Hub whitelisted IP address is required; sometimes, a generic UK IP won’t cut it, you need an “ITV zone” one (e.g. excluding Scotland) … My analysis has indeed proven (to me, at least, but in accordance to comments by other people here) that this new ITV HUB issue is not related to geo-fencing…

While I am discussing geo-fencing, the ITV Hub people, some time in the month of March (2021), imposed even stricter geo-blocking schemes… 💢 For many years, their Playlist APIs (= the ones that yield stream data) would only enforce a “soft” geo-block, that could be circumvented via an X-Forwarded-For:UK IP request header; if you look closely at logs people have posted, it’s line [debug] Using fake IP 25.161.160.85 (GB) as X-Forwarded-For. From mid-March, if you happen to be overseas, you need a whitelisted UK HTTPS/SOCKS proxy/UK VPN to even get the master HLS playlist URI (“Downloading JSON metadata” in yt-dl’s logs); of course, the HLS stream itself is fully geo-blocked, you have to use the same “means” to dump if overseas… Again, this new “issue” isn’t related to stricter geo-fencing practices imposed last March (as it also occurs for people within the UK…).

It well appears that the ITV Hub people made some changes during these last days, certainly during last week, that make recently uploaded ITV hub shows inaccessible to yt-dl; case in point is the Jonathan Ross show: Last week’s episode

youtube-dl -F https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0204

[ITV] 2a1166a0204: Downloading webpage
[ITV] 2a1166a0204: Downloading JSON metadata
[ITV] 2a1166a0204: Downloading m3u8 information
[info] Available formats for 2a1166a0204:
format code  extension  resolution note
hls-105      mp4        audio only  105k , mp4a.40.2@ 98k
hls-565      mp4        512x288     565k , avc1.64001F@ 434k, mp4a.40.2@ 98k
hls-762      mp4        512x288     762k , avc1.64001F@ 620k, mp4a.40.2@ 98k
hls-1058     mp4        896x504    1058k , avc1.64001F@ 899k, mp4a.40.2@ 98k
hls-1344     mp4        896x504    1344k , avc1.64001F@1169k, mp4a.40.2@ 98k
hls-1633     mp4        896x504    1633k , avc1.64001F@1442k, mp4a.40.2@ 98k (be
st)

Yerterday’s episode:

youtube-dl -F https://www.itv.com/hub/the-jonathan-ross-show/2a1166a0205

[ITV] 2a1166a0205: Downloading webpage
[ITV] 2a1166a0205: Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 500: Internal Server Error (
caused by <HTTPError 500: 'Internal Server Error'>); please report this issue on
 https://yt-dl.org/bug . Make sure you are using the latest version; type  youtu
be-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and incl
ude its complete output.

(the test was done using the very same connection/IP).

Now, my deeper analysis was done focusing on Viewpoint-SE01EP03 (sadly, the whole series is due to expire imminently): https://www.itv.com/hub/viewpoint/10a0556a0003 As most of you may know already, desktop browsers are being served MPEG-DASH streams with Common Encryption (cenc=DRM), the Google-owned WidevineCDM is required for playback 👎 💢 ; yt-dl can’t/won’t handle cenc, instead it has to resort to HLSe (AES-128 encrypted) streams, targeting older iOS devices…

So the very first thing I did was to emulate such a device in my browser, by setting up a site-specific-user-agent-override (SSUAO) for itv.com; depending on browser used, this can be achieved via an about:config setting (Firefox derived browsers) or, more commonly, via an extension (Chromium browsers) that spoofs your user-agent; please, don’t ask for instructions/how-to here, it isn’t the appropriate place… I chose to use the iPhone5 UA below:

Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A334 Safari/7534.48.3

Of course, ITVHub requires you to be logged-in to send any stream data to your browser (this hasn’t been yet a requirement for yt-dl 👍 ); being logged in and with the above “mobile” UA, I inspected browser’s Web Console (CTRL+SHIFT+K) and I clearly saw the POST XHR request to their playlist API: https://magni.itv.com/playlist/itvonline/ITV/10_0556_0003.001 (don’t try this directly with a GET request), the response of which I am attaching here: POST-ITV.json.txt BTW, this is equivalent to yt-dl “Downloading JSON metadata”; as one can see, no HTTP Error 500: Internal Server Error was generated in my test; this rules out intermittent server issues on the ITV side…

From that (see attached) JSON response, I extracted the master HLSe playlist URI:

https://itvpnpmobile.content.itv.com/10-0556-0003-001/6/2/VAR003/10-0556-0003-001_6_2_VAR003.ism/.m3u8?hdnea=st=1619890241~exp=1619911841~acl=/10-0556-0003-001/*~data=nohubplus~hmac=ec46cb4825a413f0681ce4b94f72c5f82c9fb42fb6d424e746976a5d2c26aa99

(the same URI can be seen and copied from within the browser’s Web Console 😉 ), which looks like below:

#EXTM3U
#EXT-X-VERSION:4
## Created with OEM(version=1.7.25)


# AUDIO groups
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio-aacl-128",NAME="audio",AUTOSELECT=YES,DEFAULT=YES,CHANNELS="2"

# variants
#EXT-X-STREAM-INF:BANDWIDTH=313000,CODECS="mp4a.40.2,avc1.4D401E",RESOLUTION=512x288,FRAME-RATE=50,AUDIO="audio-aacl-128",CLOSED-CAPTIONS=NONE
10-0556-0003-001_6_2_VAR003-audio=128000-video=167161.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=596000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=512x288,FRAME-RATE=50,AUDIO="audio-aacl-128",CLOSED-CAPTIONS=NONE
10-0556-0003-001_6_2_VAR003-audio=128000-video=433735.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=794000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=512x288,FRAME-RATE=50,AUDIO="audio-aacl-128",CLOSED-CAPTIONS=NONE
10-0556-0003-001_6_2_VAR003-audio=128000-video=620155.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1064000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=896x504,FRAME-RATE=50,AUDIO="audio-aacl-128",CLOSED-CAPTIONS=NONE
10-0556-0003-001_6_2_VAR003-audio=128000-video=875433.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1655000,CODECS="mp4a.40.2,avc1.4D401F",RESOLUTION=896x504,FRAME-RATE=50,AUDIO="audio-aacl-128",CLOSED-CAPTIONS=NONE
10-0556-0003-001_6_2_VAR003-audio=128000-video=1432734.m3u8

# variants
#EXT-X-STREAM-INF:BANDWIDTH=136000,CODECS="mp4a.40.2",AUDIO="audio-aacl-128"
10-0556-0003-001_6_2_VAR003-audio=128000.m3u8

# keyframes
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=23000,CODECS="avc1.4D401E",RESOLUTION=512x288,URI="keyframes/10-0556-0003-001_6_2_VAR003-video=167161.m3u8"
#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=116000,CODECS="avc1.4D401F",RESOLUTION=896x504,URI="keyframes/10-0556-0003-001_6_2_VAR003-video=875433.m3u8"

This URI is short-lived, you have to act swiftly before its auth-token expires… Feeding that directly to yt-dl, I can initiate a successful download:

youtube-dl --console-title --hls-prefer-native --hls-use-mpegts -c --fixup never "https://itvpnpmobile.content.itv.com/10-0556-0003-001/6/2/VAR003/10-0556-0003-001_6_2_VAR003.ism/.m3u8?hdnea=st=1619890241~exp=1619911841~acl=/10-0556-0003-001/*~data=nohubplus~hmac=ec46cb4825a413f0681ce4b94f72c5f82c9fb42fb6d424e746976a5d2c26aa99" -o "ITV_Viewpoint_Se1Ep3_202104282100[10a0556a0003].ts" =>

[generic] *~data=nohubplus~hmac=ec46cb4825a413f0681ce4b94f72c5f82c9fb42fb6d424e7
46976a5d2c26aa99: Requesting header
[generic] *~data=nohubplus~hmac=ec46cb4825a413f0681ce4b94f72c5f82c9fb42fb6d424e7
46976a5d2c26aa99: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 461
[download] Destination: ITV_Viewpoint_Se1Ep3_202104282100[10a0556a0003].ts
[download]   2.6% of ~543.70MiB at 942.18KiB/s ETA 12:38

I did spend a lot of time trying to emulate the above POST XHR my browser sent in a yt-dl command, by specifying all needed request headers (referer/user-agent/cookies etc.), I even tried the cookies.txt method, sadly all to no avail 😭 … It’s a real pity that I can’t code (in Python or other language), I hope my findings can be put to good use by actual coders…

In conclusion: ITV staff have implemented in recent days “changes” (I wouldn’t rule out this having been done deliberately to thwart downloaders) that would render current itvIE useless once old shows expire, perhaps sooner if these changes propagate to all ITVhub content… At the time of this writing, this isn’t related to an implementation of pure DRM (e.g. Apple FairPlay), nor to further changes to their geo-fencing practices… Therefore, I humbly concluded it’s a genuine itv.py bug that needs to be addressed in code… @remitamine, you seem to be the defacto maintainer of the itvIE, please, when your schedules permit, could you take a deeper look into this? Also pinging @dirkf who happens to be UK-based, a Python coder and quite knowledgeable 😄 in UK media sites…

My best wishes to all, my deepest thanks to the yt-dl dev team 👍 , Happy Easter Sunday to all those that observe it today… 😃

[sleaux-meaux]

@torbs723: Carrying over from https://github.com/Ascoware/get-iplayer-automator/issues/331#issuecomment-832252389

Is there any chance of a windows .exe version while we wait for youtube-dl to approve the patch and release it?

Use yt-dlp version 2021.12.01 or later

https://github.com/yt-dlp/yt-dlp/releases/latest