BBC Sounds - Upgrade to V6 of BBC Open Live API (open.live.bbc.co.uk) and Fix Timeouts During Extraction and Eliminate Port 80 Requests

Checklist

• I’m reporting a broken site support issue
• I’ve verified that I’m running youtube-dl version 2020.12.29
• I’ve checked that all provided URLs are alive and playable in a browser
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar bug reports including closed ones
• I’ve read bugs section in FAQ

Verbose log

ytdl --verbose https://www.bbc.co.uk/sounds/play/m000qlyg
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.bbc.co.uk/sounds/play/m000qlyg']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.12.29
[debug] Python version 3.9.1 (CPython) - Linux
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1
[debug] Proxy map: {}
[bbc.co.uk] m000qlyg: Downloading video page
[bbc.co.uk] m000qlyg: Downloading playlist JSON
[bbc.co.uk] m000qlyf: Downloading media selection XML
[bbc.co.uk] m000qlyf: Downloading MPD manifest
[bbc.co.uk] m000qlyf: Downloading m3u8 information
WARNING: Failed to download m3u8 information: <urlopen error [Errno 60] Operation timed out>
[bbc.co.uk] m000qlyf: Downloading m3u8 information
...

Description

BBC Sounds (including programme and sounds urls) get stuck at the “Downloading m3u8 information” stage (at least here outside the UK) and loops on “Operation timed out” until eventually (sometimes) extracting (this is after making requests to aod-dash-ww-live.akamaized.net or aod-hls-ww.live.bbcfmt.hs.llnwd.net). I note that youtube-dl still uses v5 of the Open Live MediaSelector API (open.live.bbc.co.uk) as opposed to v6 which returns in JSON.

Something seems to be going wrong regarding extraction of the media URLs (href in the API output from https://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/iptv-all/vpid/VPID where the corresponding VPID is found using https://bbc.co.uk/programmes/PID/playlist.json). I have managed to make a simple shell script call the APIs manually and play/extract instantly, so surely this should not be that difficult to implement…

I also noted that the extractor was making requests on port 80 at times as well as 443 to all the relevant APIs (open.live.bbc.co.uk, bbc.co.uk, aod-hls-ww.live.bbcfmt.hs.llnwd.net, etc…) which seems unnecessary and something which should be eliminated since all the endpoints support TLS.

[dirkf]

None of the quoted test URIs/URLs are active now (404) because the BBC has forgotten the point of them.

Also, I agree with the comment above.

Unless an active test URL is provided, I think this issue could be closed.

[Vangelis66]

@dirkf wrote:

None of the quoted test URIs/URLs are active now (404) because the BBC has forgotten the point of them.

When a BBC radio/audio programme has expired and is no longer available on BBC Sounds as AOD, the */sounds/play/* URIs will 404, or at least that’s my understanding of this… 😉 ; however, because every radio/audio programme (on Sounds) has been assigned a unique PID string (which lasts, practically, for ever), the */programmes/pid URIs remain valid, e.g.

https://www.bbc.co.uk/sounds/play/m000r3jn => https://www.bbc.co.uk/programmes/m000r3jn =>

BBC Radio 1 - Diplo and Friends - Damian Lazarus in the Mix
First broadcast:  Sun 10 Jan 2021 00:00 
Sorry, this episode is not currently available

… so, actually, the BBC hasn’t forgotten the details of that particular programme … Of course, trying now

youtube-dl https://www.bbc.co.uk/programmes/m000r3jn

you’ll simply get:

[bbc.co.uk] m000r3jn: Downloading video page
ERROR: Sorry, this episode is not currently available

BTW, thanks for the vote of confidence! 😃