ffmpeg's `-movflags +faststart` fails on gvfs

Checklist

• I’m reporting a broken site
• I’ve verified that I’m running yt-dlp version 2022.01.21. (update instructions)
• I’ve checked that all provided URLs are alive and playable in a browser
• I’ve checked that all URLs and arguments with special characters are properly quoted or escaped
• I’ve searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I’ve read the guidelines for opening an issue
• I’ve read about sharing account credentials and I’m willing to share it if required

Region

France (maybe EU?)

Description

Trying to fetch https://www.france.tv/france-5/ernest-et-celestine/saison-1/3068333-pendant-que-l-orage-gronde.html with no particular CLI options, the downloaded file cannot be played with any of the players I have on my system. mpv notably reports

$ mpv Ernest\ et\ Célestine\ -\ Pendant\ que\ l\'orage\ gronde\ \[bc5bfd1c-2813-48d6-8c0e-524581bfce46\].mp4.old 
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: moov atom not found
[lavf] avformat_open_input() failed
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: moov atom not found
[lavf] avformat_open_input() failed
Failed to recognize file format.

Exiting... (Errors when loading file)

Reverting back to yt-dlp==2021.10.22 allows the downloaded file to be played properly.

I tried diffing the files (my MPEG4 skills are rusty) and it looks like some chucks are missing at the very end of the file.

-rw-r--r-- 1 remi users 182480138 25 janv. 12:57 'Ernest et Célestine - Pendant que l'\''orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].mp4'
-rw-r--r-- 1 remi users 182068891 25 janv. 12:57 'Ernest et Célestine - Pendant que l'\''orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].mp4.2022.1.21'

In fact, the files are identical up to their common size. So really, the newer version of yt-dlp omits the end of the file, which probably include the index bits (is that what it’s called? sorry if I get the terminology wrong)

$ dd bs=182068891 count=1 if=Ernest\ et\ Célestine\ -\ Pendant\ que\ l\'orage\ gronde\ \[bc5bfd1c-2813-48d6-8c0e-524581bfce46\].mp4 | sha256sum 
1+0 enregistrements lus
1+0 enregistrements écrits
182068891 octets (182 MB, 174 MiB) copiés, 3,60738 s, 50,5 MB/s
cbdf4aa2343c555311c14e8ccb13122944ec381fc8cc9ca9649d4e2b9d7cb6a7  -
$ dd bs=182068891 count=1 if=Ernest\ et\ Célestine\ -\ Pendant\ que\ l\'orage\ gronde\ \[bc5bfd1c-2813-48d6-8c0e-524581bfce46\].mp4.2022.1.21 | sha256sum 
1+0 enregistrements lus
1+0 enregistrements écrits
182068891 octets (182 MB, 174 MiB) copiés, 3,49349 s, 52,1 MB/s
cbdf4aa2343c555311c14e8ccb13122944ec381fc8cc9ca9649d4e2b9d7cb6a7  -

I’ll try to bisect later, that’s all I have for now.

Verbose log

$ yt-dlp -vU "https://www.france.tv/france-5/ernest-et-celestine/saison-1/3068333-pendant-que-l-orage-gronde.html"
[debug] Command-line config: ['-vU', 'https://www.france.tv/france-5/ernest-et-celestine/saison-1/3068333-pendant-que-l-orage-gronde.html']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, err utf-8, pref UTF-8
[debug] yt-dlp version 2022.01.21 [f20d607b0]
[debug] Python version 3.9.10 (CPython 64bit) - Linux-5.12.6-gentoo-x86_64-Intel-R-_Core-TM-_i5-4670_CPU_@_3.40GHz-with-glibc2.34
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.01.21, Current version: 2022.01.21
yt-dlp is up to date (2022.01.21)
[debug] [FranceTVSite] Extracting URL: https://www.france.tv/france-5/ernest-et-celestine/saison-1/3068333-pendant-que-l-orage-gronde.html
[FranceTVSite] 3068333-pendant-que-l-orage-gronde: Downloading webpage
[debug] [FranceTV] Extracting URL: francetv:bc5bfd1c-2813-48d6-8c0e-524581bfce46
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading desktop video JSON
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading mobile video JSON
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading signed dash manifest URL
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading MPD manifest
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading signed hls manifest URL
[FranceTV] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] bc5bfd1c-2813-48d6-8c0e-524581bfce46: Downloading 1 format(s): hls-2218+dash-audio_fre=96000
[debug] Invoking downloader on "https://cloudreplay.ftven.fr/11d11784efe16/1086049405_france-domtom_TA.ism/ZXhwPTE2NDM0MzI3MjV+YWNsPSUyZjExZDExNzg0ZWZlMTYlMmYxMDg2MDQ5NDA1X2ZyYW5jZS1kb210b21fVEEuaXNtKn5obWFjPWEwNDdiMzllZjZjM2IyZjJlY2ZjZTI0NzdkZjMyN2I2ZGZjZWI3ZmFhY2M2MzNmOGVjZjg3N2RiM2NjNDU4MjI=/1086049405_france-domtom_TA-video=2000000.m3u8"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 91
[download] Destination: Ernest et Célestine - Pendant que l'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fhls-2218.mp4
[download] 100% of 171.06MiB in 00:36
[debug] Invoking downloader on "https://cloudreplay.ftven.fr/11d11784efe16/1086049405_france-domtom_TA.ism/manifest.mpd?hdnea=ip=78.196.20.108~exp=1643411725~acl=%2f11d11784efe16%2f1086049405_france-domtom_TA.ism*~hmac=faf3bffd25fe71d0f1e38101506550299a8c5c092ccbc44bf14a8db0f3b9e46f"
[dashsegments] Total fragments: 365
[download] Destination: Ernest et Célestine - Pendant que l'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fdash-audio_fre=96000.m4a
[download] 100% of 8.52MiB in 00:42
[Merger] Merging formats into "Ernest et Célestine - Pendant que l'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:Ernest et Célestine - Pendant que l'"'"'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fhls-2218.mp4' -i 'file:Ernest et Célestine - Pendant que l'"'"'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fdash-audio_fre=96000.m4a' -c copy -map 0:v:0 -map 1:a:0 -movflags +faststart 'file:Ernest et Célestine - Pendant que l'"'"'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].temp.mp4'
Deleting original file Ernest et Célestine - Pendant que l'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fdash-audio_fre=96000.m4a (pass -k to keep)
Deleting original file Ernest et Célestine - Pendant que l'orage gronde [bc5bfd1c-2813-48d6-8c0e-524581bfce46].fhls-2218.mp4 (pass -k to keep)

[RemiCardona]

Turns out I know the fella who wrote ffmpeg’s faststart so he pointed me to the underlying implementation. It’s fairly straightforward (once all the necessary checks are done): it opens a second file descriptor on the file and then cleverly reads a chunk from that second fd, buffers it and writes a previously read chunk on the original fd. The code that moves stuff around is here https://github.com/FFmpeg/FFmpeg/blob/d61240f8c95e9cf7a0aaef2bb4495960d3fec62c/libavformat/utils.c#L2038 for anyone interested.

In any case, this code works just fine on “normal” file systems and I’m very much inclined to lay the blame on gvfs’ sftp handling (which had indeed caused me some trouble in the past). I’ll just work around the issue by downloading locally and then copying the final file afterwards. Maybe I’ll poke Gnome/GLib people about this but this is probably a very niche feature.

[fstirlitz]

The ffmpeg-formats(1) manual page warns:

-movflags faststart
Run a second pass moving the index (moov atom) to the beginning of the file. This operation can take a while, and will not work in various situations such as fragmented output, thus it is not enabled by default.

I would not be too surprised if this were a bug in that second pass; a common type of bug results from assumptions like ‘bytes_written < bytes_given if and only if there is no disk space left’ and ‘bytes_got < bytes_wanted if and only if EOF is reached’, which are usually true for regular files on local file systems, but not for more exotic file descriptors like sockets and remote mounts. Alternatively, it may be that gvfs’s random-access I/O implementation for SSHFTP is faulty.

But honestly, if you have SSH access to the machine, why not run yt-dlp on it directly?