FFMPEG CannotReadFrameError for 29.46 fps video

Hello,

Appreciate your advice on why imageio is having trouble reading an mp4. I suspect it has to do with the input rate of 29.46 fps. I am attempting to step through the mp4 frame by frame using the following code:

r = imageio.get_reader(file)
for i in range(r.get_length()):
  im = r.get_data(i)
  # Process image here...

Here is the print_info output for the video

ffmpeg version 3.2.2 Copyright (c) 2000-2016 the FFmpeg developers
  built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/3.2.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-frei0r --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid --enable-opencl --disable-lzma --enable-libopenjpeg --disable-decoder=jpeg2000 --extra-cflags=-I/usr/local/Cellar/openjpeg/2.1.2/include/openjpeg-2.1 --enable-nonfree --enable-vda
  libavutil      55. 34.100 / 55. 34.100
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.100 / 57. 56.100
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/echeng/video.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: isom3gp43gp5
  Duration: 00:16:05.80, start: 0.000000, bitrate: 1764 kb/s
    Stream #0:0(eng): Audio: aac (LC) (mp4a / 0x6134706D), 8000 Hz, mono, fltp, 40 kb/s (default)
    Metadata:
      handler_name    : soun
    Stream #0:1(eng): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 1720 kb/s, 29.46 fps, 26.58 tbr, 90k tbn, 1k tbc (default)
    Metadata:
      handler_name    : vide
Output #0, image2pipe, to 'pipe:':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: isom3gp43gp5
    encoder         : Lavf57.56.100
    Stream #0:0(eng): Video: rawvideo (RGB[24] / 0x18424752), rgb24, 640x480 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 26.58 fps, 26.58 tbn, 26.58 tbc (default)
    Metadata:
      handler_name    : vide
      encoder         : Lavc57.64.101 rawvideo
Stream mapping:
--------------------------------------------------------------------------------

I notice the input fps is 29.46, but the Output stream fps is reported as 26.58 (same as input tbr). Is this discrepancy a problem?

imageio throws a imageio.core.format.CannotReadFrameError: Could not read frame: exception starting at frame 25676:

Traceback (most recent call last):
  File "test_video.py", line 7, in <module>
    pputils.reencode(videofile, outfile, drawframenums=True) 
  File "/Users/echeng/pputils.py", line 94, in reencode
    im = r.get_data(i)
  File "/Users/echeng/venv/lib/python2.7/site-packages/imageio/core/format.py", line 341, in get_data
    im, meta = self._get_data(index, **kwargs)
  File "/Users/echeng/venv/lib/python2.7/site-packages/imageio/plugins/ffmpeg.py", line 340, in _get_data
    result = self._read_frame()
  File "/Users/echeng/venv/lib/python2.7/site-packages/imageio/plugins/ffmpeg.py", line 543, in _read_frame
    s = self._read_frame_data()
  File "/Users/echeng/venv/lib/python2.7/site-packages/imageio/plugins/ffmpeg.py", line 529, in _read_frame_data
    raise CannotReadFrameError(fmt % (err1, err2))
imageio.core.format.CannotReadFrameError: Could not read frame:

=== stderr ===
ffmpeg version 3.2.2 Copyright (c) 2000-2016 the FFmpeg developers
  built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/3.2.2 --enable-shared --enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-frei0r --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxvid --enable-opencl --disable-lzma --enable-libopenjpeg --disable-decoder=jpeg2000 --extra-cflags=-I/usr/local/Cellar/openjpeg/2.1.2/include/openjpeg-2.1 --enable-nonfree --enable-vda
  libavutil      55. 34.100 / 55. 34.100
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.100 / 57. 56.100
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/echeng/video.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: isom3gp43gp5
  Duration: 00:16:05.80, start: 0.000000, bitrate: 1764 kb/s
    Stream #0:0(eng): Audio: aac (LC) (mp4a / 0x6134706D), 8000 Hz, mono, fltp, 40 kb/s (default)
    Metadata:
      handler_name    : soun
    Stream #0:1(eng): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 640x480 [SAR 1:1 DAR 4:3], 1720 kb/s, 29.46 fps, 26.58 tbr, 90k tbn, 1k tbc (default)
    Metadata:
      handler_name    : vide
Output #0, image2pipe, to 'pipe:':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: isom3gp43gp5
    encoder         : Lavf57.56.100
    Stream #0:0(eng): Video: rawvideo (RGB[24] / 0x18424752), rgb24, 640x480 [SAR 1:1 DAR 4:3], q=2-31, 200 kb/s, 26.58 fps, 26.58 tbn, 26.58 tbc (default)
    Metadata:
      handler_name    : vide
      encoder         : Lavc57.64.101 rawvideo
Stream mapping:
... showing only last few lines ...
Past duration 0.760582 too large
Past duration 0.672997 too large
Past duration 0.822319 too large
Past duration 0.971657 too large
frame=25303 fps= 63 q=-0.0 size=22772700kB time=00:15:51.83 bitrate=195993.6kbits/s dup=18 drop=2752 speed=2.37x    
frame=25331 fps= 63 q=-0.0 size=22797900kB time=00:15:52.89 bitrate=195993.6kbits/s dup=18 drop=2755 speed=2.37x    
Past duration 1.411659 too large
Past duration 0.896416 too large
...

Due to the Past duration errors and this post, I suspect imageio is having trouble reading the video because of the odd 29.46 fps input frame rate.

Working directly with ffmpeg using this command, it fails to extract frames starting at frame 25676 as well:

ffmpeg -i video.mp4 frames/%06d.jpg

However, if I specify the input frame rate, it successfully extracts all the frames:

ffmpeg -r 29.46 -i video.mp4 frames/%06d.jpg

Is there a way to specify the input frame rate in get_reader as above so I can get past frame 25676? I’ve attempted to pass get_reader the ffmpeg_params keyword parameter, but it gives the following error:

In [2]: r = imageio.get_reader('video.mp4', ffmpeg_params=['-r','29.46'])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-3d2573506623> in <module>()
----> 1 r = imageio.get_reader('video.mp4', ffmpeg_params=['-r','29.46'])

/Users/echeng/anaconda2/lib/python2.7/site-packages/imageio/core/functions.pyc in get_reader(uri, format, mode, **kwargs)
    109 
    110     # Return its reader object
--> 111     return format.get_reader(request)
    112 
    113 

/Users/echeng/anaconda2/lib/python2.7/site-packages/imageio/core/format.pyc in get_reader(self, request)
    167             raise RuntimeError('Format %s cannot read in mode %r' % 
    168                                (self.name, select_mode))
--> 169         return self.Reader(self, request)
    170 
    171     def get_writer(self, request):

/Users/echeng/anaconda2/lib/python2.7/site-packages/imageio/core/format.pyc in __init__(self, format, request)
    216             self._request = request
    217             # Open the reader/writer
--> 218             self._open(**self.request.kwargs.copy())
    219 
    220         @property

TypeError: _open() got an unexpected keyword argument 'ffmpeg_params'

Apologies for the length of this issue. Thanks in advance!

[almarklein]

Yes, that could very well be it. As it is now, ffmpeg searches for either “fps” or “tbr”, and just picks the first. I guess it should prefer tbr instead.

You could try by modifying the lines here: https://github.com/imageio/imageio/blob/master/imageio/plugins/ffmpeg.py#L503-L507 Try adding matches.sort(key=lambda x: x[1]=='tbr', reverse=True).

Regarding the ffmpeg_params option, any ideas as to why this is failing? Is this an incorrect usage of that keyword parameter?

That looks good to me.

[almarklein]

In #321 I just added some fixes:

• tbr is preferred over fps
• added tests for the info parsing to make the parsing easier to maintain/improve while avoiding regressions (please submit the ffmpeg output for files for which it does not work correctly!)
• added an fps param to the reader, so that the fps can be overloaded, when you want a different (fixed) fps, or when imageio fails to detect it correctly.