Thumbnail track support

This was previously discussed briefly in google groups but I figure it’s better suited here, so I’m moving it.

The initial question from @wader:

Hello

Im researching different ways for embedding custom data in DASH manifests. In my case i would like to add custom information for seek thumbnails. For example one way would be to add a AdaptationSet with contentType like image/jpeg etc. But currently it seems shaka is quite picky with what it finds in the manifest so i run into various errors (DASH_EMPTY_ADAPTATION_SET, DASH_UNSUPPORTED_CONTAINER, …) also this would also require some kind of API for accessting the raw manifest.

Is there already a way of doing this? if not would something like this be interesting for other people and fit into shaka?

-Mattias

And the response from @joeyparrish:

Hi Mattias,

It seems to me that AdaptationSets are not a great place for a custom extension because they are already used for something standard. It is common for DASH extensions to use their own unique elements in their own namespace to avoid conflicting with standard parts of DASH.

For example, you might do something like this:

<MPD xmlns="urn:mpeg:dash:schema:mpd:2011"
    xmlns:myApp="http://foo.bar/myApp"
    minBufferTime="PT2S" type="static">
  <Period id="0" duration="PT60S">
    <AdaptationSet id="0" contentType="video/mp4">
      <myApp:seekThumbs mimeType="image/jpeg">
        <myApp:thumb time="0" img="thumb-0.jpg" />
        <myApp:thumb time="2" img="thumb-2.jpg" />
        <myApp:thumb time="4" img="thumb-4.jpg" />
        ...
        <myApp:thumb time="56" img="thumb-56.jpg" />
        <myApp:thumb time="58" img="thumb-58.jpg" />
      </myApp:seekThumbs>
      <Representation ... />
      <Representation ... />
      <Representation ... />
    </AdaptationSet>
  </Period>
</MPD>

As for accessing that from Shaka Player, we don’t currently have an API that is exactly for this, but we do have network request and response filters. You could listen for manifest responses and get access to the XML before it goes to the manifest parser, but correlating that to the internal manifest representation might get tricky.

Another approach would be to create your own manifest parser that inherits from our DASH parser. All manifest parsers are plugins, so you can easily register one that replaces ours. You can also customize the build to include your plugin: http://shaka-player-demo.appspot.com/docs/api/tutorial-plugins.html

[wilaw]

Hey Joey

The DASH IF view has been that thumbnails should be generated from trick view tracks, rather than adding hundreds of jpegs to a package. Yes, the thumbnail would need to be rendered by a videoElement, either offscreen (and copied to a canvas element) or on-screen, with the thumb video element superimposed over the primary element. The point about encryption is a fair point, although it would apply equally to the jpeg images themselves. In reality, small thumb videos that are keyframe-only would not require encryption.

Your proposal for a new EssentialProperty looks clean and not the least bit crazy ☺ My only comment is that the @frameRate attribute would not be required. I will take it to DASH IF. I realize that playing another video requires essentially another player instance and this has a lot more overhead than loading and displaying some jpegs, so it’s pretty clear to me personally that in a MSE player I would rather implement the jpeg approach than the second video element approach.

Have you made a sample video with this adaption set? If not, we can collaborate on a Buck Bunny sample.

[avelad]

Another sample stream:

https://image.roku.com/ZHZscHItc2Ft/roku/trickplay/bbb-with-multiple-tiled-thumbnails.mpd