Media player support

Implementation status

• .NET (#1356)
• Android (#1353)
• iOS (#1352)
• JavaScript (#1361) (in PR: #1864)
• UWP (#1354)
• Documentation

Problem

It is not possible to embed media elements in an Adaptive Card today. Media would initially support audio and video.

Scenario Requirements

• Authors need to be able to provide both video and audio. Some bots only send audio snippets.

• Media does NOT auto play.

• NEW: Media is NOT downloaded until user clicks play

  • That means in HTML, set preload="none"

• Hosts can choose to allow inline playback (or disable interactivity completely as usual).

• By default, without any host changes…

  • Inline playback is enabled
  • Videos/audio are fully functional, hosts don’t need to do any work!

• For video…

  • Authors can provide a thumbnail (poster) to be shown
  • If author didn’t provide a poster, host’s default poster is used (or blank/grey/black if host didn’t provide a default poster)
  • If host disabled interactivity, stop here
  • Otherwise, a default play button should be overlayed and centered horizontally/vertically (hosts can override this play button)
  • Clicking the entire thumbnail+play button should…
    • If inline playback allowed…
      • Overlay play button disappears
      • Media controls appear (native OS media controls)
      • Video starts playing
    • Else if inline not allowed…
      • Simply raises an event to the host, host is then responsible for playing the video

• For audio…

  • Authors can provide a thumbnail (poster) to be shown
  • If author didn’t provide a poster, host’s default poster is used (or blank/grey/black if host didn’t provide a default poster)
  • If host disabled interactivity, stop here
  • Otherwise, a default play button should be overlayed and centered horizontally/vertically (hosts can override this play button)
  • Clicking the entire thumbnail+play button should…
    • If inline playback allowed…
      • Overlay play button disappears
      • Media controls appear (native OS media controls)
      • Displayed poster should remain displayed
      • Audio starts playing
    • Else if inline not allowed…
      • Simply raises an event to the host, host is then responsible for playing the audio

• If hosts disabled inline playback, hosts need to know when the media element was tapped, so that they can handle playing the media themselves

• Hosts need to be able to limit the max download size of the http media, so that a 1 GB file doesn’t get downloaded, for example.

• Authors can provide AltText for accessibility purposes

• Authors can provide closed captions file

Asks

• Cortana
• Objects
• Skype (in header, not body)

What Facebook does…

• On Android…
  • When the video scrolls off screen, they pause the video
  • When user plays a different video, they pause the video
  • When user clicks already-playing video, it expands to full screen
• On Desktop web
  • When the video scrolls off screen, they continue playing the video
  • When the user plays a different video, they continue playing the existing video too (both play at once)
  • User can click expand button to expand to full screen
  • They do pre-load videos, but they display thumbnail first

Cortana requirements

• Events for…
  • Video start/stop
  • user pauses/stops/starts
  • user clicks away
  • video duration/elapse time
  • user skips ahead/back
  • user replays video
• Ability to play/pause/skip video
• Author option to auto-play

WebChat requirements

• Controls for…
  • Entering full screen (would probably get user asks if that’s not there)
  • Mute or volume control (probably only need mute)
  • Seek bar: At minimum -10s and +30s, depends on length, definitely in full screen people expect a way to seek
  • Captions (for accessibility)

Comps

Default state

The state when the card first appears. This is the state that appears regardless of inline playback allowed. If interactivity isn’t allowed, the play icon shouldn’t be rendered.

Inline playing/paused state

If inline playback allowed, clicking from the default state results in what’s seen below: Play button disappears, poster disappears, and native OS media player and controls should be used, no specific requirements on buttons available.

Open Questions

• Does the play/pause icon need to be adjustable?
• Auto Play. Decision: no
• We need to raise an event when the media is initially played from the poster preview, should we use the existing OnAction event or create a new one for OnMediaStarted?
• Captions?
  • Decision: Make this part of 1.1… need to add that to spec schema
• Max download size: what if someone attaches a 100MB mp4? Do we need to revisit host config max download sizes? Decision: Yes, we’ve added a media max size to host config which renderers must respect
• Does the media element need to be declared in the header? Do hosts only want the video at the top of the card, and not in the free-form body?
  • We already know that Skype only wants it at the top
  • Andrew to reach out to Cortana/WebChat (7/30)
• Cortana needs optional auto play (author specified)
  • Decision: Pushed to 1.2
• Allow hosts to specify different default audio poster
  • Decision: Pushed to 1.2

Example Comps

Spec

Schema Changes

Media

Property	Type	Required	Description
type	“Media”	true	Must be “Media”
sources	MediaSource[]	true	The media sources by MIME type. The first recognized format will be used. It is not supported to mix video and audio in the same Media element.
poster	string	false	Strongly recommended thumbnail preview image for media. If not provided, host’s default poster will be used (or blank screen shown).
altText	string	false	Optional text to include information about the preview image for users who are visually impaired

MediaSource

Property	Type	Required	Description
mimeType	string	true	The MIME type of the media. E.g., “video/mp4”. The primary portion must be either “video” or “audio”
url	string	true	The URL of the media source

Example

{
  "type": "Media",
  "poster": "http://...",
  "sources": [
   { "mimeType": "video/mp4", "url": "http://.mp4" },
   { "mimeType": "...", "url": ".ogg" }
  ]
},
{
  "type": "Media",
  "poster": "http://...",
  "sources": [
   { "mimeType": "audio/mpeg", "url": ".ogg" },
   { "mimeType": "...", "url": ".ogg" }
  ]
}

Host Config

The following options allow hosts to override the defaultPoster if the card does not include a poster. It also allows customization of the play overlay button before the media has played.

{
  "media": {
      "defaultPoster": "http://...",
      "playButton": "http://...",
      "allowInlinePlayback": true|false,
      "maxMediaSize": 1024 // 1 MB
   }
}

Media states

Each platform represents media states in different ways. Listing them out here to determine the best union for what we want to use.

UWP	Android	iOS (AVPlayerItem)
MediaOpened	setOnPreparedListener() (media loaded and ready)	AVPlayerItem.Status.readyToPlay
Buffering	MEDIA_INFO_BUFFERING_START, MEDIA_INFO_BUFFERING_END	AVPlayerItemPlaybackStalled
Closed (no media)
Opening
Paused	Override pause() in MediaController	AVPlayerTimeControlStatusPaused
Playing	MEDIA_INFO_VIDEO_RENDERING_START	AVPlayerTimeControlStatusPlaying
Stopped	setOnCompletionListener() (end of media reached)	AVPlayerItemDidPlayToEndTimeNotification

Renderer Requirements

• A Renderer MUST enumerate the list of sources and use the first recognizable mimeType to determine the type of media (audio or video)
• A renderer MUST display video/audio as described in the scenario requirements in For video… and For audio…
• A renderer MUST raise OnMediaClicked event (using whatever local naming schema makes sense) ONLY if host disabled inline playback, so that host can display the media
• A renderer MUST respect the maxMediaSize from host config to prevent accidental download of excessively large files.
• A renderer MUST provide play/pause controls. It SHOULD include volume controls if possible. Using native OS media controls is fine.
• If sources contains an invalid mix of MIME types (audio and video) the element MUST be dropped and a warning SHOULD be logged.
• A renderer MUST support and follow http redirects in the URL source

Media features intentionally left out

The following options are not provided to card authors.

"controls": true, // implicit true
"loop": true, // no auto loop
"autoPlay": true, // no auto play

Reference

For reference, HTML supports these:

• MP4 = MPEG 4 files with H264 video codec and AAC audio codec
• WebM = WebM files with VP8 video codec and Vorbis audio codec
• Ogg = Ogg files with Theora video codec and Vorbis audio codec

Downlevel Impact

High

Preview image will display, but any actionable media will not play. It is important to not display play button when on a 1.0 renderer to limit confusion.

1.2 features

• Optional auto-play (hosts have supreme authority, authors can choose whether their video should auto-play)
  • This is needed by Cortana, when you ask “Show me my home security camera”, you want that to auto-play, but when you say “Videos of cats”, you don’t want those to auto-play
• Host ability to provide a different default poster for audio (defaultAudioPoster?)
• Host APIs to access native media element, so they can access all events (like play/pause/seek), and so that they can programmatically play/pause
• Potentially future scenario: Host events for OnMediaStarting, OnMediaPaused and OnMediaStopped events to allow hosts to know the state of the media
  • This could instead be solved by accessing native media element?
• Hosts need to know whether the media is currently playing, so that for scenarios like notifications, they know to keep the notification on-screen rather than letting it time out. Therefore, they need to know when the media started playing (hold on screen). And they need to know when the media was paused (possibly release hold on screen?) and when the media ended playing (definitely release the hold on screen?).
  • Could use the two above features for that?
• Host option to choose to enable pre-load (so that video plays instantly when clicked, but therefore uses more data)

[danmarshall]

cc: @compulim

[paulcam206]

Documentation is done and in the mahiding/site1.1 branch.