How to extract timed metadata from HLS stream?

[REQUIRED] Searched documentation and issues I’ve searched through the ExoPlayer issues , and I’ve done some R&D & experimentation using different streams including Apple’s test stream (http://devimages.apple.com/samplecode/adDemo/ad.m3u8).

[END RESULT OF ALL EXPERIMENT]:

• Apple’s test stream (http://devimages.apple.com/samplecode/adDemo/ad.m3u8):- This stream not even playing in Exoplayer,

• https://vcloud.blueframetech.com/file/hls/13836.m3u8 :- This stream playing & @Override public void onMetadata(Metadata metadata) { Log.d("METADATA", metadata.toString()); } call back event triggered perfectly but other two call back not triggering. But, this stream is not as per the specification, data_alignment_indicator

[REQUIRED] Question I have an HLS video (sometimes live, sometimes not) and I wish to extract timed ID3 metadata frames from it.

The content can be viewed here: https://adinsertplayer.s3.ap-south-1.amazonaws.com/sampleoutput0302_modified_1_Live.m3u8

Alternatively, just the ID3 metadata packets can be seen in HLS file in following format:

#EXTM3U
#EXT-X-VERSION:1
#EXT-X-TARGETDURATION:17
#EXTINF:16.729778,
pre_process_output1.ts
#EXT-X-CUE-OUT:16.729778
#EXTINF:16.729778,
pre_process_output2.ts
#EXTINF:16.729778,
pre_process_output3.ts
#EXT-X-CUE-IN
#EXTINF:16.683333,
pre_process_output4.ts
#EXTINF:16.683333,
pre_process_output5.ts
#EXT-X-CUE-OUT:16.729778
#EXTINF:16.729778,
pre_process_output6.ts
#EXT-X-CUE-IN
#EXTINF:16.683333,
pre_process_output7.ts
#EXTINF:16.683333,
pre_process_output8.ts
#EXT-X-CUE-OUT:16.729778
#EXTINF:16.729778,
pre_process_output9.ts

Requirement is to trigger an ad event & play an in stream ima ad when we hit this metadata( #EXT-X-CUE-IN & #EXT-X-CUE-OUT). Here is what I’ve tried using demo-ima :

/*
 * Copyright (C) 2017 The Android Open Source Project
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.google.android.exoplayer2.imademo;

import android.content.Context;
import android.net.Uri;
import android.util.Log;
import com.google.android.exoplayer2.C;
import com.google.android.exoplayer2.C.ContentType;
import com.google.android.exoplayer2.ExoPlayer;
import com.google.android.exoplayer2.ExoPlayerFactory;
import com.google.android.exoplayer2.Player;
import com.google.android.exoplayer2.SimpleExoPlayer;
import com.google.android.exoplayer2.ext.ima.ImaAdsLoader;
import com.google.android.exoplayer2.metadata.Metadata;
import com.google.android.exoplayer2.metadata.MetadataOutput;
import com.google.android.exoplayer2.source.MediaSource;
import com.google.android.exoplayer2.source.MediaSourceFactory;
import com.google.android.exoplayer2.source.ProgressiveMediaSource;
import com.google.android.exoplayer2.source.TrackGroup;
import com.google.android.exoplayer2.source.TrackGroupArray;
import com.google.android.exoplayer2.source.ads.AdsMediaSource;
import com.google.android.exoplayer2.source.dash.DashMediaSource;
import com.google.android.exoplayer2.source.hls.HlsMediaSource;
import com.google.android.exoplayer2.source.smoothstreaming.SsMediaSource;
import com.google.android.exoplayer2.text.Cue;
import com.google.android.exoplayer2.text.TextOutput;
import com.google.android.exoplayer2.trackselection.TrackSelectionArray;
import com.google.android.exoplayer2.ui.PlayerView;
import com.google.android.exoplayer2.upstream.DataSource;
import com.google.android.exoplayer2.upstream.DefaultDataSourceFactory;
import com.google.android.exoplayer2.upstream.DefaultHttpDataSourceFactory;
import com.google.android.exoplayer2.util.Util;
import java.util.List;

/**
 * Manages the {@link ExoPlayer}, the IMA plugin and all video playback.
 */
/* package */ final class PlayerManager implements MediaSourceFactory, MetadataOutput, TextOutput,
    Player.EventListener {

  private final ImaAdsLoader adsLoader;
  private final DataSource.Factory dataSourceFactory;

  private SimpleExoPlayer player;
  private long contentPosition;

  public PlayerManager(Context context) {
    String adTag = context.getString(R.string.[ad_tag_url](https://pubads.g.doubleclick.net/gampad/ads?sz=640x480&iu=/124319096/external/single_ad_samples&ciu_szs=300x250&impl=s&gdfp_req=1&env=vp&output=vast&unviewed_position_start=1&cust_params=deployment%3Ddevsite%26sample_ct%3Dlinear&correlator=));
    adsLoader = new ImaAdsLoader(context, Uri.parse(adTag));
    dataSourceFactory =
        new DefaultDataSourceFactory(
            context, Util.getUserAgent(context, context.getString(R.string.application_name)));
  }

  public void init(Context context, PlayerView playerView) {
    // Create a player instance.
    player = new SimpleExoPlayer.Builder(context).build();
    adsLoader.setPlayer(player);
    playerView.setPlayer(player);

    // This is the MediaSource representing the content media (i.e. not the ad).
    String contentUrl = context.getString(R.string.content_url);
    MediaSource contentMediaSource = buildMediaSource(Uri.parse([contentUrl](https://adinsertplayer.s3.ap-south-1.amazonaws.com/sampleoutput0302_modified_1_Live.m3u8)));

    // Compose the content media source into a new AdsMediaSource with both ads and content.
    MediaSource mediaSourceWithAds =
        new AdsMediaSource(
            contentMediaSource,  /*adMediaSourceFactory= */ this, adsLoader, playerView);

    player.addMetadataOutput(this);
    player.addTextOutput(this);
    player.addListener(this);

    player.seekTo(contentPosition);
    player.prepare(mediaSourceWithAds);

    player.setPlayWhenReady(true);
  }


  @Override
  public void onTracksChanged(TrackGroupArray trackGroups, TrackSelectionArray trackSelections) {
    for (int i = 0; i < trackGroups.length; i++) {
      TrackGroup trackGroup = trackGroups.get(i);
      for (int j = 0; j < trackGroup.length; j++) {
        Metadata trackMetadata = trackGroup.getFormat(j).metadata;
        if (trackMetadata != null) {
          Log.d("METADATA TRACK", trackMetadata.toString());
        }
      }
    }
  }

  @Override
  public void onMetadata(Metadata metadata) {
    Log.d("METADATA", metadata.toString());
  }

  @Override
  public void onCues(List<Cue> cues) {
    Log.d("CUES", cues.toString());
  }

  public void reset() {
    if (player != null) {
      contentPosition = player.getContentPosition();
      player.release();
      player = null;
      adsLoader.setPlayer(null);
    }
  }

  public void release() {
    if (player != null) {
      player.release();
      player = null;
    }
    adsLoader.release();
  }

  // MediaSourceFactory implementation.

  @Override
  public MediaSource createMediaSource(Uri uri) {
    return buildMediaSource(uri);
  }

  @Override
  public int[] getSupportedTypes() {
    // IMA does not support Smooth Streaming ads.
    return new int[]{C.TYPE_DASH, C.TYPE_HLS, C.TYPE_OTHER};
  }

  // Internal methods.

  private MediaSource buildMediaSource(Uri uri) {
    @ContentType int type = Util.inferContentType(uri);
    switch (type) {
      case C.TYPE_DASH:
        return new DashMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
      case C.TYPE_SS:
        return new SsMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
      case C.TYPE_HLS:
        return new HlsMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
      case C.TYPE_OTHER:
        return new ProgressiveMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
      default:
        throw new IllegalStateException("Unsupported type: " + type);
    }
  }

}

The METADATA TRACK METADATA and CUES logs never happen.

Any clue or expedite help on this would highly appreciated 👍 Thanks in advance 😃

[ojw28]

First, it should be noted that Apple’s test stream is an example of server-side ad insertion. The purpose of the ID3 metadata is simply to indicate to the client when the ad starts and ends (and progress within it). It’s not intended that the metadata be used for actually triggering a client-side ad to be inserted.

Note:The existing demo-ima only having the pre-roll & post-roll client side ad integration guide.

ExoPlayer’s main demo app has additional examples that demonstrate mid-roll ad insertion as well. Note that you need to build the withExtensions build variant for them to work, as documented here.

Would be great if you could suggest the process to implement client side in-stream ad insertion, when we hit this metadata ID3 event.

This seems like custom behavior that you want for your application, and so is outside of scope for this issue tracker. If you really want to do this, then roughly speaking, I guess you should listen to the ID3 event via onMetadata, store the current playback position, re-prepare the player to play the ad, and then when the ad is completed re-prepare the player with the content MediaSource, restoring the stored playback position. This is something you’ll have to figure out for yourself, however.

[ojw28]

But other two callback onCues & ‘onTracksChanged’ not triggering.

Why are you expecting these to be triggered? The former is related to subtitles and is documented as such. The latter is related to tracks changing, which isn’t something that happens for the test stream. Metadata is output via onMetadata, which you’re seeing is being fired correctly.