Adaptive bitrate doesn't work

Hi @pedroSG94 I’m testing BitrateAdapter and I’ve implemented it the same way as it is in the example. When I switch from high-speed WiFI to 3G the bitrate is not adapted properly and it is very high. The same situation when I start streaming from 3G. The tested upload speed of 3G is ~1.5Mb/s, however, adapted bitrate is ~5Mb/s (which is my max bitrate) resulting in lags on stream.

Maybe I’m doing something wrong, but I’ve checked the source code and the bitrate that you pass to onNewBitrateRtmp(bitrate) callback may not be correct. It’s taken from RtmpConnection.publishVideoData(size) just after you write the data to the socket. I think that socket has an internal buffer and the data size you write to socket’s outputstream doesn’t have to be equal to the size of data that has been uploaded. I’ve compared this bitrate value to the value from TrafficStats.getTotalTxBytes(), which shows the amount of data that has been uploaded and there are discrepancies between them. Here is the code to quickly compare it

private long bitrateSum = 0;
  private long initialTotalUploadBytes = 0;
  private long previousCheckTimeMs = 0L;
  private long lastTotalUploadBytes = 0;

  private BitrateAdapter adapter = new BitrateAdapter(new BitrateAdapter.Listener() {
    @Override
    public void onBitrateAdapted(int bitrate) {
      Log.d("lol2", "bitrate adapted: " + bitrate);
      rtmpCamera2.setVideoBitrateOnFly(bitrate);
    }
  });

  @Override
  public void onConnectionSuccessRtmp() {
    adapter.setMaxBitrate(5 * 1024 * 1024);
    initialTotalUploadBytes = TrafficStats.getTotalTxBytes();
  }

  @Override
  public void onNewBitrateRtmp(long bitrate) {
    Log.d("lol", "onNewBitrate: " + bitrate);
    adapter.adaptBitrate((int) bitrate);

    bitrateSum += bitrate;
    Log.d("lol", "Bitrate sum so far: " + bitrateSum / 1024f / 1024f + " Mb");

    long uploadedBytesSoFar = TrafficStats.getTotalTxBytes() - initialTotalUploadBytes;
    Log.d("lol", "Real upload so far: " + uploadedBytesSoFar * 8f / 1024f / 1024f + " Mb");

    long bytesDiff = uploadedBytesSoFar - lastTotalUploadBytes;
    long nowMs = System.currentTimeMillis();
    int timeDiff = (int) ((nowMs - previousCheckTimeMs) / 1000f);
    float realUploadSpeed = bytesDiff * 8f / timeDiff / 1024f / 1024f;
    Log.d("lol", "Real upload speed: " + realUploadSpeed + " Mb/s");

    previousCheckTimeMs = nowMs;
    lastTotalUploadBytes = uploadedBytesSoFar;
  }

There is also one interesting thing that I don’t fully understand, maybe you could explain. Being on high-speed WiFI, setting rtmpCamera2.setVideoBitrateOnFly(20 * 1024 * 1024); determines the video bitrate but also the upload speed. I mean, when I set it to 20Mb/s like above the upload speed will be 20Mb/s, when I set it to 2Mb/s the upload speed will be 2Mb/s. So is there any contract in the RTMP that video quality affects upload speed? Can’t I upload 2Mb/s quality video with 5Mb/s speed? Or maybe it depends on the internal buffer size?

[mkrn]

I took a deep dive into the adaptive bitrate in this library (latest version, 2.1.4), and I’ve been able to fix most of the adaptive bitrate problems. Here’s what I found:

Problems with default BitrateAdapter

• Average was not properly calculated
• Bandwidth in the last second (onNewBitrateRtmp) is calculated as the total data size send to the socket, with packet headers and audio packets. Yet bitrate is set for video encoder only (audio bitrate set separately).
• If there is congestion, need to immediately reduce bitrate, to minimize drop frames and even worse congestion
• Keep the cache size (queue length) to the minimum, so congestion can be quickly discovered (it’s 20% of the cache size), and so the backlog of old frames is easier to push out. I suggest 30.
• It is necessary to account for what was in the queue for bandwidth calculation before raising bitrate, because of the bitrate adjustments. Wait for a few cycles after lowering to raise again.

Notice: All of this makes sense with Constant Bitrate only! Here’s my version of BitrateAdapter that accounts for audio bitrate, packet overhead, and reacts faster to congestion.

package pro.eventlive;

import android.util.Log;

public class BitrateAdapter {

  public interface Listener {
    void onBitrateAdapted(int bitrate);
  }

  private int maxBitrate;
  private int minBitrate = 100 * 1024;
  private int audioBitrate = 96 * 1000; // TODO: This should be passed into the class
  private int oldBitrate;
  private int averageBitrate;
  private int cont;
  private int cyclesSinceReduced = 0;
  private Listener listener;
  private float decreaseRange = 0.8f; //20%
  private float increaseRange = 1.2f; //20%

  private int decreaseBy = 256 * 1024; 
  private int increaseBy = 128 * 1024; 
  private int packetOverhead = 15 * 1024; // Magic number

  public BitrateAdapter(Listener listener) {
    this.listener = listener;
    reset();
  }

  public void setMaxBitrate(int bitrate) {
    this.maxBitrate = bitrate;
    this.oldBitrate = bitrate;
    reset();
  }

  public void adaptBitrate(long actualBitrate) {
    averageBitrate += actualBitrate;
    if (cont > 0) {
        averageBitrate /= 2;
    }
    cont++;
    if (cont >= 3) { // lowered the measurement interval from 5s to 3s
      if (listener != null && maxBitrate != 0) {
        listener.onBitrateAdapted(getBitrateAdapted(averageBitrate));
        reset();
      }
    }
  }

  /**
   * Adapt bitrate on fly based on queue state.
   */
  public void adaptBitrate(long actualBitrate, boolean hasCongestion) {
    
    averageBitrate += actualBitrate;
    if (cont > 0) {
        averageBitrate /= 2;
    }
    cont++;

    if (hasCongestion) {
        // Immediately react with adjustments
        listener.onBitrateAdapted(getBitrateAdapted(averageBitrate, hasCongestion));
        reset();
    }

    if (cont >= 3) { // lowered the measurement interval from 5s to 3s
      if (listener != null && maxBitrate != 0) {
        cyclesSinceReduced++;
        listener.onBitrateAdapted(getBitrateAdapted(averageBitrate, hasCongestion));
        reset();
      }
    }
  }

  // Version that doesn't take the queue size into account
  private int getBitrateAdapted(int averageBw) { 
    if (averageBw >= maxBitrate) { //You have high speed and max bitrate. Keep max speed
      oldBitrate = maxBitrate;
    } else if (averageBw <= oldBitrate * 0.9f) { //You have low speed and bitrate too high. Reduce bitrate
      oldBitrate = Math.max(averageBw - audioBitrate - decreaseBy, minBitrate);
    } else if (averageBw >= oldBitrate) { //You have high speed and bitrate too low. Increase bitrate 
      oldBitrate = Math.min(oldBitrate + increaseBy, maxBitrate);
    }
    // keep it otherwise
    return oldBitrate;
  }

  private int getBitrateAdapted(int averageBw, boolean hasCongestion) { 
    // what we expect should have been sent over the network
    int expectedBandwidth = oldBitrate + audioBitrate + packetOverhead;
    // Explicitly has congestion in the queue or average bandwidth 90% less than expected
    if (hasCongestion || (averageBw <= expectedBandwidth * 0.9f)) {
        // Reduce! decreaseBy is added for when there's congestion but bitrate - audioBitrate is within current bitrate
        oldBitrate = Math.max(averageBw - audioBitrate - decreaseBy, minBitrate);
        cyclesSinceReduced = 0; // wait a few cycles to let the higher bitrate frames in the queue pass through
    } else if (averageBw >= expectedBandwidth&& cyclesSinceReduced >= 3) { 
        // When fully recovered, attempt to increase by a bit
        oldBitrate = Math.min(oldBitrate + increaseBy, maxBitrate);
    }

    // keep it otherwise
    return oldBitrate;
  }

  public void reset() {
    averageBitrate = 0;
    cont = 0;
  }

  public float getDecreaseRange() {
    return decreaseRange;
  }

  /**
   * @param decreaseRange in percent. How many bitrate will be reduced based on oldBitrate.
   * valid values:
   * 0 to 100 not included
   */
  public void setDecreaseRange(float decreaseRange) {
    if (decreaseRange > 0f && decreaseRange < 100f) {
      this.decreaseRange = 1f - (decreaseRange / 100f);
    }
  }

  public float getIncreaseRange() {
    return increaseRange;
  }

  /**
   * @param increaseRange in percent. How many bitrate will be increment based on oldBitrate.
   * valid values:
   * 0 to 100
   */
  public void setIncreaseRange(float increaseRange) {
    if (increaseRange > 0f && increaseRange < 100f) {
      this.increaseRange = 1f + (increaseRange / 100f);
    }
  }
}

Suggestions to improve further:

• If the queue is congested and not sending the frames, then calculateBitrate will not be invoked and onNewBitrateRtmp will not be called and bitrate would not be adjusted. Instead, I suggest using a timer or callback for congestion.
• When the queue is full we should not discard audio packets because they’re small compared to video. Ideally, we should not discard Keyframes because without them the picture falls apart. How to do it? Two separate queues? Or queue clean up? It is better to miss some video frames than audio - because you will hear pauses in the audio but may not notice low frame rate.

[mkrn]

@marcin-adamczewski Actually I end up not relying on upload speed calculation at all in adaptive bitrate. What we do now is - if there is congestion (i.e. queue is 15% full) then we immediately drop the target bitrate by set amount, then re-evaluate every 2s, if there’s no congestion raise by lower amount. Basically adding and subtracting to the same target, instead of manipulating the value received from upload speed calculation.