[RTDB] High memory usage when observing large datasets

[REQUIRED] Step 2: Describe your environment

• Android Studio version: Android Studio Dolphin | 2021.3.1 Canary 4
• Firebase Component: Database
• Component version: 29.0.4 (BOM)

[REQUIRED] Step 3: Describe the problem

At the root, the problem is that whenever Im signed into Firebase RTDB, my app stutters at times. Its clearly noticeable, and after running the memory profiler (in a release build) Ive noticed that the memory usage grows a lot whenever the firebase integration is active - and remains high until the app is closed / you sign out.

To compare, normally my app uses about 40 mb of memory, and when signed in that number grows to 90 mb for a completely new & empty account, and if theres a lot of data stored, it sits at about 160 mb, or 4x the normal usage.

I dont know if its fair/correct to compare it with Google Fit considering how different they are, but I will say that Fit behaves “correctly” per my perspective - the memory does grow by about 30-40 mb, but as it grows towards 40 it always drops back down to 30, whereas firebase just continiously grows and grows, and then never drops.

Im observing 6 different references for as long as the app is active (and youre signed in), each reference resides under path/uid/, so “exercises/my_id/…list”. The list in my code is 2200 elements long, and Im observing the whole thing as Id like to download updates to any of the elements as soon as it happens. Every element stored in firebase has a version field associated with it, which I compare with locally to find out if the “downloaded” data is actually newer than whats already stored locally.

My hope in posting this is that its a bug in firebase. That something keeps things around in memory for much longer than needed. At the worst, Ive made some mistakes in my implementation - Im open to that too!

Steps to reproduce:

Sign in, and add ChildEventListener to 6 different references.

Relevant Code:

For each reference in firebase, this code runs.

val reference = database
            .getReference(branch.path()) // "exercises"
            .child(token.userId()) // "uid"

// Scope = IO + SupervisorJob()
scope.launch {
        observe().collect{ 
        // Even if the collect is empty, memory usage remains the same 
    }
}

  private  fun observe(): Flow<SyncNotification> {
        return callbackFlow {
            val observer = SyncObserver(this)

            reference.addChildEventListener(observer)

            awaitClose {
                reference.removeEventListener(observer)
            }
        }.buffer(UNLIMITED)
    }

SyncNotification contains the DataSnapshot and a token that describes whether the notification is UPDATE/DELETE.

The SyncObserver just emits values everytime onChild.. is called.

[zoltish]

Gotcha, @argzdev! Ill focus my efforts on the indexing then 😃 I totally missed that in the docs, good to know that its there. Ill get back to you when Ive tried the indexes, it will likely take a few days as Im in the middle of some tech challenges!

[argzdev]

Hi @zoltish, thanks for the extra details. I was able to reproduce the same issue. It looks like there is a 30 ~ 40mb overhead when a ChildEventListener is added.

Using the profiler I tried testing with the following:

1. starting memory at 86.4, after adding listener it increased to 116.3 (limitToLast 20) ~30 mb overhead
2. starting memory at 70.4, after adding listener it increased to 109.4 (no limit) ~30 mb overhead
3. starting memory at 73, after adding listener it increased to 108 mb (limitToLast 10) ~30 mb overhead
4. starting memory at 74.4, after adding listener it increased to 110.5 mb (limitToLast 1) ~30 mb overhead

Relevant code:

fun addChildEventListener(database: DatabaseReference){
        Log.d(TAG, "addChildEventListener: ")
        childEventListener = object : ChildEventListener {
            override fun onChildAdded(snapshot: DataSnapshot, previousChildName: String?) {
                Log.d(TAG, "onChildAdded:" + snapshot.key!!)
            }

            override fun onChildChanged(snapshot: DataSnapshot, previousChildName: String?) {
                Log.d(TAG, "onChildChanged: ${snapshot.key}")
            }

            override fun onChildRemoved(snapshot: DataSnapshot) {
                Log.d(TAG, "onChildRemoved:" + snapshot.key!!)
            }

            override fun onChildMoved(snapshot: DataSnapshot, previousChildName: String?) {
                Log.d(TAG, "onChildMoved:" + snapshot.key!!)
            }

            override fun onCancelled(error: DatabaseError) {
                Log.w(TAG, "postComments:onCancelled", error.toException())
            }
        }
//        database.child("workout").limitToLast(1).addChildEventListener(childEventListener)
        database.child("workout").addChildEventListener(childEventListener)
        Log.d(TAG, "addChildEventListener: ")

I’ll try to consult with an engineer and see what can be done here.