Missing Collection notifications

We have huge problems on CI with tests timing out with what looks like notifications being skipped. I was able to somewhat reproduce it locally, but it looks very timing dependant and @beeender has not been able to verify on his machine:

Reproduce

1. Install ROS locally npm install -g realm-object-server@rc
2. Disable startSyncServer/stopSyncServer in StandardIntegrationTest class
3. Run PermissionManagerTests#getPermissions_updatedWithNewRealms once from Android Studio to install test app on the device and see that it runs green
4. Goto commandline and run I’m running this script:

#!/bin/bash
adb shell am instrument -w -r   -e debug false -e class io.realm.PermissionManagerTests#getPermissions_updatedWithNewRealms io.realm.test/android.support.test.runner.AndroidJUnitRunner
while [ $? -eq 0 ]; do
    adb shell am instrument -w -r   -e debug false -e class io.realm.PermissionManagerTests#getPermissions_updatedWithNewRealms io.realm.test/android.support.test.runner.AndroidJUnitRunner
done

with

while ./test.sh; do :; done

1. Prepare to wait for a while. ROS might eventually run out of filedescriptors on Mac, in that case, restart server and script
2. When the bug is caught, the test will timeout and show the below output. The log statements probably doesn’t make much sense, but the important ones are the before_advance and after_advance which is used to trigger notifications.

This is the code in after_advance:

void CollectionNotifier::after_advance()
{
    __android_log_print(ANDROID_LOG_ERROR, "OS", "After advance: Size %d", m_callbacks.size());
    for_each_callback([&](auto& lock, auto& callback) {
        if (callback.initial_delivered && callback.changes_to_deliver.empty()) {
            __android_log_print(ANDROID_LOG_ERROR, "OS", "After advance: Ignore callback (%s, %s)", callback.initial_delivered?"true":"false", callback.changes_to_deliver.empty()?"true":"false");
            return;
        }
        callback.initial_delivered = true;

        auto changes = std::move(callback.changes_to_deliver);
        // acquire a local reference to the callback so that removing the
        // callback from within it can't result in a dangling pointer
        auto cb = callback.fn;
        lock.unlock();
        __android_log_print(ANDROID_LOG_ERROR, "OS", "After advance: Trigger callback (%s, %s)", callback.initial_delivered?"true":"false", callback.changes_to_deliver.empty()?"true":"false");
        cb.after(changes);
    });
}

Logs

Successfull attempt
10-12 10:02:45.266 8690-8709/? E/OS: Add callback
10-12 10:02:45.267 8690-8709/? E/OS: After advance: Size 1
10-12 10:02:45.267 8690-8709/? E/OS: After advance: Trigger callback (true, true)
10-12 10:02:45.268 8690-8709/? E/REALM_JAVA: Size: 2, State: ACTIVE
10-12 10:02:45.268 8690-8709/? E/OS: Remove callback
10-12 10:02:45.320 8690-8709/? E/OS: Add callback
10-12 10:02:45.321 8690-8709/? E/OS: After advance: Trigger callback (true, true)
10-12 10:02:45.321 8690-8709/? E/OS: After advance: Size 1
10-12 10:02:45.321 8690-8709/? E/OS: After advance: Ignore callback (true, true) <!--
10-12 10:02:45.321 8690-8709/? E/OS: After advance: Size 0
10-12 10:02:45.333 8690-8709/? E/OS: Before advance: Size 1
10-12 10:02:45.333 8690-8709/? E/OS: Before advance: Size 0
10-12 10:02:45.333 8690-8709/? E/OS: After advance: Size 1
10-12 10:02:45.334 8690-8709/? E/OS: After advance: Trigger callback (true, true)
10-12 10:02:45.336 8690-8709/? E/REALM_JAVA: Testlistener: 3
10-12 10:02:45.338 8690-8709/? E/OS: After advance: Size 0


Failed attemp:
10-12 10:09:37.937 13397-13416/? E/OS: Add callback
10-12 10:09:37.937 13397-13416/? E/OS: After advance: Size 1
10-12 10:09:37.937 13397-13416/? E/OS: After advance: Trigger callback (true, true)
10-12 10:09:37.938 13397-13416/? E/REALM_JAVA: Size: 2, State: ACTIVE
10-12 10:09:37.938 13397-13416/? E/OS: Remove callback
10-12 10:09:37.988 13397-13416/? E/OS: Add callback
10-12 10:09:37.988 13397-13416/? E/OS: After advance: Trigger callback (true, true)
10-12 10:09:37.988 13397-13416/? E/OS: Before advance: Size 1
10-12 10:09:37.988 13397-13416/? E/OS: Before advance: Size 0
10-12 10:09:37.989 13397-13416/? E/OS: After advance: Size 1
10-12 10:09:37.989 13397-13416/? E/OS: After advance: Ignore callback (true, true) <!-- Did it move here? 
10-12 10:09:37.989 13397-13416/? E/OS: After advance: Size 0 
<!-- Test times out here
10-12 10:09:38.991 13397-13416/io.realm.test E/REALM_JAVA: Real size: 3 <!-- Opening the Realm on another thread verified that the change is in the DB

I have narrowed it down to https://github.com/realm/realm-object-store/blob/master/src/impl/collection_notifier.cpp#L326 but I have no explanation as to why it is happening:

Open questions:

• after_advance is called twice. It is unclear why. Probably because two Results exist, but it doesn’t look like I create two. Conclussion: Doesn’t look related
• When failing, it looks like code inside the callback isn’t triggered. Add more logging to verify how far it gets. Conclussion: Done, see https://github.com/realm/realm-java/issues/5413#issuecomment-336212117
• https://github.com/realm/realm-java/issues/5411 did not catch this

[bdash]

I think process_available_sync is a typo for process_available_async?

[cmelchior]

Folding this into #5507 We have reports from multiple developers that point to notifications sometimes being skipped, this is another datapoint