False stats report by Jicofo

Description

Some stats reports by Jicofo are not the real one

Current behavior

After a test with hundreds of room and users, the stats report by Jicofo are not the real one. It appears that when the load decrease, users and conf, some stats stays high, and so they are false, until a restart of Jicofo.

Example of false stats reported : Here in reality we have 4 conferences and 16 participants. The number of JVB have been increased and decreased, don’t know if it is releavant.

[{“jibri_detector”:{“count”:0,“available”:0},“largest_conference”:4,“total_sip_call_failures”:0,“live_streaming_pending”:0,“conference_sizes”:[0,70,25,6,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],“total_conferences_created”:246,“total_recording_failures”:0,“threads”:272,“sip_call_active”:0,“xmpp”:{“total_recv”:344572,“total_sent”:212504},“bridge_failures”:{“participants_moved”:81,“bridges_removed”:81},“total_live_streaming_failures”:0,“recording_pending”:0,“avg_allocate_channels_req_time_nanos”:0,“total_participants”:4198,“participant_notifications”:{“ice_failed”:0,“request_restart”:19},“live_streaming_active”:0,“bridge_selector”:{“total_least_loaded_in_region”:0,“total_split_due_to_load”:0,“total_not_loaded_in_region_in_conference”:0,“total_least_loaded_in_region_in_conference”:0,“total_not_loaded_in_region”:0,“total_split_due_to_region”:0,“bridge_count”:3,“operational_bridge_count”:3,“total_least_loaded_in_conference”:0,“total_least_loaded”:0},“sip_call_pending”:0,“conferences”:103,“participants”:146,“recording_active”:0}]

Expected Behavior

The stats reflect the reality.

Possible Solution

No idea yet

Steps to reproduce

Test with torture a Jitsi infra and see what appened after 1 hours of bench.

Environment details

Jicofo with org.jitsi.jicofo.BridgeSelector.BRIDGE_SELECTION_STRATEGY=SplitBridgeSelectionStrategy

Octo activated JVB autoscale, meaning the number of JVB has increase and decrease.

[goacid]

All stats for jicofo and jvb when no more conf in file attached : stat.txt wrong stat : “conferences” “participants” “conference_sizes" array, false value inside.

Those stats have grown until more thant 100 for conferences, and more than 1000 for participants, And when number of participants and conf decrease until 0, stats still report a value higher than real one, I called it ‘blocking value’. If new participants arrive, the participant stats report a value : blocking value + new partcipants … Same for conference.

[bgrozev]

That’s strange, I don’t see any suspicious changes between 654 and 660. Follow up if it happens again.