Pinot Ingestion Rate Sudden Drop to 0

We ingested data from Kafka into six Pinot real-time tables. A few hours back, the ingestion on all the tables dropped to 0. Server logs:

expected=ONLINE, found=OFFLINE, creationTime=1632409068678, modifiedTime=1632409072718, version=13, waitingFor=CurrentStateMatch, resource=spanEventView_REALTIME, numResourcesLeft=3, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:58:15.903 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=3, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:58:25.914 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409104146, version=16, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=3, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:58:35.953 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:58:45.969 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:58:55.977 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:05.987 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:15.995 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:26.019 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:36.027 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:46.040 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 14:59:56.050 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409072726, version=8, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:00:06.061 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
Exception in thread "rawTraceView__0__1455__20210923T1346Z" Exception in thread "rawServiceView__0__3350__20210923T1224Z" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
	at org.apache.pinot.core.realtime.impl.dictionary.StringOffHeapMutableDictionary.index(StringOffHeapMutableDictionary.java:57)
	at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.updateDictionary(MutableSegmentImpl.java:521)
	at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:466)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:497)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
	at java.lang.Thread.run(Thread.java:748)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
	at org.apache.pinot.core.realtime.impl.dictionary.BytesOffHeapMutableDictionary.getMinVal(BytesOffHeapMutableDictionary.java:115)
	at org.apache.pinot.core.realtime.impl.dictionary.BytesOffHeapMutableDictionary.getMinVal(BytesOffHeapMutableDictionary.java:34)
	at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.updateDictionary(MutableSegmentImpl.java:525)
	at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:466)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:497)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "backendEntityView__2__23061__20210923T1352Z" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
	at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.addNewRow(MutableSegmentImpl.java:663)
	at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:475)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:497)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
	at java.lang.Thread.run(Thread.java:748)
Exception in thread "serviceCallView__2__9949__20210923T1352Z" Exception in thread "spanEventView__5__869__20210923T1454Z" Exception in thread "spanEventView__1__33514__20210923T1454Z" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:1845)
	at groovy.lang.MetaClassImpl.getProperty(MetaClassImpl.java:3773)
	at org.codehaus.groovy.runtime.InvokerHelper.getProperty(InvokerHelper.java:195)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.getProperty(ScriptBytecodeAdapter.java:469)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.getPropertySafe(ScriptBytecodeAdapter.java:477)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.getPropertySpreadSafe(ScriptBytecodeAdapter.java:485)
	at Script1.run(Script1.groovy:1)
	at org.apache.pinot.core.data.function.GroovyFunctionEvaluator.evaluate(GroovyFunctionEvaluator.java:95)
	at org.apache.pinot.core.data.recordtransformer.ExpressionTransformer.transform(ExpressionTransformer.java:95)
	at org.apache.pinot.core.data.recordtransformer.CompositeTransformer.transform(CompositeTransformer.java:82)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:491)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
	at java.lang.Thread.run(Thread.java:748)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
	at java.util.HashMap.putVal(HashMap.java:642)
	at java.util.HashMap.put(HashMap.java:612)
	at org.apache.pinot.spi.data.readers.GenericRow.putValue(GenericRow.java:127)
	at org.apache.pinot.plugin.inputformat.avro.AvroRecordExtractor.extract(AvroRecordExtractor.java:70)
	at org.apache.pinot.plugin.inputformat.avro.AvroRecordExtractor.extract(AvroRecordExtractor.java:38)
	at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:110)
	at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:115)
	at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:51)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:471)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
	at java.lang.Thread.run(Thread.java:748)
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
	at org.apache.pinot.core.realtime.impl.dictionary.BaseOffHeapMutableDictionary.indexValue(BaseOffHeapMutableDictionary.java:361)
	at org.apache.pinot.core.realtime.impl.dictionary.StringOffHeapMutableDictionary.index(StringOffHeapMutableDictionary.java:61)
	at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.updateDictionary(MutableSegmentImpl.java:521)
	at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:466)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:497)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:402)
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:538)
	at java.lang.Thread.run(Thread.java:748)
2021/09/23 15:00:16.067 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=rawServiceView__0__3342__20210922T1809Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068648, modifiedTime=1632409215577, version=11, waitingFor=CurrentStateMatch, resource=rawServiceView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:00:26.077 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=2, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:00:36.093 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:00:46.103 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409107080, version=19, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:00:56.113 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23025__20210922T1909Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409255125, version=22, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:01:06.122 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Sleep for 10000ms as service status has not turned GOOD: PinotServiceManagerStatusCallback:Started;MultipleCallbackServiceStatusCallback:IdealStateAndCurrentStateMatchServiceStatusCallback:partition=backendEntityView__2__23040__20210923T0619Z, expected=ONLINE, found=OFFLINE, creationTime=1632409068635, modifiedTime=1632409266062, version=48, waitingFor=CurrentStateMatch, resource=backendEntityView_REALTIME, numResourcesLeft=1, numTotalResources=5, minStartCount=5,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;;
2021/09/23 15:01:16.215 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Service status is GOOD after 190391ms
2021/09/23 15:01:16.222 INFO [HelixServerStarter] [Start a Pinot [SERVER]] Pinot server ready
2021/09/23 15:30:06.057 WARN [ZkBaseDataAccessor] [ZkClient-EventThread-23-apache-pinot-zookeeper-bitnami-headless.pinot.svc.cluster.local:2181] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/74dc1531-5db0-47a4-b97c-95cabc4067dc=-101}

Controller logs:

Cluster: prod-green
2021/09/23 14:01:12.778 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a392b5d6_DEFAULT)] Event a392b5d6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:12.879 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c9cc5f8e_DEFAULT)] Event c9cc5f8e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:12.996 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(fb813904_DEFAULT)] Event fb813904_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:13.184 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event d0ca4d41_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:01:13.200 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d0ca4d41_DEFAULT)] Event d0ca4d41_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:13.325 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c9563711_DEFAULT)] Event c9563711_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:13.405 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c7d1eaa6_DEFAULT)] Event c7d1eaa6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:13.798 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(43b22c37_DEFAULT)] Event 43b22c37_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:14.001 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3f59c606_DEFAULT)] Event 3f59c606_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:14.089 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ebacdc57_DEFAULT)] Event ebacdc57_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:14.276 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(6743a09e_DEFAULT)] Event 6743a09e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:15.004 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f6334c16_DEFAULT)] Event f6334c16_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:15.117 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(74e299c8_DEFAULT)] Event 74e299c8_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:15.272 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8d39b730_DEFAULT)] Event 8d39b730_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:20.303 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(bf2e9df7_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/2d3beedd-4731-4036-a8d9-20bbc7b27d27=-101}
2021/09/23 14:01:20.304 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(bf2e9df7_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/2d3beedd-4731-4036-a8d9-20bbc7b27d27=-101}
2021/09/23 14:01:20.376 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(bf2e9df7_DEFAULT)] Event bf2e9df7_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:01:20.497 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(c86a979c_DEFAULT)] Event c86a979c_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:01:27.242 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(34702a6e_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/99bdeb71-78b9-41da-b9ca-9c9acf6f18db=-101}
2021/09/23 14:01:27.243 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(34702a6e_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/99bdeb71-78b9-41da-b9ca-9c9acf6f18db=-101}
2021/09/23 14:02:19.624 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 597a12e9_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:03:20.125 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(9313c732_DEFAULT)] Event 9313c732_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:03:38.595 WARN [SegmentCompletionFSM_spanEventView__6__863__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2160000000 now=1632405818595 start=1632405563326
2021/09/23 14:03:39.761 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(1c6e25a9_DEFAULT)] Event 1c6e25a9_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:03:41.917 WARN [SegmentCompletionFSM_spanEventView__2__33544__20210923T1357Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85968037372 now=1632405821917 start=1632405559500
2021/09/23 14:03:52.468 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event c4136a1e_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:04:32.970 WARN [SegmentCompletionFSM_spanEventView__5__864__20210923T1355Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2162500000 now=1632405872970 start=1632405653317
2021/09/23 14:04:33.808 WARN [SegmentCompletionFSM_spanEventView__1__33509__20210923T1355Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85826526828 now=1632405873808 start=1632405653837
2021/09/23 14:04:39.609 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event f2d45c6d_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:04:39.993 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event f7894bf7_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:04:49.945 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 2152fb5b_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:05:57.279 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1248c422_DEFAULT)] Event 1248c422_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:57.369 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(fd9b0a0a_DEFAULT)] Event fd9b0a0a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:57.488 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(20a00442_DEFAULT)] Event 20a00442_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:57.634 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4424c7c6_DEFAULT)] Event 4424c7c6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:57.685 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 6ac40d3f_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:05:57.733 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(6ac40d3f_DEFAULT)] Event 6ac40d3f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:57.824 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3fc8e670_DEFAULT)] Event 3fc8e670_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:58.189 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b4358f84_DEFAULT)] Event b4358f84_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:58.353 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a145559b_DEFAULT)] Event a145559b_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:58.455 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8c54bc97_DEFAULT)] Event 8c54bc97_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:59.036 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(6c2378d1_DEFAULT)] Event 6c2378d1_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:59.135 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2af892ef_DEFAULT)] Event 2af892ef_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:59.276 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(159f3e0a_DEFAULT)] Event 159f3e0a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:59.370 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(99da1fd0_DEFAULT)] Event 99da1fd0_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:05:59.472 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a9e2c61a_DEFAULT)] Event a9e2c61a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:00.603 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(58858c7a_DEFAULT)] Event 58858c7a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:00.719 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b12a1c43_DEFAULT)] Event b12a1c43_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:00.843 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9acf27ca_DEFAULT)] Event 9acf27ca_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:01.404 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ea5fb5a1_DEFAULT)] Event ea5fb5a1_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:01.525 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9a4328a6_DEFAULT)] Event 9a4328a6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:01.609 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(e90911a7_DEFAULT)] Event e90911a7_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:06:14.619 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(8f49d193_DEFAULT)] Event 8f49d193_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:06:16.217 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event e399e4d8_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:06:27.437 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(a66c4822_DEFAULT)] Event a66c4822_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:06:39.954 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(8152f13b_DEFAULT)] Event 8152f13b_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:07:46.932 WARN [SegmentCompletionFSM_spanEventView__5__864__20210923T1355Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2162500000 now=1632406066932 start=1632405877806
2021/09/23 14:07:47.786 WARN [SegmentCompletionFSM_spanEventView__1__33509__20210923T1355Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85826526828 now=1632406067786 start=1632405877806
2021/09/23 14:08:29.048 WARN [SegmentCompletionFSM_spanEventView__2__33544__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85968037372 now=1632406109048 start=1632405824922
2021/09/23 14:08:30.287 WARN [SegmentCompletionFSM_spanEventView__6__863__20210923T1357Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2160000000 now=1632406110286 start=1632405821613
2021/09/23 14:09:00.486 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(b49b11d8_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/dfdafac4-90d8-487c-909a-17cb69d25100=-101}
2021/09/23 14:09:00.487 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(b49b11d8_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/dfdafac4-90d8-487c-909a-17cb69d25100=-101}
2021/09/23 14:09:00.765 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(b1e6c730_DEFAULT)] Event b1e6c730_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:09:06.571 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 2fc509c5_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:09:07.656 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(4dedf5c4_DEFAULT)] Event 4dedf5c4_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:09:11.922 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(4dac10bd_DEFAULT)] Event 4dac10bd_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:09:14.788 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 8091ca04_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:10:09.733 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__2__33540__20210923T1342Z of table spanEventView_REALTIME has no online replicas
2021/09/23 14:10:09.736 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__6__860__20210923T1346Z of table spanEventView_REALTIME has no online replicas
2021/09/23 14:10:09.736 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__6__861__20210923T1349Z of table spanEventView_REALTIME has no online replicas
2021/09/23 14:10:09.737 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 3 segments with no online replicas
2021/09/23 14:10:09.737 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:10:31.380 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(2f10fe65_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/8c9bacc8-fb8b-4f3c-8733-3576f80406f3=-101}
2021/09/23 14:10:31.382 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(2f10fe65_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-2.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/75beafe3-955d-4a07-86ad-aaae0d8a694d=-101}
2021/09/23 14:10:31.529 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(10ad37c3_DEFAULT)] Event 10ad37c3_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:10:41.024 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 87097d51_TASK(LiveInstanceChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:10:41.345 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event ff683f83_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:10:41.725 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d543cb23_DEFAULT)] Event d543cb23_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:41.829 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(830983b5_DEFAULT)] Event 830983b5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:41.916 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(66119eb0_DEFAULT)] Event 66119eb0_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:42.135 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b18a3b8c_DEFAULT)] Event b18a3b8c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:42.267 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(bb1f2b85_DEFAULT)] Event bb1f2b85_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:42.359 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f315c94d_DEFAULT)] Event f315c94d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:42.451 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9ffd9f20_DEFAULT)] Event 9ffd9f20_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:10:42.550 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(17d66edd_DEFAULT)] Event 17d66edd_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:11:15.854 WARN [SegmentCompletionFSM_spanEventView__1__33509__20210923T1355Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85826526828 now=1632406275854 start=1632406071930
2021/09/23 14:11:19.585 WARN [SegmentCompletionFSM_spanEventView__5__864__20210923T1355Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2162500000 now=1632406279585 start=1632406069950
2021/09/23 14:12:07.724 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(e14c411e_DEFAULT)] Event e14c411e_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:12:21.039 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(338cf08d_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/8a810e52-db6a-4e5e-98a8-04e1036c6658=-101}
2021/09/23 14:12:21.040 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(338cf08d_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/8a810e52-db6a-4e5e-98a8-04e1036c6658=-101}
2021/09/23 14:12:21.295 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(9e0f9405_DEFAULT)] Event 9e0f9405_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:13:14.094 WARN [SegmentCompletionFSM_spanEventView__6__863__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2160000000 now=1632406394093 start=1632406113386
2021/09/23 14:13:33.649 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(e89fc9ed_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-1.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/d116a355-20d9-4a6d-9603-91a4c3e4f94e=-101}
2021/09/23 14:13:33.650 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(e89fc9ed_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-1.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/d116a355-20d9-4a6d-9603-91a4c3e4f94e=-101}
2021/09/23 14:13:33.820 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(ffb29267_DEFAULT)] Event ffb29267_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:13:41.159 WARN [SegmentCompletionFSM_spanEventView__2__33544__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85968037372 now=1632406421159 start=1632406112066
2021/09/23 14:14:57.334 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d25f8721_DEFAULT)] Event d25f8721_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:14:57.456 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d68becb0_DEFAULT)] Event d68becb0_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:14:57.551 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d67c676d_DEFAULT)] Event d67c676d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:14:57.786 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(92a57d49_DEFAULT)] Event 92a57d49_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:14:57.949 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f24f8a65_DEFAULT)] Event f24f8a65_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:14:58.057 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4a5b10a5_DEFAULT)] Event 4a5b10a5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.264 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(66d8b7cd_DEFAULT)] Event 66d8b7cd_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.390 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(caf4e251_DEFAULT)] Event caf4e251_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.530 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8d53b02d_DEFAULT)] Event 8d53b02d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.662 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(957556b3_DEFAULT)] Event 957556b3_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.817 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0576c0fe_DEFAULT)] Event 0576c0fe_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:00.982 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f4fa5ee7_DEFAULT)] Event f4fa5ee7_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:01.070 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8ab8af28_DEFAULT)] Event 8ab8af28_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:15:09.804 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__5__859__20210923T1314Z of table spanEventView_REALTIME has no online replicas
2021/09/23 14:15:09.806 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 1 segments with no online replicas
2021/09/23 14:15:09.806 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:15:19.682 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(04980f3b_DEFAULT)] Event 04980f3b_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:15:22.552 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(19589fce_DEFAULT)] Event 19589fce_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:17:18.048 WARN [SegmentCompletionFSM_spanEventView__6__863__20210923T1357Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2160000000 now=1632406638048 start=1632406397119
2021/09/23 14:17:19.030 WARN [SegmentCompletionFSM_spanEventView__2__33544__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85968037372 now=1632406639029 start=1632406424164
2021/09/23 14:18:38.995 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(a26e9f71_DEFAULT)] Event a26e9f71_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:18:43.310 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(5769fcf5_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/837e0cf5-ae48-44aa-b4c8-5ae39319b266=-101}
2021/09/23 14:18:46.222 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(b6a2f71f_DEFAULT)] Event b6a2f71f_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:19:02.897 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ed0d0b7c_DEFAULT)] Event ed0d0b7c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.009 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a99f1bc5_DEFAULT)] Event a99f1bc5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.103 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9beb2d6c_DEFAULT)] Event 9beb2d6c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.212 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(14e0215e_DEFAULT)] Event 14e0215e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.322 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(789bc57a_DEFAULT)] Event 789bc57a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.671 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(16f58f4c_DEFAULT)] Event 16f58f4c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.803 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b3f5e4ca_DEFAULT)] Event b3f5e4ca_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.902 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8542debc_DEFAULT)] Event 8542debc_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:03.933 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 29695a3d_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:19:04.002 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(29695a3d_DEFAULT)] Event 29695a3d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.324 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c5c89c4d_DEFAULT)] Event c5c89c4d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.435 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d60cea69_DEFAULT)] Event d60cea69_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.565 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d9a446e3_DEFAULT)] Event d9a446e3_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.573 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 89c8b6e3_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:19:05.668 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(5415e7a7_DEFAULT)] Event 5415e7a7_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.793 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(923d18f5_DEFAULT)] Event 923d18f5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:05.916 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4f4c0076_DEFAULT)] Event 4f4c0076_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.026 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2a060e26_DEFAULT)] Event 2a060e26_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.137 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4d9f0638_DEFAULT)] Event 4d9f0638_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.261 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f37a8dd0_DEFAULT)] Event f37a8dd0_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.365 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(77936c0c_DEFAULT)] Event 77936c0c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.545 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7c199f34_DEFAULT)] Event 7c199f34_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.637 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0642ba83_DEFAULT)] Event 0642ba83_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.778 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(fef10f2f_DEFAULT)] Event fef10f2f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:06.877 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(725dca81_DEFAULT)] Event 725dca81_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:07.064 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c940957d_DEFAULT)] Event c940957d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:07.115 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event a35f028c_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:19:07.153 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a35f028c_DEFAULT)] Event a35f028c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:07.401 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7f925dd8_DEFAULT)] Event 7f925dd8_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:07.533 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a995bf0b_DEFAULT)] Event a995bf0b_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:19:07.675 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(90adb41d_DEFAULT)] Event 90adb41d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:21:48.353 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(f99fc6a1_DEFAULT)] Event f99fc6a1_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:21:48.459 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 00eedc5e_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:22:17.789 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(0682eb12_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/d720424e-9252-4dd9-a2a0-ae14025d1b34=-101}
2021/09/23 14:22:18.077 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(406d020d_DEFAULT)] Event 406d020d_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:24:17.058 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b157060e_DEFAULT)] Event b157060e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:17.201 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(14bc7478_DEFAULT)] Event 14bc7478_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:17.282 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c88f2947_DEFAULT)] Event c88f2947_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:17.927 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1466657d_DEFAULT)] Event 1466657d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:18.034 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(96009226_DEFAULT)] Event 96009226_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:18.131 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7f63f7c1_DEFAULT)] Event 7f63f7c1_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:18.243 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(543c7fdf_DEFAULT)] Event 543c7fdf_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:18.330 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(192ff523_DEFAULT)] Event 192ff523_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.201 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ec890f2e_DEFAULT)] Event ec890f2e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.318 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7a44ddd3_DEFAULT)] Event 7a44ddd3_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.410 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0e70de49_DEFAULT)] Event 0e70de49_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.558 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(56ddae42_DEFAULT)] Event 56ddae42_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.671 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9cbf5245_DEFAULT)] Event 9cbf5245_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.801 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(92890cc6_DEFAULT)] Event 92890cc6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:22.900 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(fcc6929d_DEFAULT)] Event fcc6929d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.011 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3511a196_DEFAULT)] Event 3511a196_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.152 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2b6be989_DEFAULT)] Event 2b6be989_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.302 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d8d22884_DEFAULT)] Event d8d22884_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.396 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(168eb610_DEFAULT)] Event 168eb610_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.496 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(77ba9443_DEFAULT)] Event 77ba9443_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.625 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3760a7e7_DEFAULT)] Event 3760a7e7_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.743 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a18d1f06_DEFAULT)] Event a18d1f06_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.842 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3632bcb8_DEFAULT)] Event 3632bcb8_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:23.940 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2d8c8414_DEFAULT)] Event 2d8c8414_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:24.033 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7021fcc9_DEFAULT)] Event 7021fcc9_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:24.170 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8499a211_DEFAULT)] Event 8499a211_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:24.258 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(26eb1da1_DEFAULT)] Event 26eb1da1_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:24:24.372 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ba1c9c9d_DEFAULT)] Event ba1c9c9d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:25:08.578 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(b28448e5_DEFAULT)] Event b28448e5_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:25:09.876 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment rawTraceView__0__1454__20210923T1119Z of table rawTraceView_REALTIME has no replicas
2021/09/23 14:25:09.876 WARN [SegmentStatusChecker] [pool-9-thread-5] Table rawTraceView_REALTIME has 1 segments with no online replicas
2021/09/23 14:25:09.876 WARN [SegmentStatusChecker] [pool-9-thread-5] Table rawTraceView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:25:09.885 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__2__9947__20210923T1258Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:25:09.885 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__2__9948__20210923T1344Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:25:09.885 WARN [SegmentStatusChecker] [pool-9-thread-5] Table serviceCallView_REALTIME has 2 segments with no online replicas
2021/09/23 14:25:09.885 WARN [SegmentStatusChecker] [pool-9-thread-5] Table serviceCallView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:25:09.934 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33504__20210923T1315Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:25:09.934 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33505__20210923T1340Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:25:09.934 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33506__20210923T1344Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:25:09.934 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33507__20210923T1348Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:25:09.940 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 10 segments with no online replicas
2021/09/23 14:25:09.940 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:25:50.994 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 171a1662_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:26:13.785 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 2ddb8cff_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:26:16.433 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 7ca4b237_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:28:05.170 WARN [SegmentCompletionFSM_spanEventView__0__33272__20210923T1418Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85446706323 now=1632407285169 start=1632407036623
2021/09/23 14:29:41.633 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 5a420789_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:30:09.944 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment rawTraceView__0__1454__20210923T1119Z of table rawTraceView_REALTIME has no replicas
2021/09/23 14:30:09.944 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment rawTraceView__2__1446__20210923T1127Z of table rawTraceView_REALTIME has no online replicas
2021/09/23 14:30:09.944 WARN [SegmentStatusChecker] [pool-9-thread-3] Table rawTraceView_REALTIME has 2 segments with no online replicas
2021/09/23 14:30:09.944 WARN [SegmentStatusChecker] [pool-9-thread-3] Table rawTraceView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:30:09.952 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment serviceCallView__0__9856__20210923T1319Z of table serviceCallView_REALTIME has no online replicas
2021/09/23 14:30:09.953 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment serviceCallView__2__9947__20210923T1258Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:30:09.953 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment serviceCallView__2__9948__20210923T1344Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:30:09.953 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment serviceCallView__3__9908__20210923T1304Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:30:09.953 WARN [SegmentStatusChecker] [pool-9-thread-3] Table serviceCallView_REALTIME has 5 segments with no online replicas
2021/09/23 14:30:09.953 WARN [SegmentStatusChecker] [pool-9-thread-3] Table serviceCallView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:30:10.004 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment spanEventView__1__33504__20210923T1315Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:30:10.004 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment spanEventView__1__33505__20210923T1340Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:30:10.004 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment spanEventView__1__33506__20210923T1344Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:30:10.004 WARN [SegmentStatusChecker] [pool-9-thread-3] Segment spanEventView__1__33507__20210923T1348Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:30:10.009 WARN [SegmentStatusChecker] [pool-9-thread-3] Table spanEventView_REALTIME has 28 segments with no online replicas
2021/09/23 14:30:10.009 WARN [SegmentStatusChecker] [pool-9-thread-3] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:31:53.077 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 2c52bb4c_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:31:54.082 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(075b0909_DEFAULT)] Event 075b0909_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.104 WARN [SegmentCompletionFSM_spanEventView__4__877__20210923T1421Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2195000000 now=1632407514104 start=1632407285691
2021/09/23 14:31:54.179 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ea83ad38_DEFAULT)] Event ea83ad38_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.266 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2c918baf_DEFAULT)] Event 2c918baf_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.450 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(15865d09_DEFAULT)] Event 15865d09_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.566 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d4673f42_DEFAULT)] Event d4673f42_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.618 WARN [SegmentCompletionFSM_spanEventView__0__33272__20210923T1418Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85446706323 now=1632407514618 start=1632407288189
2021/09/23 14:31:54.664 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(bb25679c_DEFAULT)] Event bb25679c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:54.980 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3bae3e48_DEFAULT)] Event 3bae3e48_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.110 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b2e60d7f_DEFAULT)] Event b2e60d7f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.266 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(bad8d025_DEFAULT)] Event bad8d025_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.374 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3a8f1e4d_DEFAULT)] Event 3a8f1e4d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.498 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ac0a67aa_DEFAULT)] Event ac0a67aa_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.630 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4121bd6a_DEFAULT)] Event 4121bd6a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.815 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(a55e8cef_DEFAULT)] Event a55e8cef_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:55.951 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(aef2a821_DEFAULT)] Event aef2a821_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.039 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(526c2e71_DEFAULT)] Event 526c2e71_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.138 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(253710c3_DEFAULT)] Event 253710c3_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.352 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d5946f1b_DEFAULT)] Event d5946f1b_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.507 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(6b211404_DEFAULT)] Event 6b211404_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.628 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d93767b8_DEFAULT)] Event d93767b8_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:56.708 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0a8a7d92_DEFAULT)] Event 0a8a7d92_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:59.509 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1fc64af6_DEFAULT)] Event 1fc64af6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:59.696 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1ac7573a_DEFAULT)] Event 1ac7573a_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:31:59.776 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(949e7238_DEFAULT)] Event 949e7238_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 14:32:04.572 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
2021/09/23 14:32:04.598 WARN [SegmentCompletionFSM_spanEventView__3__33058__20210923T1422Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098 offset=84641863493 now=1632407524598 start=1632407277460
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 14:32:04.661 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 14:32:04.713 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
2021/09/23 14:33:26.019 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 58778b0f_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:33:27.831 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 39460e63_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:34:11.070 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 57bff217_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:34:13.201 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 81f18031_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:34:13.489 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 9d1ecc43_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:34:14.463 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 155dd94f_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:35:10.013 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment rawTraceView__0__1454__20210923T1119Z of table rawTraceView_REALTIME has no replicas
2021/09/23 14:35:10.014 WARN [SegmentStatusChecker] [pool-9-thread-5] Table rawTraceView_REALTIME has 1 segments with no online replicas
2021/09/23 14:35:10.014 WARN [SegmentStatusChecker] [pool-9-thread-5] Table rawTraceView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__2__9947__20210923T1258Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__2__9948__20210923T1344Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__3__9908__20210923T1304Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment serviceCallView__3__9909__20210923T1346Z of table serviceCallView_REALTIME has no replicas
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Table serviceCallView_REALTIME has 4 segments with no online replicas
2021/09/23 14:35:10.022 WARN [SegmentStatusChecker] [pool-9-thread-5] Table serviceCallView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:35:10.068 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33504__20210923T1315Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:35:10.068 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33505__20210923T1340Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:35:10.068 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33506__20210923T1344Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:35:10.068 WARN [SegmentStatusChecker] [pool-9-thread-5] Segment spanEventView__1__33507__20210923T1348Z of table spanEventView_REALTIME has no replicas
2021/09/23 14:35:10.073 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 20 segments with no online replicas
2021/09/23 14:35:10.073 WARN [SegmentStatusChecker] [pool-9-thread-5] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 14:35:10.948 WARN [SegmentCompletionFSM_spanEventView__4__877__20210923T1421Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2195000000 now=1632407710948 start=1632407517123
2021/09/23 14:35:12.238 WARN [SegmentCompletionFSM_spanEventView__0__33272__20210923T1418Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85446706323 now=1632407712238 start=1632407521300
2021/09/23 14:36:24.736 WARN [SegmentCompletionFSM_spanEventView__3__33058__20210923T1422Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098 offset=84641863493 now=1632407784735 start=1632407527616
2021/09/23 14:37:21.728 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 052d00c2_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:38:12.172 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 543abb06_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:38:18.941 WARN [SegmentCompletionFSM_spanEventView__1__33509__20210923T1355Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85826526828 now=1632407898941 start=1632406279036
2021/09/23 14:38:19.597 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 1a4b4d7e_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:38:20.664 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 1740608b_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:38:25.173 WARN [SegmentCompletionFSM_spanEventView__5__864__20210923T1355Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2162500000 now=1632407905173 start=1632406863796
2021/09/23 14:39:24.782 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(bcc2829e_DEFAULT)] Event bcc2829e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:24.888 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(383e5361_DEFAULT)] Event 383e5361_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:24.975 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(97671be4_DEFAULT)] Event 97671be4_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.243 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(24f2e692_DEFAULT)] Event 24f2e692_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.447 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7a461e88_DEFAULT)] Event 7a461e88_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.606 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(51f01163_DEFAULT)] Event 51f01163_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.696 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(639db461_DEFAULT)] Event 639db461_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.801 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b5ff5629_DEFAULT)] Event b5ff5629_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:25.945 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c5cf649f_DEFAULT)] Event c5cf649f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:26.036 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ae150bbc_DEFAULT)] Event ae150bbc_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:26.165 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(732ead7d_DEFAULT)] Event 732ead7d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:26.266 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(8b5e4d6c_DEFAULT)] Event 8b5e4d6c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:39:34.983 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 0038b802_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:39:36.839 WARN [SegmentCompletionFSM_spanEventView__3__33058__20210923T1422Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098 offset=84641863493 now=1632407976839 start=1632407791886
2021/09/23 14:39:37.239 WARN [SegmentCompletionFSM_spanEventView__7__884__20210923T1425Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2212500000 now=1632407977239 start=1632407792258
2021/09/23 14:39:38.457 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(c4518cce_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/84a0363a-63a1-4971-a696-da778ff71662=-101}
2021/09/23 14:39:38.619 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(929327fc_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-3.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/76229ce7-2ac6-4ad3-9845-c5b11824381a=-101}
2021/09/23 14:39:43.413 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 45c9393a_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:41:16.409 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(5e3b47bf_DEFAULT)] Event 5e3b47bf_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:41:17.465 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(f218dbcd_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/641328b3-20ac-4c25-9286-c9dc7826f5e8=-101}
2021/09/23 14:41:17.706 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(56482011_DEFAULT)] Event 56482011_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:41:19.036 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(d7aac4f0_DEFAULT)] Event d7aac4f0_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:41:19.313 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 25783ff5_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:41:19.915 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(41561440_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/cc658de2-0e49-4166-bb30-7d76e089c0ec=-101}
2021/09/23 14:41:19.916 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(41561440_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/cc658de2-0e49-4166-bb30-7d76e089c0ec=-101}
2021/09/23 14:41:20.077 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(106686a3_DEFAULT)] Event 106686a3_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:41:35.098 WARN [SegmentCompletionFSM_spanEventView__6__863__20210923T1357Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2160000000 now=1632408095098 start=1632406641538
2021/09/23 14:41:36.682 WARN [SegmentCompletionFSM_spanEventView__2__33544__20210923T1357Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85968037372 now=1632408096682 start=1632406642035
2021/09/23 14:43:26.799 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(4752001c_DEFAULT)] Event 4752001c_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:43:28.394 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(2b064936_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/49d8cdcb-c396-464b-9708-32bb0663b8a5=-101}
2021/09/23 14:43:28.577 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(92bf9b02_DEFAULT)] Event 92bf9b02_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:44:33.659 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(f32c6d76_DEFAULT)] Event f32c6d76_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:44:39.457 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event d08f735b_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:44:39.657 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(7678d829_DEFAULT)] Event 7678d829_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:44:59.208 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(69758db7_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/f2d95b25-48fe-4138-980a-5643f6adc295=-101}
2021/09/23 14:44:59.208 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(69758db7_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/f2d95b25-48fe-4138-980a-5643f6adc295=-101}
2021/09/23 14:44:59.379 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(b46a1e3b_DEFAULT)] Event b46a1e3b_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:44:59.688 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event b638c3e6_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:45:01.507 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(72516529_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/4718ffa9-8583-43ea-ba6e-abe8e0f1f505=-101}
2021/09/23 14:45:01.678 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(97af3835_DEFAULT)] Event 97af3835_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:45:14.547 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(2b6c7a15_DEFAULT)] Event 2b6c7a15_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:45:19.690 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(2b3de2ae_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-2.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/d9ace221-bfed-4ba0-bdc5-7b9885317fcc=-101, /pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/aa3390e1-2407-4bf5-bbd7-b0fcb8e09a09=-101}
2021/09/23 14:45:19.691 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(2b3de2ae_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/aa3390e1-2407-4bf5-bbd7-b0fcb8e09a09=-101}
2021/09/23 14:45:19.904 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(ff9ed343_DEFAULT)] Event ff9ed343_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:45:33.941 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(58ac53c8_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/c175ccea-2fcd-46d8-81c9-844756319079=-101}
2021/09/23 14:45:34.092 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(02795e6c_DEFAULT)] Event 02795e6c_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:05.103 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(d38e1b94_DEFAULT)] Event d38e1b94_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:11.107 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(deea182d_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/4cfca853-a822-4d8c-baca-8fcdc6a9244c=-101}
2021/09/23 14:47:11.360 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(8243fec6_DEFAULT)] Event 8243fec6_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:15.547 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(561261bd_DEFAULT)] Event 561261bd_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:41.292 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(9bdacc59_DEFAULT)] Event 9bdacc59_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:50.375 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(9a36841c_DEFAULT)] Event 9a36841c_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:47:56.034 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(6072a4b0_DEFAULT)] Event 6072a4b0_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:48:34.470 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(c4bd1e84_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/414d6a14-404b-4cdb-a2b9-df386b0cc66f=-101}
2021/09/23 14:48:34.657 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(68708ceb_DEFAULT)] Event 68708ceb_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:48:45.468 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(633b5f48_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/88020225-a093-4af5-b003-f96bd85e0068=-101}
2021/09/23 14:48:45.469 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(633b5f48_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/88020225-a093-4af5-b003-f96bd85e0068=-101}
2021/09/23 14:48:45.727 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(9e95b70d_DEFAULT)] Event 9e95b70d_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:50:33.714 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(dc0f2b34_DEFAULT)] Event dc0f2b34_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:50:40.483 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(1267f641_DEFAULT)] Event 1267f641_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:51:01.340 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(0ca9415a_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/772f26b3-0e8b-4d2b-989b-adc816280ff1=-101}
2021/09/23 14:51:01.606 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(3e9132c4_DEFAULT)] Event 3e9132c4_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:51:01.701 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 9df6b091_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:51:08.222 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(e3dc68f4_DEFAULT)] Event e3dc68f4_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:52:04.277 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(a3f20ba0_DEFAULT)] Event a3f20ba0_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:52:26.758 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(c354ee27_DEFAULT)] Event c354ee27_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:53:29.297 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(468f4ed5_DEFAULT)] Event 468f4ed5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:29.432 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(fd296a0d_DEFAULT)] Event fd296a0d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:29.548 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(7f094c70_DEFAULT)] Event 7f094c70_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:29.697 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(20e151db_DEFAULT)] Event 20e151db_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:29.852 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c426c27c_DEFAULT)] Event c426c27c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:29.971 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4b61c912_DEFAULT)] Event 4b61c912_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:30.104 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(16257a88_DEFAULT)] Event 16257a88_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:30.216 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(5cef74b5_DEFAULT)] Event 5cef74b5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:30.379 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(e205097e_DEFAULT)] Event e205097e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:30.481 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(9a283bd5_DEFAULT)] Event 9a283bd5_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:53:30.593 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ab809be6_DEFAULT)] Event ab809be6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:54:19.371 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(24844f37_DEFAULT)] Event 24844f37_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:54:28.693 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(e869bd7b_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/668520b0-95db-45a3-abf5-178dc4b5a4a5=-101}
2021/09/23 14:54:28.694 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(e869bd7b_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/668520b0-95db-45a3-abf5-178dc4b5a4a5=-101}
2021/09/23 14:54:28.939 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(d65706a4_DEFAULT)] Event d65706a4_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:55:35.648 WARN [SegmentCompletionFSM_spanEventView__2__33547__20210923T1450Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85975537372 now=1632408935648 start=1632408760513
2021/09/23 14:55:38.305 WARN [SegmentCompletionFSM_spanEventView__6__866__20210923T1450Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2167500000 now=1632408938305 start=1632408754425
2021/09/23 14:55:44.644 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(60a61425_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/002aa711-82f9-4bf7-86be-1ced5c93d47c=-101}
2021/09/23 14:55:44.866 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(382c5177_DEFAULT)] Event 382c5177_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:56:32.517 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(dfecd0d6_DEFAULT)] Event dfecd0d6_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 14:56:32.734 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event f8ad9360_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:56:58.690 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(84422a6d_DEFAULT)] Event 84422a6d_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:58.794 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(3b450013_DEFAULT)] Event 3b450013_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:58.931 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(07f2036b_DEFAULT)] Event 07f2036b_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.054 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(595d86e1_DEFAULT)] Event 595d86e1_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.154 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(b16dc905_DEFAULT)] Event b16dc905_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.291 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c7c94c77_DEFAULT)] Event c7c94c77_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.386 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(594c8adc_DEFAULT)] Event 594c8adc_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.568 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(104453ae_DEFAULT)] Event 104453ae_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.658 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(84d42d93_DEFAULT)] Event 84d42d93_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.768 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(18dc1232_DEFAULT)] Event 18dc1232_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.855 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(ed8df7fe_DEFAULT)] Event ed8df7fe_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:56:59.970 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(e6f0a262_DEFAULT)] Event e6f0a262_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 14:57:52.504 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 6be4c23b_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:57:52.630 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event 1d24356d_TASK(MessageChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 14:59:06.351 WARN [SegmentCompletionFSM_spanEventView__2__33547__20210923T1450Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85975537372 now=1632409146351 start=1632408938667
2021/09/23 14:59:07.139 WARN [SegmentCompletionFSM_spanEventView__6__866__20210923T1450Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2167500000 now=1632409147139 start=1632408941309
2021/09/23 14:59:18.979 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(075e3ef5_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/f107f7e0-af8d-4467-9040-897ee0e761dc=-101}
2021/09/23 14:59:19.277 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(0c13d584_DEFAULT)] Event 0c13d584_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:00:04.661 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(420b6066_DEFAULT)] Event 420b6066_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:04.834 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2fbf762e_DEFAULT)] Event 2fbf762e_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:04.922 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(78ede6b2_DEFAULT)] Event 78ede6b2_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.035 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1e480059_DEFAULT)] Event 1e480059_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.210 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(1cd26804_DEFAULT)] Event 1cd26804_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.355 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(f1bb47db_DEFAULT)] Event f1bb47db_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.455 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(590b10d6_DEFAULT)] Event 590b10d6_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.579 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c94697ab_DEFAULT)] Event c94697ab_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.681 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(33eeaf99_DEFAULT)] Event 33eeaf99_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.815 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2bbfca27_DEFAULT)] Event 2bbfca27_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:05.961 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(e4869a21_DEFAULT)] Event e4869a21_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:00:06.059 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(611e1452_DEFAULT)] Event 611e1452_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:02:17.827 WARN [SegmentCompletionFSM_spanEventView__2__33547__20210923T1450Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85975537372 now=1632409337826 start=1632409149376
2021/09/23 15:02:21.397 WARN [SegmentCompletionFSM_spanEventView__6__866__20210923T1450Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2167500000 now=1632409341397 start=1632409150144
2021/09/23 15:03:30.974 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(c1ca9535_DEFAULT)] Event c1ca9535_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:31.073 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0eba862f_DEFAULT)] Event 0eba862f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:31.249 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(0e17fedc_DEFAULT)] Event 0e17fedc_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:31.528 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(175ad4f3_DEFAULT)] Event 175ad4f3_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:31.633 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(2e581f07_DEFAULT)] Event 2e581f07_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:31.732 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(d15fe12c_DEFAULT)] Event d15fe12c_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:36.986 ERROR [MessageGenerationPhase] [HelixController-pipeline-default-pinot-prod-(4976150f_DEFAULT)] Event 4976150f_DEFAULT : Unable to find a next state for resource: logEventView_REALTIME partition: logEventView__0__0__20210604T1139Z from stateModelDefinitionclass org.apache.helix.model.StateModelDefinition from:ERROR to:CONSUMING
2021/09/23 15:03:37.099 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(ce724eb0_DEFAULT)] Event ce724eb0_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:04:21.947 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(ce79c31a_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/d1a9bafa-373f-446e-9708-7c9684e539ca=-101}
2021/09/23 15:04:22.145 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(591553fd_DEFAULT)] Event 591553fd_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:05:43.828 WARN [SegmentCompletionFSM_spanEventView__2__33547__20210923T1450Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=85975537372 now=1632409543828 start=1632409340846
2021/09/23 15:05:44.527 WARN [SegmentCompletionFSM_spanEventView__6__866__20210923T1450Z] [grizzly-http-server-0] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098 offset=2167500000 now=1632409544527 start=1632409346011
2021/09/23 15:06:24.684 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(2769a31c_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/df71bbe2-2ac4-4b2c-9b3b-31888d4e3912=-101}
2021/09/23 15:06:24.685 WARN [ZkBaseDataAccessor] [HelixController-pipeline-default-pinot-prod-(2769a31c_DEFAULT)] Fail to read record for paths: {/pinot-prod/INSTANCES/Broker_broker-0.broker-headless.pinot.svc.cluster.local_8099/MESSAGES/df71bbe2-2ac4-4b2c-9b3b-31888d4e3912=-101}
2021/09/23 15:06:24.853 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(ec4c3ee0_DEFAULT)] Event ec4c3ee0_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:11:01.240 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(434ee506_DEFAULT)] Event 434ee506_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:13:14.363 WARN [TopStateHandoffReportStage] [HelixController-pipeline-default-pinot-prod-(cf5f3456_DEFAULT)] Event cf5f3456_DEFAULT : Cannot confirm top state missing start time. Use the current system time as the start time.
2021/09/23 15:18:06.991 WARN [TaskGarbageCollectionStage] [TaskJobPurgeWorker-pinot-prod] ResourceControllerDataProvider or HelixManager is null for event e30e8c91_TASK(CurrentStateChange) in cluster pinot-prod. Skip TaskGarbageCollectionStage.
2021/09/23 15:20:17.211 WARN [SegmentCompletionFSM_spanEventView__3__33065__20210923T1511Z] [grizzly-http-server-1] COMMITTER_NOTIFIED:Aborting FSM (too late) instance=Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098 offset=84659363493 now=1632410417211 start=1632410214302
2021/09/23 15:30:06.111 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(834e3694_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/57fa43d4-fd0b-48c1-8405-be3df7bd859d=-101, /pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/78ba0c39-0aef-48c2-8b02-71635bef606a=-101, /pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/faa647fc-faf2-4f67-a47c-f0fd1d4be264=-101, /pinot-prod/INSTANCES/Server_server-realtime-0.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/23065e4a-07dd-4b15-b8f8-0ddcee662f97=-101}
2021/09/23 15:30:06.380 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-prod-(63e942a2_TASK)] Fail to read record for paths: {/pinot-prod/INSTANCES/Server_server-realtime-2.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/b1d22374-5f5b-4ac9-b651-b41792054808=-101, /pinot-prod/INSTANCES/Server_server-realtime-1.server-realtime-headless.pinot.svc.cluster.local_8098/MESSAGES/fba70cad-12a3-49a0-ac54-8cb7e49308aa=-101}
2021/09/23 15:30:10.726 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment rawTraceView__1__1458__20210923T1232Z of table rawTraceView_REALTIME has no replicas
2021/09/23 15:30:10.727 WARN [SegmentStatusChecker] [pool-9-thread-4] Table rawTraceView_REALTIME has 1 segments with no online replicas
2021/09/23 15:30:10.727 WARN [SegmentStatusChecker] [pool-9-thread-4] Table rawTraceView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 15:30:10.736 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment serviceCallView__0__9858__20210923T1406Z of table serviceCallView_REALTIME has no replicas
2021/09/23 15:30:10.736 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment serviceCallView__0__9859__20210923T1444Z of table serviceCallView_REALTIME has no replicas
2021/09/23 15:30:10.737 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment serviceCallView__3__9910__20210923T1353Z of table serviceCallView_REALTIME has no replicas
2021/09/23 15:30:10.737 WARN [SegmentStatusChecker] [pool-9-thread-4] Table serviceCallView_REALTIME has 3 segments with no online replicas
2021/09/23 15:30:10.737 WARN [SegmentStatusChecker] [pool-9-thread-4] Table serviceCallView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 15:30:10.877 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment spanEventView__1__33509__20210923T1355Z of table spanEventView_REALTIME has no replicas
2021/09/23 15:30:10.878 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment spanEventView__1__33510__20210923T1441Z of table spanEventView_REALTIME has no replicas
2021/09/23 15:30:10.878 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment spanEventView__1__33511__20210923T1444Z of table spanEventView_REALTIME has no replicas
2021/09/23 15:30:10.878 WARN [SegmentStatusChecker] [pool-9-thread-4] Segment spanEventView__1__33512__20210923T1447Z of table spanEventView_REALTIME has no replicas
2021/09/23 15:30:10.883 WARN [SegmentStatusChecker] [pool-9-thread-4] Table spanEventView_REALTIME has 30 segments with no online replicas
2021/09/23 15:30:10.883 WARN [SegmentStatusChecker] [pool-9-thread-4] Table spanEventView_REALTIME has 0 replicas, below replication threshold :1
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 15:32:04.870 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
2021/09/23 15:32:04.930 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 15:32:04.931 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.url' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'streamType' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.decoder.prop.schema.registry.rest.url' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.segment.size' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config.
2021/09/23 15:32:04.994 WARN [ConsumerConfig] [pool-9-thread-1] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.

[suddendust]

@richardstartin apologies, I gave you the xmx for the broker earlier. The correct xmx value is 10G.

[suddendust]

@richardstartin thanks for the fixes and all the quick help 😃 Closing this issue.