Errors in consuming segment completely stops consumption

Just in the last day, we had 2 reports of consumption stopping completely for a partition, because of some error occurrences in consumption.

Example 1: User reported that consumption is totally stopped for that partition, and saw this error when we asked him to check logs.

ERROR [LLRealtimeSegmentDataManager_spanEventView__0__15__20201201T0448Z] [spanEventView__0__15__20201201T0448Z] Could not build segment
java.lang.IllegalStateException: Cannot create output dir: /var/pinot/server/data/index/spanEventView_REALTIME/_tmp/tmp-spanEventView__0__15__20201201T0448Z-160688315943

This is coming from LLRealtimeSegmentDataManager#buildSegmentInternal(). As a result of this, we reach this in LLRealtimeSegmentDataManager

// We could not build the segment. Go into error state.
                _state = State.ERROR;

Problem: After that, the consumer thread simply exits. I believe an assumption is made here that some other replica will have built the segment, and this errored replica can simply download the segment. But this assumption fails if user has only 1 replica. Lack of detection mechanism: Because the ideal state and external view match (everything is still CONSUMING), the Cluster Manager continues to report GOOD. Users find out of the problem because they notice data lag in their application. Most users who are just trying out Pinot have not set up metrics monitoring, so asking them to look for LLC_PARTITION_CONSUMING metric is also not an option. Lack of easy recovery mechanism: The validation manager does nothing to fix this, because the ideal state is still CONSUMING and segment metadata is still IN_PROGRESS. Only way to recover is restarting that server.

Example 2: Again, user reported that consumption is totally stopped on the partition, and reported this error.

Exception while executing a state transition task realtimesolines__234__4__20201127T1841Z
java.lang.reflect.InvocationTargetException: null
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:404) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb9
6cb36efb3e4dbceb10461e73c0d6465e81493]
	at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:331) [pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSH
OT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:97) [pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
	at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:49) [pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:693) ~[?:1.8.0_77]
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) ~[?:1.8.0_77]
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_77]
	at org.apache.pinot.core.segment.memory.PinotByteBuffer.allocateDirect(PinotByteBuffer.java:38) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbceb
10461e73c0d6465e81493]
	at org.apache.pinot.core.segment.memory.PinotDataBuffer.allocateDirect(PinotDataBuffer.java:116) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbce
b10461e73c0d6465e81493]
	at org.apache.pinot.core.io.writer.impl.DirectMemoryManager.allocateInternal(DirectMemoryManager.java:53) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36e
fb3e4dbceb10461e73c0d6465e81493]
	at org.apache.pinot.core.io.readerwriter.RealtimeIndexOffHeapMemoryManager.allocate(RealtimeIndexOffHeapMemoryManager.java:79) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6
.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
	at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.addBuffer(FixedByteSVMutableForwardIndex.java:208) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0
.6.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
	at org.apache.pinot.core.realtime.impl.forward.FixedByteSVMutableForwardIndex.<init>(FixedByteSVMutableForwardIndex.java:77) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0
-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
	at org.apache.pinot.core.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:294) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4
dbceb10461e73c0d6465e81493]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.<init>(LLRealtimeSegmentDataManager.java:1245) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-S
NAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
	at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:312) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSH
OT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
	at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:133) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAP
SHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:164)
 ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
	at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOffline(SegmentOnlineOfflineStateModelFactory.java:8
8) ~[pinot-all-0.6.0-SNAPSHOT-jar-with-dependencies.jar:0.6.0-SNAPSHOT-fb96cb36efb3e4dbceb10461e73c0d6465e81493]
	... 12 more

This appears to have happened when the state transition for OFFLINE to CONSUMING was being processed (during memory allocation for starting a new consuming segment. The consumption loop hasn’t even started). The state transition method onBecomeOnlineFromOffline threw the exception, and the segment must’ve gone into ERROR state in the external view. Ideal state will still be CONSUMING, and segment metadata will be IN_PROGRESS. Problem: Similar to above case, the ideal state and segment metadata indicate that nothing is wrong with the system. But consumption is not happening, the consumer thread never even started. Lack of detection mechanism: This must’ve been flagged on the Cluster Manager as BAD, as there is a mismatch between IS and EV. But user was only seeing the segment metadata IN_PROGRESS and ideal state CONSUMING, and wondering why consumption has stopped if everything looks fine. Lack of recovery mechanism: Again, the validation manager won’t fix this, because it only looks at Ideal State, and according to the IS everything looks fine. Only way to recover will be restart of the server.

We need

1. A way to restart consumption automatically in both cases - Possibly make Validation Manager handle both these cases.
2. A way to reset/restart consumption at will for any given partition.
3. A way to track state of consumers for every CONSUMING segment - an API on the controller would be ideal

[kishoreg]

Can do this instead

1. Add an API to get the status of consuming segment across all servers (including offset is possible)
2. If the status is bad/error because of whatever reason, disable and enable the consuming segment.

Once we have controller API for these two, the invocation can be manual or via validation manager.

[mcvsubbu]

I am fine adding admin API, but let us not do anything via validation manager. It can lead to some timing issues (depending on what we are trying to do) and not work in production correctly. If you do want to change automatic validation, please put forward a proposal, and we can discuss that.

Btw, here are some ideas for an API https://github.com/apache/incubator-pinot/issues/4035