Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

During segment container ownership transfer readers get NoSuchSegment

See original GitHub issue

Problem description Readers were reading from a segment on a particular segment container. Then segmentContainer failover happened and post ownership transfer readers failed with NoSuchSegment. This lead to readers rereading data from offset 0 in the segment 0#epoch.10 again. It repeated 3 times before they were successfully able to read it.

segmentstore_2018-06-08.1.log:2018-06-08 14:24:26,941 103854 [core-10] WARN  i.p.s.s.h.h.PravegaRequestProcessor - Segment 'testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10' does not exist and cannot perform operation 'Read segment'.
segmentstore_2018-06-08.1.log:2018-06-08 14:24:27,974 104887 [core-24] WARN  i.p.s.s.h.h.PravegaRequestProcessor - Segment 'testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10' does not exist and cannot perform operation 'Read segment'.
segmentstore_2018-06-08.1.log:2018-06-08 14:24:29,016 105929 [core-28] WARN  i.p.s.s.h.h.PravegaRequestProcessor - Segment 'testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10' does not exist and cannot perform operation 'Read segment'.

Problem location SegmentStore

Suggestions for an improvement Fix the issue

Issue Analytics

State:
Created 5 years ago
Comments:12 (12 by maintainers)

Top GitHub Comments

1reaction

shridscommented, Jun 11, 2018

There are 5 readers in the test and each reader stops reading if it encounters a TruncationException. The below log snippets explain why duplicate events were read (4 instances of the same event were found)

==> First test reader got SegmentTruncationException from SSS after reading upto offset 59220+1170 = 60390

2018-06-08 14:24:25,645 320261 [epollEventLoopGroup-4-2] TRACE i.p.c.s.i.AsyncSegmentInputStreamImpl - Received read result WireCommands.SegmentRead(type=SEGMENT_READ, segment=testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10, offset=0, atTail=false, endOfSegment=false, data=java.nio.HeapByteBuffer[pos=0 lim=58320 cap=58320])
2018-06-08 14:24:25,746 320362 [epollEventLoopGroup-4-2] TRACE i.p.c.s.i.AsyncSegmentInputStreamImpl - Received read result WireCommands.SegmentRead(type=SEGMENT_READ, segment=testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10, offset=58320, atTail=false, endOfSegment=false, data=java.nio.HeapByteBuffer[pos=0 lim=900 cap=900])
2018-06-08 14:24:25,755 320371 [epollEventLoopGroup-4-2] TRACE i.p.c.s.i.AsyncSegmentInputStreamImpl - Received read result WireCommands.SegmentRead(type=SEGMENT_READ, segment=testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10, offset=59220, atTail=false, endOfSegment=false, data=java.nio.HeapByteBuffer[pos=0 lim=1170 cap=1170])
2018-06-08 14:24:26,941 321557 [epollEventLoopGroup-4-2] INFO  i.p.c.s.i.AsyncSegmentInputStreamImpl - Received noSuchSegment No such segment: testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10

==> The second reader acquired this segment and successfully read upto offset 60390 before throwing an exception.

2018-06-08 14:24:26,970 321586 [epollEventLoopGroup-4-2] TRACE i.p.c.s.i.AsyncSegmentInputStreamImpl - Received read result WireCommands.SegmentRead(type=SEGMENT_READ, segment=testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10, offset=0, atTail=false, endOfSegment=false, data=java.nio.HeapByteBuffer[pos=0 lim=60390 cap=60390])
2018-06-08 14:24:27,979 322595 [epollEventLoopGroup-4-2] INFO  i.p.c.s.i.AsyncSegmentInputStreamImpl - Received noSuchSegment No such segment: testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10

===> The third reader acquired this segment and successfully read upto offset 60390 before throwing an exception:

2018-06-08 14:24:28,002 322618 [epollEventLoopGroup-4-3] DEBUG i.p.client.netty.impl.RawClient - Received reply WireCommands.StreamSegmentInfo(type=STREAM_SEGMENT_INFO, requestId=1, segmentName=testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10, exists=true, isSealed=false, isDeleted=false, lastModified=1528467863209, writeOffset=85590, startOffset=0)
2018-06-08 14:24:28,012 322628 [epollEventLoopGroup-4-2] TRACE i.p.c.s.i.AsyncSegmentInputStreamImpl - Received read result WireCommands.SegmentRead(type=SEGMENT_READ, segment=testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10, offset=0, atTail=false, endOfSegment=false, data=java.nio.HeapByteBuffer[pos=0 lim=60390 cap=60390])
2018-06-08 14:24:29,023 323639 [epollEventLoopGroup-4-2] INFO  i.p.c.s.i.AsyncSegmentInputStreamImpl - Received noSuchSegment No such segment: testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10
2018-06-08 14:24:29,049 323665 [epollEventLoopGroup-4-5] DEBUG i.p.client.netty.impl.RawClient - Received reply WireCommands.StreamSegmentInfo(type=STREAM_SEGMENT_INFO, requestId=1, segmentName=testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10, exists=true, isSealed=false, isDeleted=false, lastModified=1528467863209, writeOffset=85590, startOffset=0)

===> The fourth reader successfully read this segment until it received a Segment sealed.

2018-06-08 14:24:29,052 323668 [epollEventLoopGroup-4-2] TRACE i.p.c.s.i.AsyncSegmentInputStreamImpl - Received read result WireCommands.SegmentRead(type=SEGMENT_READ, segment=testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10, offset=0, atTail=false, endOfSegment=false, data=java.nio.HeapByteBuffer[pos=0 lim=60390 cap=60390])
2018-06-08 14:24:30,122 324738 [epollEventLoopGroup-4-2] TRACE i.p.c.s.i.AsyncSegmentInputStreamImpl - Received read result WireCommands.SegmentRead(type=SEGMENT_READ, segment=testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10, offset=60390, atTail=false, endOfSegment=false, data=java.nio.HeapByteBuffer[pos=0 lim=180 cap=180])
2018-06-08 14:24:30,129 324745 [epollEventLoopGroup-4-2] TRACE i.p.c.s.i.AsyncSegmentInputStreamImpl - Received read result WireCommands.SegmentRead(type=SEGMENT_READ, segment=testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10, offset=60570, atTail=false, endOfSegment=false, data=java.nio.HeapByteBuffer[pos=0 lim=2340 cap=2340])
...
Last read is at offset `169830`
2018-06-08 14:25:42,191 396807 [epollEventLoopGroup-4-2] INFO  i.p.c.s.i.AsyncSegmentInputStreamImpl - Received segmentSealed WireCommands.SegmentIsSealed(type=SEGMENT_IS_SEALED, requestId=169830, segment=testReadTxnWriteAutoScaleScope2092305162/testReadTxnWriteAutoScaleStream/0#epoch.10)

0reactions

andreipaduroiucommented, Jun 13, 2018

Thanks for all the help in debugging this. After further digging in the logs (which are really, really stuffy), I realized that this was indeed the test failure cause. The test itself could be improved to make it more clear what failed (and when), but that is an issue for another time (#2688).

I have a PR out that hopefully fixes this corner case bug. This needs fixing, despite the fact that we had a good number of master build test successes recently.