Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Readers finished with Error " io.pravega.client.segment.impl.SegmentTruncatedException"

See original GitHub issue

While running heavy IO workload experiencing some of the readers are finished with io.pravega.client.segment.impl.SegmentTruncatedException

Workload details: 1000 readers and 1000 writers with a distribution of 10 reader/writer per POD and 100 workers, with 1024 routing keys, FIXED_NUM_SEGMENTS scale policy and retention enabled.

Reader failure:

INFO  [2019-08-20 14:19:32,550] io.pravega.longevity.utils.PerformanceUtils: Readers (7/10): events:0, events/sec:0, KB/sec:0.0

Exception:

2019-08-20 10:46:39,268 2303239 [pool-7-thread-1] ERROR i.p.l.t.w.readers.PravegaReader - Reader finished with Error
io.pravega.client.stream.TruncatedDataException: io.pravega.client.segment.impl.SegmentTruncatedException: java.util.concurrent.CompletionException: io.pravega.client.segment.impl.SegmentTruncatedException
        at io.pravega.client.state.impl.RevisionedStreamClientImpl$StreamIterator.next(RevisionedStreamClientImpl.java:155)
        at io.pravega.client.state.impl.RevisionedStreamClientImpl$StreamIterator.next(RevisionedStreamClientImpl.java:126)
        at io.pravega.client.state.impl.StateSynchronizerImpl.handleTruncation(StateSynchronizerImpl.java:104)
        at io.pravega.client.state.impl.StateSynchronizerImpl.fetchUpdates(StateSynchronizerImpl.java:93)
        at io.pravega.client.stream.impl.ReaderGroupStateManager.fetchUpdatesIfNeeded(ReaderGroupStateManager.java:290)
        at io.pravega.client.stream.impl.ReaderGroupStateManager.getCheckpoint(ReaderGroupStateManager.java:383)
        at io.pravega.client.stream.impl.EventStreamReaderImpl.updateGroupStateIfNeeded(EventStreamReaderImpl.java:179)
        at io.pravega.client.stream.impl.EventStreamReaderImpl.readNextEventInternal(EventStreamReaderImpl.java:109)
        at io.pravega.client.stream.impl.EventStreamReaderImpl.readNextEvent(EventStreamReaderImpl.java:94)
        at io.pravega.longevity.testworker.workers.readers.PravegaReader.start(PravegaReader.java:99)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.pravega.client.segment.impl.SegmentTruncatedException: java.util.concurrent.CompletionException: io.pravega.client.segment.impl.SegmentTruncatedException
        at io.pravega.client.segment.impl.SegmentInputStreamImpl.handleRequest(SegmentInputStreamImpl.java:141)
        at io.pravega.client.segment.impl.SegmentInputStreamImpl.read(SegmentInputStreamImpl.java:121)
        at io.pravega.client.segment.impl.EventSegmentReaderImpl.readEvent(EventSegmentReaderImpl.java:75)
        at io.pravega.client.segment.impl.EventSegmentReaderImpl.read(EventSegmentReaderImpl.java:62)
        at io.pravega.client.segment.impl.EventSegmentReader.read(EventSegmentReader.java:53)
        at io.pravega.client.state.impl.RevisionedStreamClientImpl$StreamIterator.next(RevisionedStreamClientImpl.java:151)
        ... 12 common frames omitted
Caused by: java.util.concurrent.CompletionException: io.pravega.client.segment.impl.SegmentTruncatedException
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
        at io.pravega.client.segment.impl.AsyncSegmentInputStreamImpl$ResponseProcessor.segmentIsTruncated(AsyncSegmentInputStreamImpl.java:87)
        at io.pravega.shared.protocol.netty.WireCommands$SegmentIsTruncated.process(WireCommands.java:202)
        at io.pravega.shared.protocol.netty.ReplyProcessor.process(ReplyProcessor.java:20)
        at io.pravega.client.netty.impl.FlowHandler.lambda$channelRead$2(FlowHandler.java:247)
        at java.base/java.util.Optional.ifPresent(Optional.java:183)
        at io.pravega.client.netty.impl.FlowHandler.channelRead(FlowHandler.java:245)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.pravega.shared.protocol.netty.ExceptionLoggingHandler.channelRead(ExceptionLoggingHandler.java:37)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:421)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:321)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        ... 1 common frames omitted
Caused by: io.pravega.client.segment.impl.SegmentTruncatedException: null

Pravega-Operator deployment with 2 controllers, 3 Zookeepers, 4 Bookies and 3 Segment stores.

Environment details: PKS / K8 with medium cluster:

3 master nodes @ large.cpu (4 CPU, 4 GB Ram, 16 GB Disk)
8 worker nodes @ xlarge.cpu(8 cpu, 16 GB Ram, 32 GB Disk)
Tier-1 storage is from VSAN datastore
Tier-2 storage curved on NFS Client provisioner using Isilon as backend

Pravega details:

Pravega version: 0.6.0-2333.fed6fd5
Pravega Operator: pravega/pravega-operator:0.4.2-rc0
Zookeeper Operator : pravega/zookeeper-operator:0.2.1
Zookeeper version:  pravega/zookeeper:0.2.2

Issue Analytics

State:
Created 4 years ago
Comments:8 (4 by maintainers)

Top GitHub Comments

1reaction

sumit-bmcommented, Aug 20, 2019

Still collecting… will post you in the internal channel

0reactions

sumit-bmcommented, Aug 20, 2019

below is complete workload details:

{
      "type": "Pravega",
      "name": "longevity/swarm",
      "forever": true,
      "scope": "longevity",
      "stream": "swarm",
      "createStream": true,
      "payload": {
        "numberOfKeys": 1024
      },
      "throttle": {
        "maxEventsPerSecond": 100,
        "maxOutstandingAcks": 1
      },
      "streamPolicies": {
        "scalingType": "FIXED_NUM_SEGMENTS",
        "targetRate": 2500,
        "scaleFactor": 2,
        "minNumSegments": 10,
        "retentionType": "LIMITED_TIME_MILLIS",
        "retentionLimit": 3600000
      },
      "tasks": [
        {
          "numReaders": 10,
          "numWriters": 10,
          "duplicates": 100
        }
      ],
      "longevityAssertions": {
        "isRunningState": null,
        "hasAtLeastXActiveReaders": 1000,
        "hasAtLeastXActiveWriters": 1000,
        "readerBytesAreIncreasing": null,
        "writerBytesAreIncreasing": null,
        "eventsInSequence": null
      }
    }