java.lang.IllegalStateException: Unable to index 268435456

With the attached appender and tailer after ~268M messages I get the following exception

Exception in thread "main" java.lang.IllegalStateException: Unable to index 268435456
	at net.openhft.chronicle.queue.impl.single.SCQIndexing.setPositionForSequenceNumber(SCQIndexing.java:625)
	at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueStore.setPositionForSequenceNumber(SingleChronicleQueueStore.java:350)
	at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreAppender.writeIndexForPosition(SingleChronicleQueueExcerpts.java:689)
	at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreAppender$StoreAppenderContext.close(SingleChronicleQueueExcerpts.java:818)
	at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreAppender$StoreAppenderContext.close(SingleChronicleQueueExcerpts.java:784)
	at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreAppender.writeBytes(SingleChronicleQueueExcerpts.java:174)
	at bitmex.gateway.trading.engine.chronicle.TimestampAppender.main(TimestampAppender.java:21)

And at that point the queue is corrupt.

import net.openhft.chronicle.queue.ExcerptAppender;
import net.openhft.chronicle.queue.RollCycles;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueue;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder;

public class TimestampAppender {

    public static void main(String[] args) {

        SingleChronicleQueue build = SingleChronicleQueueBuilder
                .binary("src/test/resources/sandbox/times_publication")
                .rollCycle(RollCycles.HOURLY)
                .timeProvider(System::currentTimeMillis)
                .build();

        ExcerptAppender excerptAppender = build.acquireAppender().methodWriterBuilder().recordHistory();

        while (true) {
            excerptAppender.writeBytes(bytesOut -> {
                bytesOut.writeLong(System.nanoTime());
            });
        }
    }
}

import net.openhft.chronicle.queue.ExcerptTailer;
import net.openhft.chronicle.queue.RollCycles;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueue;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder;
import net.openhft.chronicle.wire.DocumentContext;

public class TimestampTailer {

    public static void main(String[] args) throws InterruptedException {
        SingleChronicleQueue build = SingleChronicleQueueBuilder
                .binary("src/test/resources/sandbox/times_publication")
                .rollCycle(RollCycles.HOURLY)
                .timeProvider(System::currentTimeMillis)
                .build();

        ExcerptTailer tailer = build.createTailer();

        long previousTimestamp = 0;
        long counter = 0;

        while (true) {
            try (DocumentContext documentContext = tailer.readingDocument()) {
                if (!documentContext.isPresent()) {
                    Thread.sleep(1);
                    continue;
                }

                long timestamp = documentContext.wire().bytes().readLong();
                if (timestamp < previousTimestamp) {
                    throw new RuntimeException();
                }

                ++counter;
                if (counter % 1000000 == 0) {
                    System.out.println("Seen " + counter + " events");
                }

                previousTimestamp = timestamp;
            }
        }
    }
}

uname -a: Darwin MacBook-Pro 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64 JDK:

openjdk version "1.8.0_252"
OpenJDK Runtime Environment (Zulu 8.46.0.19-CA-macosx) (build 1.8.0_252-b14)
OpenJDK 64-Bit Server VM (Zulu 8.46.0.19-CA-macosx) (build 25.252-b14, mixed mode)

Chronicle 5.19.32

[dpisklov]

@mikeb01 the time is in the name of the roll cycle. Queue rolls starting from midnight UTC, so e.g. for *HOURLY roll cycles (there’re few) the roll will happen at the hour boundary. As for the number of messages left, it’s max number of messages per roll cycle minus current sequence number (RollCycle#toSequenceNumber(long) can help you to convert output of appender#index() or tailer#index() to the sequence number within the current roll cycle)

[mikeb01]

I can confirm that the queue is not corrupt, it was just continuing to through the IllegalStateException until the hourly rollover.