When first and last cycle are out of date, should we try refreshing them and retrying before we throw?

In StoreTailer.toEnd, we get the lastCycle which is only checked against the file listing every 60s. In the event we’ve got lastCycle then failed to retrieve it from the SingleChronicleQueueStore is it worth forcing a directoryListing.refresh(true) and retrying before throwing an exception?

[nicktindall]

I think the scenario we saw was in optimizedToEnd, where an IllegalStateException is thrown

https://github.com/OpenHFT/Chronicle-Queue/blob/db7a1ecb65fa6b7a97f8c6b01f7ff2da3ef3b0b7/src/main/java/net/openhft/chronicle/queue/impl/single/StoreTailer.java#L887-L891

It’s also repeated in approximateLastCycle2: https://github.com/OpenHFT/Chronicle-Queue/blob/db7a1ecb65fa6b7a97f8c6b01f7ff2da3ef3b0b7/src/main/java/net/openhft/chronicle/queue/impl/single/StoreTailer.java#L748-L752 but that seems to be retried already https://github.com/OpenHFT/Chronicle-Queue/blob/db7a1ecb65fa6b7a97f8c6b01f7ff2da3ef3b0b7/src/main/java/net/openhft/chronicle/queue/impl/single/StoreTailer.java#L768-L775

This is probably seperate enough to what you’re working on that I could try and do a PR for this today, it doesn’t seem that hard

[glukos]

I’m ok with closing this.

In my understanding, https://github.com/OpenHFT/Chronicle-Queue/issues/924 is mostly about performance. It will save us from unnecessary refreshes, but it will not fix a data race - obviously, WatchService doesn’t priovide any guarantees on how fast events will arrive. As I see it, the initial purpose of refresh was dealing with manual manipulations with queue dir. Ideally, we shouldn’t rely on the refresh to resolve data races that happen in normal workflow.

Some more spit-balling - that’s how tailers should find out that a new cycle is created:

                    if (!readOnly && createIfAbsent && wire.writeFirstHeader()) {
                        // implicitly reserves the wireStore for this StoreSupplier
                        wireStore = storeFactory.apply(that, wire);
                        wire.updateFirstHeader();
                        wire.usePadding(wireStore.dataVersion() > 0);

                        wireStore.initIndex(wire);
                        // do not allow tailer to see the file until it's header is written
                        directoryListing.onFileCreated(path, cycle); // increments shared variable
                        // allow directoryListing to pick up the file immediately

Firstly, we write header to a mmapped memory, then we increment shared counter. Tailer in another thread / process may go here:

                if (!createIfAbsent &&
                        (cycle > directoryListing.getMaxCreatedCycle()
                                || cycle < directoryListing.getMinCreatedCycle()
                                || !path.exists())) {
                    return null;

and find updated getMaxCreatedCycle(), but !path.exists() way be true as per file creation hasn’t been synced by OS yet. Should we consider spinning for a while here before returning null?