Using StoreFileListener for removing CQ files - dealing with multiple onAcquired/onReleased events

We use CQ version 5.17.30, setting up a single CQ instance like this:

SingleChronicleQueueBuilder.single(queueDir)
                .rollCycle(RollCycles.valueOf(rollCycle))
                .storeFileListener(new QueueFileListener())
                .build();

and for writing to the queue: fileQueue.acquireAppender().writeText().

We see many onAcquired / onReleased calls: 2020-05-17 20:42:42,740 DEBUG h.l.a.i.QueueFileListener - Acquired file 20200517-18.cq4 … work … 2020-05-17 20:42:43,073 DEBUG h.l.a.i.QueueFileListener - Released file 20200517-18.cq4

And, presumably, when using multiple threads: 2020-05-17 20:42:44,926 DEBUG h.l.a.i.QueueFileListener - Acquired file 20200517-18.cq4 2020-05-17 20:42:44,926 DEBUG h.l.a.i.QueueFileListener - Acquired file 20200517-18.cq4 2020-05-17 20:42:44,927 DEBUG h.l.a.i.QueueFileListener - Acquired file 20200517-18.cq4 … work … 2020-05-17 20:42:45,091 DEBUG h.l.a.i.QueueFileListener - Released file 20200517-18.cq4 2020-05-17 20:42:45,093 DEBUG h.l.a.i.QueueFileListener - Released file 20200517-18.cq4 2020-05-17 20:42:45,094 DEBUG h.l.a.i.QueueFileListener - Released file 20200517-18.cq4

Questions on this behavior and the correct way to delete files:

1. Based on this Stack Overflow question and your answer to it: can you confirm that this is expected behavior?

2. onAcquired / onReleased are used to maintain a ConcurrentHashMap with pointers to files that can (eventually) be deleted (in a separate thread).

    @Override
    public void onAcquired(int cycle, File file) {

        if (file != null) {
            releasedFiles.remove(file);
            logger.debug(format("Acquired file %s", file.getAbsolutePath()));
        }
    }

    @Override
    public void onReleased(int cycle, File file) {

        if (file != null) {
            releasedFiles.put(file, new ReleasedFileData(cycle));
            logger.debug(format("Released file %s", file.getAbsolutePath()));
        }
    }

In ReleasedFileData we track the cycle and the current timestamp:

    public ReleasedFileData(int cycle) {
        this.cycle = cycle;
        this.releasedAt = LocalDateTime.now();
    }

and determine if the file can be deleted using the following logic:

    public boolean fileCanBeDeleted(int currentCycle) {

        LocalDateTime earlier = LocalDateTime.now().minusMinutes(10);
        return cycle < currentCycle && releasedAt.isBefore(earlier);
    }

Are we overly cautious, e.g. would return cycle < currentCycle be sufficient?

Thanks and kudos on the great product!

[sjoerd-michels]

Thanks for your fast feedback and for confirming that this is the expected behavior. Perhaps you could update the documentation, because it means that files cannot be deleted based on the onReleased event only. I would also appreciate your feedback on the “time interval safety check”, for now we are keeping that in place.

[dpisklov]

@sjoerd-michels you’re welcome, although in the future we would appreciate if you raise questions on stackoverflow (we monitor it closely), we are trying to keep github for the real code issues.