Save and restore questions for RaftStore and StateMachine

Hello,

I’m quite interested in implementing microraft in one of my internal project, however I’m not sure I did understand everything, even after several read and re-read, and 2 nights thinking about it 😃

If you’re not too busy to help, it will me much appreciated!

To better understand the library, I started to replay part of the tutorial, oriented on my specific needs (let’s say a sort of kv to keep it simple):

• run in multiple jvm instead of multi-thread, to have serialization and network
• network is very basic socket to keep it simple until I have everything I want working
• have high availability and failover and resiliency…

My questions at this time is around the RaftStore and StateMachine. I understand I need to implement recovery in RaftStore and provide RestoredRaftState on startup, but not sure what to save (on disk at first) to implement this. It seems one need to save (so we can restore): localEndpoint, localEnpointVoting, initialGroupMembers and term. All these are easy to do. Then comes the log entry and snapshot chunks, and here I’m quite lost, because from the only implemtation I can see (InMemoryRaftStore), it is using a RaftLog to do so, but the code looks weird to me:

• persistSnapshotChunk: is the code reusable like it or should I so something completely different?
• truncateSnapshotChunksUntil: I’m not sure clear() is the good thing I should do… I had the impression this was a fast go to implement the store, but I may be wrong.

Concerning the StateMachine, I wonder wether I should also save the data to disk, or would everything be replayed? Does it mean that during Store save, I should also save the state machine, or is ti 2 totally different things to do (as per the doc), and if so when should it happen?

    @Override
    public void persistSnapshotChunk(SnapshotChunk snapshotChunk) throws IOException {
        snapshotChunks.add(snapshotChunk);

        if (snapshotChunk.getSnapshotChunkCount() == snapshotChunks.size()) {
            snapshotChunks.sort(comparingInt(SnapshotChunk::getSnapshotChunkIndex));
            SnapshotEntry snapshotEntry = new DefaultSnapshotEntryOrBuilder().setTerm(snapshotChunk.getTerm())
                    .setIndex(snapshotChunk.getIndex())
                    .setSnapshotChunks(snapshotChunks)
                    .setGroupMembersView(
                            snapshotChunk.getGroupMembersView())
                    .build();
            raftLog.setSnapshot(snapshotEntry);
            snapshotChunks = new ArrayList<>();
        }
    }

    @Override
    public void truncateSnapshotChunksUntil(long logIndexInclusive) throws IOException {
        snapshotChunks.clear();
    }

Regards,

[momiji]

Yes, for serialization, this is exactly what I’m doing.

For each object, I just get the fields values needed by the builder, serialize them “my” way, then use the builder back to get the object. This way I’m sure everything will be replayable in the future, if I have coded “my” way correctly (with serials for examples).

This why I said wow!, this is great that this was already implemented in your code!

[metanet]

Serialization

https://github.com/MicroRaft/MicroRaft/tree/master/microraft/src/main/java/io/microraft/model this package mostly works with builders and interfaces to enable developers to use different serde and rpc protocols under the hood. For e.g., you can use protobuf to represent MicroRaft’s model interfaces and wrap them in the implementations of the model interfaces. Then those wrappers can delegate the set and get calls to the underlying protobuf message objects. See the following example:

public class ProtobufLogEntry implements LogEntry, LogEntryBuilder {

    private LogEntryProto.Builder builder;
    private LogEntryProto entry;

    public ProtobufLogEntry() {
        this.builder = LogEntryProto.newBuilder();
    }

    public LogEntryProto getEntry() {
        return entry;
    }

    @Nonnull
    @Override
    public ProtobufLogEntry setIndex(long index) {
        builder.setIndex(index);
        return this;
    }

    // other setter impls to populate this.builder's fields

    @Nonnull
    @Override
    public LogEntry build() {
        entry = builder.build();
        builder = null;
        return this;
    }

    @Override
    public String toString() {
        if (builder != null) {
            return "ProtobufLogEntry{builder=" + builder + "}";
        }

        return "ProtobufLogEntry{" + "index=" + getIndex() + ", term=" + getTerm() + ", operation=" + getOperation() + '}';
    }

    @Override
    public long getIndex() {
        return entry.getIndex();
    }

    @Override
    public int getTerm() {
        return entry.getTerm();
    }

    // other getter methods.
}

Then in your transport implementation you can extract protobufs objects from these wraps send send them over the wire.

Future tests

MicroRaft doesn’t support the witness feature now. I am not sure if it worths the complexity because I think it brings a bit of complexity to the quorum calculations. But if there is a real need we can check again. IF we assume witness functionality is available in MicroRaft, the scenario you describe is ok.