Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

RaftNode example eventually stops making progress when saving 10 times x second

See original GitHub issue

This is about the issues being discussed here https://github.com/dotnet/dotNext/issues/184#issuecomment-1660673297.

The issue has been reproduced both in linux arm 32 bit in raspberry os (debian based) and linux debian 64 bit in wsl. At least one reproduction had all 3 nodes running, but due to the investigation of a different issue the cluster was often running with 2 out of the 3 nodes.

There are 2 observations surrounding the issue:

  1. there are often leader changes, so the term can easily reach 200 if left running for 1-3 hours
  2. eventually progress stops. When this happens there are leader change events reported.

Presumably the frequent leader changes make the issue easier to reproduce, as there is a chance it only happens during leader changes.

The issue has been reproduced in the 14.13, 14.12.3 and 4.4.1 and in all cases it has taken under a day to get to the failure. The specific settings + entries + behavior + even console output shown over ssh could play a part in making the issue more likely to reproduce, as we have systems that run for months with 4.4.1 writing entries to raft 10 times x second.

In 14.13 and 14.12.3 the issue was observed with similar output to https://github.com/dotnet/dotNext/issues/184#issuecomment-1660235577.

I only made one run in 4.4.1 where progress did stop but the output was different. While in the later versions things suddenly stop after the leader change and potentially a single TaskCanceledException (not sure if it always gets into the screen), in the 4.4.1 run there were a various TaskCanceledException that happened before the stop + some “fail” log entries with “EndOfStreamException: Attempted to read past the end of the stream” by the leader.

I share the detailed 4.4.1 in case the issue was still the same and the old version logs extra info. Of course there is a risk the old version failure was not the same type of failure and is irrelevant for the issue present in the latest versions.

Leader:

Saving value 2446000 generated by the leader node
Accepting value 2446000
Saving value 2446500 generated by the leader node
Accepting value 2446500
Saving value 2447000 generated by the leader node
Accepting value 2447000
Saving value 2447500 generated by the leader node
Accepting value 2447500
Saving value 2448000 generated by the leader node
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\DataModifier.cs:line 32
Consensus cannot be reached
Term of local cluster member is 50. Election timeout 00:00:00.1500000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 53. Election timeout 00:00:00.1880000
Consensus cannot be reached
Term of local cluster member is 53. Election timeout 00:00:00.1880000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 53. Election timeout 00:00:00.1880000
Accepting value 2448000
Accepting value 2448000
Accepting value 2448500
Accepting value 2449000
Building snapshot
fail: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
      Failed to process request from 192.168.100.8:59192
      System.IO.EndOfStreamException: Attempted to read past the end of the stream.
         at DotNext.IO.FileReader.ReadBlockAsync(Memory`1 output, CancellationToken token) in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 404
         at DotNext.IO.FileReader.<>c__DisplayClass1_0`1.<<ReadAsync>g__ReadSlowAsync|0>d.MoveNext() in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 71
      --- End of stack trace from previous location ---
         at RaftNode.SimplePersistentState.SimpleSnapshotBuilder.ApplyAsync(LogEntry entry) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\SimplePersistentState.cs:line 22
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.BuildSnapshotAsync(Int32 sessionId, Int64 upperBoundIndex, SnapshotBuilder builder, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 107
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ForceSequentialCompactionAsync(Int32 sessionId, Int64 upperBoundIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 608
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass23_0`1.<<AppendAndCommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 380
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.AppendAndCommitAsync[TEntry](ILogEntryProducer`1 entries, Int64 startIndex, Boolean skipCommitted, Int64 commitIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 429
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 488
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 133
Consensus cannot be reached
Term of local cluster member is 53. Election timeout 00:00:00.1880000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 54. Election timeout 00:00:00.1880000
Saving value 2449500 generated by the leader node
Accepting value 2449500
Accepting value 2449500
Building snapshot
Consensus cannot be reached
Term of local cluster member is 54. Election timeout 00:00:00.1880000
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\DataModifier.cs:line 32
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 55. Election timeout 00:00:00.1880000
Accepting value 2449500
Building snapshot
fail: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
      Failed to process request from 192.168.100.8:59212
      System.IO.EndOfStreamException: Attempted to read past the end of the stream.
         at DotNext.IO.FileReader.ReadBlockAsync(Memory`1 output, CancellationToken token) in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 404
         at DotNext.IO.FileReader.<>c__DisplayClass1_0`1.<<ReadAsync>g__ReadSlowAsync|0>d.MoveNext() in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 71
      --- End of stack trace from previous location ---
         at RaftNode.SimplePersistentState.SimpleSnapshotBuilder.ApplyAsync(LogEntry entry) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\SimplePersistentState.cs:line 22
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.BuildSnapshotAsync(Int32 sessionId, Int64 upperBoundIndex, SnapshotBuilder builder, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 107
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ForceSequentialCompactionAsync(Int32 sessionId, Int64 upperBoundIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 608
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass23_0`1.<<AppendAndCommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 380
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.AppendAndCommitAsync[TEntry](ILogEntryProducer`1 entries, Int64 startIndex, Boolean skipCommitted, Int64 commitIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 429
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 488
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 133
Consensus cannot be reached
Term of local cluster member is 55. Election timeout 00:00:00.1880000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 56. Election timeout 00:00:00.1880000
Saving value 2450000 generated by the leader node
Accepting value 2450000
Accepting value 2450000
Building snapshot
Consensus cannot be reached
Term of local cluster member is 56. Election timeout 00:00:00.1880000
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\DataModifier.cs:line 32
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 57. Election timeout 00:00:00.1880000
Accepting value 2450000
Building snapshot
fail: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
      Failed to process request from 192.168.100.8:59224
      System.IO.EndOfStreamException: Attempted to read past the end of the stream.
         at DotNext.IO.FileReader.ReadBlockAsync(Memory`1 output, CancellationToken token) in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 404
         at DotNext.IO.FileReader.<>c__DisplayClass1_0`1.<<ReadAsync>g__ReadSlowAsync|0>d.MoveNext() in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 71
      --- End of stack trace from previous location ---
         at RaftNode.SimplePersistentState.SimpleSnapshotBuilder.ApplyAsync(LogEntry entry) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\SimplePersistentState.cs:line 22
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.BuildSnapshotAsync(Int32 sessionId, Int64 upperBoundIndex, SnapshotBuilder builder, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 107
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ForceSequentialCompactionAsync(Int32 sessionId, Int64 upperBoundIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 608
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass23_0`1.<<AppendAndCommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 380
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.AppendAndCommitAsync[TEntry](ILogEntryProducer`1 entries, Int64 startIndex, Boolean skipCommitted, Int64 commitIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 429
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 488
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 133
Consensus cannot be reached
Term of local cluster member is 57. Election timeout 00:00:00.1880000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 58. Election timeout 00:00:00.1880000
Saving value 2450500 generated by the leader node
Accepting value 2450500
Accepting value 2450500
Building snapshot
Consensus cannot be reached
Term of local cluster member is 58. Election timeout 00:00:00.1880000
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\DataModifier.cs:line 32
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 59. Election timeout 00:00:00.1880000
Accepting value 2450500
Building snapshot
fail: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
      Failed to process request from 192.168.100.8:59236
      System.IO.EndOfStreamException: Attempted to read past the end of the stream.
         at DotNext.IO.FileReader.ReadBlockAsync(Memory`1 output, CancellationToken token) in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 404
         at DotNext.IO.FileReader.<>c__DisplayClass1_0`1.<<ReadAsync>g__ReadSlowAsync|0>d.MoveNext() in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 71
      --- End of stack trace from previous location ---
         at RaftNode.SimplePersistentState.SimpleSnapshotBuilder.ApplyAsync(LogEntry entry) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\SimplePersistentState.cs:line 22
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.BuildSnapshotAsync(Int32 sessionId, Int64 upperBoundIndex, SnapshotBuilder builder, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 107
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ForceSequentialCompactionAsync(Int32 sessionId, Int64 upperBoundIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 608
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass23_0`1.<<AppendAndCommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 380
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.AppendAndCommitAsync[TEntry](ILogEntryProducer`1 entries, Int64 startIndex, Boolean skipCommitted, Int64 commitIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 429
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 488
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 133
Consensus cannot be reached
Term of local cluster member is 59. Election timeout 00:00:00.1880000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 60. Election timeout 00:00:00.1880000
Saving value 2451000 generated by the leader node
Accepting value 2451000
Accepting value 2451000
Building snapshot
Consensus cannot be reached
Term of local cluster member is 60. Election timeout 00:00:00.1880000
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\DataModifier.cs:line 32
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 61. Election timeout 00:00:00.1880000
Accepting value 2451000
Building snapshot
fail: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
      Failed to process request from 192.168.100.8:59248
      System.IO.EndOfStreamException: Attempted to read past the end of the stream.
         at DotNext.IO.FileReader.ReadBlockAsync(Memory`1 output, CancellationToken token) in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 404
         at DotNext.IO.FileReader.<>c__DisplayClass1_0`1.<<ReadAsync>g__ReadSlowAsync|0>d.MoveNext() in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 71
      --- End of stack trace from previous location ---
         at RaftNode.SimplePersistentState.SimpleSnapshotBuilder.ApplyAsync(LogEntry entry) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\SimplePersistentState.cs:line 22
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.BuildSnapshotAsync(Int32 sessionId, Int64 upperBoundIndex, SnapshotBuilder builder, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 107
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ForceSequentialCompactionAsync(Int32 sessionId, Int64 upperBoundIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 608
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass23_0`1.<<AppendAndCommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 380
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.AppendAndCommitAsync[TEntry](ILogEntryProducer`1 entries, Int64 startIndex, Boolean skipCommitted, Int64 commitIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 429
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 488
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 133
Consensus cannot be reached
Term of local cluster member is 61. Election timeout 00:00:00.1880000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 62. Election timeout 00:00:00.1880000
Saving value 2451500 generated by the leader node
Accepting value 2451500
Accepting value 2451500
Building snapshot
Consensus cannot be reached
Term of local cluster member is 62. Election timeout 00:00:00.1880000
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\DataModifier.cs:line 32
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 63. Election timeout 00:00:00.1880000
Accepting value 2451500
Building snapshot
fail: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
      Failed to process request from 192.168.100.8:59260
      System.IO.EndOfStreamException: Attempted to read past the end of the stream.
         at DotNext.IO.FileReader.ReadBlockAsync(Memory`1 output, CancellationToken token) in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 404
         at DotNext.IO.FileReader.<>c__DisplayClass1_0`1.<<ReadAsync>g__ReadSlowAsync|0>d.MoveNext() in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 71
      --- End of stack trace from previous location ---
         at RaftNode.SimplePersistentState.SimpleSnapshotBuilder.ApplyAsync(LogEntry entry) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\SimplePersistentState.cs:line 22
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.BuildSnapshotAsync(Int32 sessionId, Int64 upperBoundIndex, SnapshotBuilder builder, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 107
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ForceSequentialCompactionAsync(Int32 sessionId, Int64 upperBoundIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 608
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass23_0`1.<<AppendAndCommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 380
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.AppendAndCommitAsync[TEntry](ILogEntryProducer`1 entries, Int64 startIndex, Boolean skipCommitted, Int64 commitIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 429
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 488
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 133
Consensus cannot be reached
Term of local cluster member is 63. Election timeout 00:00:00.1880000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 64. Election timeout 00:00:00.1880000
Saving value 2452000 generated by the leader node
Accepting value 2452000
Accepting value 2452000
Building snapshot
Consensus cannot be reached
Term of local cluster member is 64. Election timeout 00:00:00.1880000
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\DataModifier.cs:line 32
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 65. Election timeout 00:00:00.1880000
Accepting value 2452000
Building snapshot
fail: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
      Failed to process request from 192.168.100.8:59272
      System.IO.EndOfStreamException: Attempted to read past the end of the stream.
         at DotNext.IO.FileReader.ReadBlockAsync(Memory`1 output, CancellationToken token) in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 404
         at DotNext.IO.FileReader.<>c__DisplayClass1_0`1.<<ReadAsync>g__ReadSlowAsync|0>d.MoveNext() in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 71
      --- End of stack trace from previous location ---
         at RaftNode.SimplePersistentState.SimpleSnapshotBuilder.ApplyAsync(LogEntry entry) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\SimplePersistentState.cs:line 22
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.BuildSnapshotAsync(Int32 sessionId, Int64 upperBoundIndex, SnapshotBuilder builder, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 107
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ForceSequentialCompactionAsync(Int32 sessionId, Int64 upperBoundIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 608
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass23_0`1.<<AppendAndCommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 380
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.AppendAndCommitAsync[TEntry](ILogEntryProducer`1 entries, Int64 startIndex, Boolean skipCommitted, Int64 commitIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 429
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 488
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 133
Consensus cannot be reached
Term of local cluster member is 65. Election timeout 00:00:00.1880000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 66. Election timeout 00:00:00.1880000
Saving value 2452500 generated by the leader node
Accepting value 2452500
Accepting value 2452500
Building snapshot
Consensus cannot be reached
Term of local cluster member is 66. Election timeout 00:00:00.1880000
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\DataModifier.cs:line 32
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 67. Election timeout 00:00:00.1880000
Accepting value 2452500
Building snapshot
fail: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
      Failed to process request from 192.168.100.8:59284
      System.IO.EndOfStreamException: Attempted to read past the end of the stream.
         at DotNext.IO.FileReader.ReadBlockAsync(Memory`1 output, CancellationToken token) in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 404
         at DotNext.IO.FileReader.<>c__DisplayClass1_0`1.<<ReadAsync>g__ReadSlowAsync|0>d.MoveNext() in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 71
      --- End of stack trace from previous location ---
         at RaftNode.SimplePersistentState.SimpleSnapshotBuilder.ApplyAsync(LogEntry entry) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\SimplePersistentState.cs:line 22
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.BuildSnapshotAsync(Int32 sessionId, Int64 upperBoundIndex, SnapshotBuilder builder, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 107
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ForceSequentialCompactionAsync(Int32 sessionId, Int64 upperBoundIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 608
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass23_0`1.<<AppendAndCommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 380
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.AppendAndCommitAsync[TEntry](ILogEntryProducer`1 entries, Int64 startIndex, Boolean skipCommitted, Int64 commitIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 429
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 488
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 133
Consensus cannot be reached
Term of local cluster member is 67. Election timeout 00:00:00.1880000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 68. Election timeout 00:00:00.1880000
Saving value 2453000 generated by the leader node
Accepting value 2453000
Accepting value 2453000
Building snapshot
Consensus cannot be reached
Term of local cluster member is 68. Election timeout 00:00:00.1880000
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\DataModifier.cs:line 32
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 69. Election timeout 00:00:00.1880000
Accepting value 2453000
Building snapshot
fail: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
      Failed to process request from 192.168.100.8:59296
      System.IO.EndOfStreamException: Attempted to read past the end of the stream.
         at DotNext.IO.FileReader.ReadBlockAsync(Memory`1 output, CancellationToken token) in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 404
         at DotNext.IO.FileReader.<>c__DisplayClass1_0`1.<<ReadAsync>g__ReadSlowAsync|0>d.MoveNext() in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 71
      --- End of stack trace from previous location ---
         at RaftNode.SimplePersistentState.SimpleSnapshotBuilder.ApplyAsync(LogEntry entry) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\SimplePersistentState.cs:line 22
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.BuildSnapshotAsync(Int32 sessionId, Int64 upperBoundIndex, SnapshotBuilder builder, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 107
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ForceSequentialCompactionAsync(Int32 sessionId, Int64 upperBoundIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 608
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass23_0`1.<<AppendAndCommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 380
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.AppendAndCommitAsync[TEntry](ILogEntryProducer`1 entries, Int64 startIndex, Boolean skipCommitted, Int64 commitIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 429
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 488
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 133
Consensus cannot be reached
Term of local cluster member is 69. Election timeout 00:00:00.1880000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 70. Election timeout 00:00:00.1880000
Saving value 2453500 generated by the leader node
Accepting value 2453500
Accepting value 2453500
Building snapshot
Consensus cannot be reached
Term of local cluster member is 70. Election timeout 00:00:00.1880000
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\DataModifier.cs:line 32
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 71. Election timeout 00:00:00.1880000
Accepting value 2453500
Building snapshot
fail: DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer[74028]
      Failed to process request from 192.168.100.8:59308
      System.IO.EndOfStreamException: Attempted to read past the end of the stream.
         at DotNext.IO.FileReader.ReadBlockAsync(Memory`1 output, CancellationToken token) in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 404
         at DotNext.IO.FileReader.<>c__DisplayClass1_0`1.<<ReadAsync>g__ReadSlowAsync|0>d.MoveNext() in /_/src/DotNext.IO/IO/FileReader.Binary.cs:line 71
      --- End of stack trace from previous location ---
         at RaftNode.SimplePersistentState.SimpleSnapshotBuilder.ApplyAsync(LogEntry entry) in C:\Users\fredd\source\repos\dotNext\src\Examples\RaftNode\SimplePersistentState.cs:line 22
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.BuildSnapshotAsync(Int32 sessionId, Int64 upperBoundIndex, SnapshotBuilder builder, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 107
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.ForceSequentialCompactionAsync(Int32 sessionId, Int64 upperBoundIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 608
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.<>c__DisplayClass23_0`1.<<AppendAndCommitAsync>g__CommitAndCompactSequentiallyAsync|0>d.MoveNext() in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 380
      --- End of stack trace from previous location ---
         at DotNext.Net.Cluster.Consensus.Raft.MemoryBasedStateMachine.AppendAndCommitAsync[TEntry](ILogEntryProducer`1 entries, Int64 startIndex, Boolean skipCommitted, Int64 commitIndex, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/MemoryBasedStateMachine.cs:line 429
         at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.AppendEntriesAsync[TEntry](ClusterMemberId sender, Int64 senderTerm, ILogEntryProducer`1 entries, Int64 prevLogIndex, Int64 prevLogTerm, Int64 commitIndex, IClusterConfiguration config, Boolean applyConfig, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 488
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.AppendEntriesAsync(ProtocolStream protocol, CancellationToken token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 251
         at DotNext.Net.Cluster.Consensus.Raft.Tcp.TcpServer.HandleConnection(Socket remoteClient) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/Tcp/TcpServer.cs:line 133
Consensus cannot be reached
Term of local cluster member is 71. Election timeout 00:00:00.1880000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 72. Election timeout 00:00:00.1880000
Accepting value 2454000
Building snapshot
Saving value 2454500 generated by the leader node

Follower

Accepting value 2446000                                                                           
Accepting value 2446500
Accepting value 2447000
Accepting value 2447500
Consensus cannot be reached
Term of local cluster member is 50. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 53. Election timeout 00:00:00.2550000
Saving value 2448000 generated by the leader node
Accepting value 2448000
Accepting value 2448000
Saving value 2448500 generated by the leader node
Accepting value 2448500
Saving value 2449000 generated by the leader node
Accepting value 2449000
Building snapshot
Saving value 2449500 generated by the leader node
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToke
n token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dot
Next\src\Examples\RaftNode\DataModifier.cs:line 32
Consensus cannot be reached
Term of local cluster member is 53. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.1:3262                               
Term of local cluster member is 54. Election timeout 00:00:00.2550000
Consensus cannot be reached
Term of local cluster member is 54. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 55. Election timeout 00:00:00.2550000
Saving value 2449500 generated by the leader node
Accepting value 2449500
Accepting value 2449500
Accepting value 2449500
Saving value 2450000 generated by the leader node
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToke
n token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dot
Next\src\Examples\RaftNode\DataModifier.cs:line 32
Consensus cannot be reached
Term of local cluster member is 55. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 56. Election timeout 00:00:00.2550000
Consensus cannot be reached
Term of local cluster member is 56. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 57. Election timeout 00:00:00.2550000
Saving value 2450000 generated by the leader node                                                 
Accepting value 2450000
Accepting value 2450000
Accepting value 2450000
Saving value 2450500 generated by the leader node
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToke
n token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dot
Next\src\Examples\RaftNode\DataModifier.cs:line 32
Consensus cannot be reached
Term of local cluster member is 57. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 58. Election timeout 00:00:00.2550000
Consensus cannot be reached
Term of local cluster member is 58. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 59. Election timeout 00:00:00.2550000
Saving value 2450500 generated by the leader node
Accepting value 2450500
Accepting value 2450500
Accepting value 2450500
Saving value 2451000 generated by the leader node
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, Cancell[95/1924]
n token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dot
Next\src\Examples\RaftNode\DataModifier.cs:line 32
Consensus cannot be reached
Term of local cluster member is 59. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 60. Election timeout 00:00:00.2550000
Consensus cannot be reached
Term of local cluster member is 60. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 61. Election timeout 00:00:00.2550000
Saving value 2451000 generated by the leader node
Accepting value 2451000
Accepting value 2451000
Accepting value 2451000
Saving value 2451500 generated by the leader node
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToke
n token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dot
Next\src\Examples\RaftNode\DataModifier.cs:line 32
Consensus cannot be reached
Term of local cluster member is 61. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.1:3262                                
Term of local cluster member is 62. Election timeout 00:00:00.2550000
Consensus cannot be reached
Term of local cluster member is 62. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 63. Election timeout 00:00:00.2550000
Saving value 2451500 generated by the leader node
Accepting value 2451500
Accepting value 2451500
Accepting value 2451500
Saving value 2452000 generated by the leader node
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToke
n token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dot
Next\src\Examples\RaftNode\DataModifier.cs:line 32
Consensus cannot be reached
Term of local cluster member is 63. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 64. Election timeout 00:00:00.2550000
Consensus cannot be reached
Term of local cluster member is 64. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 65. Election timeout 00:00:00.2550000
Saving value 2452000 generated by the leader node                                                  
Accepting value 2452000
Accepting value 2452000
Accepting value 2452000
Saving value 2452500 generated by the leader node
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToke
n token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dot
Next\src\Examples\RaftNode\DataModifier.cs:line 32
Consensus cannot be reached
Term of local cluster member is 65. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 66. Election timeout 00:00:00.2550000
Consensus cannot be reached
Term of local cluster member is 66. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 67. Election timeout 00:00:00.2550000
Saving value 2452500 generated by the leader node
Accepting value 2452500
Accepting value 2452500
Accepting value 2452500
Saving value 2453000 generated by the leader node
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, Cancell[23/1924]
n token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dot
Next\src\Examples\RaftNode\DataModifier.cs:line 32
Consensus cannot be reached
Term of local cluster member is 67. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 68. Election timeout 00:00:00.2550000
Consensus cannot be reached
Term of local cluster member is 68. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 69. Election timeout 00:00:00.2550000
Saving value 2453000 generated by the leader node
Accepting value 2453000
Accepting value 2453000
Accepting value 2453000
Saving value 2453500 generated by the leader node
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToke
n token) in /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dot
Next\src\Examples\RaftNode\DataModifier.cs:line 32
Consensus cannot be reached
Term of local cluster member is 69. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 70. Election timeout 00:00:00.2550000
Consensus cannot be reached
Term of local cluster member is 70. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.8:3262
Term of local cluster member is 71. Election timeout 00:00:00.2550000
Saving value 2453500 generated by the leader node
Accepting value 2453500
Accepting value 2453500
Accepting value 2453500
Saving value 2454000 generated by the leader node
Unexpected error System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at DotNext.Net.Cluster.Consensus.Raft.RaftCluster`1.ReplicateAsync[TEntry](TEntry entry, CancellationToken token) in
 /_/src/cluster/DotNext.Net.Cluster/Net/Cluster/Consensus/Raft/RaftCluster.cs:line 898
   at RaftNode.DataModifier.ExecuteAsync(CancellationToken stoppingToken) in C:\Users\fredd\source\repos\dotNext\src\Ex
amples\RaftNode\DataModifier.cs:line 32
Consensus cannot be reached
Term of local cluster member is 71. Election timeout 00:00:00.2550000
New cluster leader is elected. Leader address is 192.168.100.1:3262
Term of local cluster member is 72. Election timeout 00:00:00.2550000
Consensus cannot be reached
Term of local cluster member is 72. Election timeout 00:00:00.2550000

Issue Analytics

  • State:closed
  • Created 2 months ago
  • Comments:12 (6 by maintainers)

github_iconTop GitHub Comments

1reaction
saknocommented, Aug 9, 2023

I’ll publish a new release this week.

1reaction
freddyrioscommented, Aug 9, 2023

looks good, the run is still going strong well past the point of all the previous failures running the example in that environment.

Read more comments on GitHub >

github_iconTop Results From Across the Web

freddyrios
Day of Week August Aug August Aug Sunday Sun No contributions on Sunday, August 7, 2022 No contributions on Sunday, Au... Monday Mon No contributions...
Read more >
In Search of an Understandable Consensus Algorithm
For example, in Raft we separated leader election, log replication, safety, and membership changes. Our second approach was to simplify the ...
Read more >
You're Age 35, 50, or 60: How Much Should You Have ...
Savings benchmarks based on age and salary can serve as a helpful way to track progress against saving for retirement. Saving 15% of...
Read more >
How to Win at Retirement Savings
Put another way: The investor who started saving 10 years earlier would have about $500,000 more at retirement. It's that simple. 2. Saving...
Read more >
10 Ways to Effectively Save for the Future
Key steps for saving include making a budget (with a live-in partner if you have one), reviewing your expenses, and understanding your household's...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Bug: summary text is generated to HTML as source code markdown

Next page

leader memory leak when a node is down in linux - raspberry or debian