Bidirectional sync robustness

Up to version 2.1, rclonesync has used a sync process that makes Path1 the perfect copy, and then uses rclone sync to replicate Path1 to Path2. The changes on Path2 are identified and then individually copied to Path1 (or deleted on Path1 if the file was deleted on Path2). This process flow works great when the changes are infrequent. The issue is when there are changes on the filesystems during the rclonesync run. In certain circumstances the new changes will be lost - see orange labels in pictures and the tables.

Changes are proposed for rclonesync’s process flow to lessen the chance of data loss. Please provide your thoughts and feedback on this proposal.

V2.1 and prior sync process flow

Sync robustness and risk of data loss in the V2.1 process flow

Changed at	Handling	Status
A only - Normal case	File copied from Path2 to Path1.	Good
A and B - Changed again after initially identified as changed	B version copied to Path1 and captured correctly in the Final LSLs.	Good
B only or F only - Not identified at A	Change will be lost in the final rclone sync Path1 to Path2.  Path1 version survives.	Data loss.
C only	Change will be copied from Path1 to Path2 by the final rclone sync.	Good.
A and (D or E) - Changed during file copy or sync operations	File access conflict.  rclone will retry.  Change at destination will be lost.  Source version survives.	Possible Critical Error Abort
A and F - Changed after copy Path2 to Path1	F change will be lost in the final rclone sync Path1 to Path2.  A version survives.	Data loss
G only	Change will be copied to Path2 in the next rclonesync run.	Good.  Out of sync until next run.
A and H	H version captured in Path2 Final LSL and not identified as a delta on the next run.  The next run will push the Path1 version to Path2, losing the H version.	Out of sync until next run.  Data loss on next run.

V2.2 proposed new sync process flow

The picture shows handling of changes identified at time A on Path2. With the V2.2 process flow the Path1 handling would be the same as the Path2 handling, so just switch Path1 and Path2 in the drawing. This sync process might be faster as it eliminates the rlcone sync operation, which I suspect contains its own rclone lsl operations.

Sync robustness and risk of data loss in the V2.2 process flow

Changed at	Handling	status
A only - Normal case	File copied from Path2 to Path1.	Good
B only or C only - Changed after initial LSLs	Change is missed.  File not copied.	Good. Out of sync until next run.
A and C - Changed on Path1 after identified as changed on Path2	Path2 version survives.  Path1 version is lost.	Data loss
A and (D or E) - Changed or deleted during actual copy	File access conflict.  rclone will retry.  Likely E version is lost and D version survives.	Possible Critical Error Abort
A and (F or G) - Changed after copy.	Identified as different in the two Final LSLs.  Earlier version saved in both Tracking LSLs.	Will be caught in next sync run
A and B -	B version copied but rclonesync is unaware that it copied B version.  B version identified in Final LSL and updated in Path1 and Path2 Tracking LSLs.	Good

[Jwink3101]

Just a followup to my post, I added rclone to my tool if you want to compare methodologies. I am not saying mine is better. It is just an alternative approach that may be of interest to you. It was also not written originally with rclone in mind so that brings its own limitations.

Best of luck on yours!

[cjnaz]

Please continue the issue/discussion in #59