Mitigate Spyder wiping users' files with more robust atomic saves/autosaves
See original GitHub issueProblem Description
What steps reproduce the problem?
- Save file to disk with connection issues (confirmed)
- Crashes during runfile or during saving immediatly prior (reported 2x)
- Other problems during saving???
Given we’ve publicly advertised the autosave feature as the solution to the reports we’ve been getting of Spyder completely wiping users’ files on occasion e.g. @ccordoba12 in #6571:
I don’t understand how Spyder can delete your script, but we’re working to provide autosaving functionality for our next major release (Spyder 4).
and particularly now that our own developer @jnsebgosselin is regularly experiencing a such a problem and can reproduce it, we should ensure autosave is reasonably “fail-safer” in the event that saving the file normally fails (silently or otherwise) and wipes or corrupts the file on disk. As currently implemented by @jitseniesen following his hard work in #7660 , the feature is reasonably sophisticated with a nice UI. However, as an initial implementation, it does leave open several significant opportunities for catastrophic data loss in several major scenarios:
Failure mode: Saving may fail silently, or in flushing the data to the storage medium, which would result in the autosave still being deleted and the save being corrupted or wiped (or at least out of date). This would possibly apply to the repeated instances of files being wiped that @jnsebgosselin is experiencing (depending on exactly how the failure occurs).
Mitigations:
- Hardening the e.g. FileInfo._write_to_file() function, e.g. with
<file>.flush()
thenos.fsync()
; switching to the more modernio.open()
on Python 2 and making sure files are explicitly closed might help a little too - Writing to a tempfile when saving, then doing an atomic replace of the original by the appropriate platform-specific method
If there are significant cases where the OS cannot be trusted to throw an error if something goes wring, we might need a readback check for the tempfile vs. the in memory version (at minimum, that the file reads without error and is the correct length, and perhaps comparing the first line and last lines), though this could be user-optional if its not determined to be critical or it takes a non-trivial amount of timeTemporarily disable autosave and warn the user if a save failure is detected, and notify the user of this; only re-enable once a manual save is successful
I’ve included the full form of the discussion between various parties on this issue below, for reference in one place.
Issue Analytics
- State:
- Created 5 years ago
- Comments:19 (19 by maintainers)
Top GitHub Comments
Instead of rolling something on our own to make saves more reliable, we could just use the atomicwrites package. Its already on
conda defaults
and its whatpytest
uses.Thanks!