Support for network drives or filesystems that don't implement FSYNC

Describe the bug

The issue encountered occurs when attempting to initialize an archivebox directory on a mounted network drive. The issue is reproducible in multiple directory locations located on the NAS. Initializing an archivebox directory on the local drive of the machine succeed. Additionally, if a successfully initialized directory is copied to the NAS, future archivebox operations on that copied directory fail. The write of the Config file appears to fail based on the traceback.

I investigated the trace output and added print statements to the atomicwrites function in which the failure occurs at line 46 of atomicwrites init.py Additionally, I uncommented the print statement at line 41 of archivebox system.py for further debug info.

Steps to reproduce

mkdir archivebox; cd archivebox; archivebox init

Screenshots or log output

OUTPUT:

$ archivebox init
[i] [2020-08-20 19:55:28] ArchiveBox v0.4.21: archivebox init
     /Volumes/Public/data/archivebox

[+] Initializing a new ArchiveBox collection in this folder...
    /Volumes/Public/data/archivebox
------------------------------------------------------------------

[+] Building archive folder structure...
    √ /Volumes/Public/data/archivebox/sources
    √ /Volumes/Public/data/archivebox/archive
    √ /Volumes/Public/data/archivebox/logs

 Atomic Write: w /Volumes/Public/data/archivebox/ArchiveBox.conf 437 overwrite=True
fd is: 3
fcntl.F_FULLFSYNC is 51
fd is: 3
fcntl.F_FULLFSYNC is 51
Traceback (most recent call last):
  File "/Users/user123/.pyenv/versions/3.8.5/bin/archivebox", line 8, in <module>
    sys.exit(main())
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/cli/__init__.py", line 122, in main
    run_subcommand(
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/cli/__init__.py", line 62, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/cli/archivebox_init.py", line 33, in main
    init(
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/util.py", line 111, in typechecked_function
    return func(*args, **kwargs)
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/main.py", line 294, in init
    write_config_file({}, out_dir=out_dir)
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/config/__init__.py", line 376, in write_config_file
    atomic_write(config_path, CONFIG_HEADER)
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/util.py", line 111, in typechecked_function
    return func(*args, **kwargs)
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/archivebox/system.py", line 46, in atomic_write
    f.write(contents)
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 171, in _open
    self.commit(f)
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 204, in commit
    replace_atomic(f.name, self._path)
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 101, in replace_atomic
    return _replace_atomic(src, dst)
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 58, in _replace_atomic
    _sync_directory(os.path.normpath(os.path.dirname(dst)))
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 52, in _sync_directory
    _proper_fsync(fd)
  File "/Users/user123/.pyenv/versions/3.8.5/lib/python3.8/site-packages/atomicwrites/__init__.py", line 46, in _proper_fsync
    fcntl.fcntl(fd, fcntl.F_FULLFSYNC)
FileNotFoundError: [Errno 2] No such file or directory

Here is the output of mount for the two disks: SUCCESS DRIVE: /dev/disk1s5 on /System/Volumes/Data (apfs, local, journaled, nobrowse) FAIL DRIVE: //;AUTH=No%20User%20Authent@Drobo-5N2-RYE._afpovertcp._tcp.local/Public on /Volumes/Public (afpfs, nodev, nosuid, mounted by user123)

Here are the file permissions of the FAIL directory drwxrwxrwx 1 user123 staff 264 Aug 20 15:55 archivebox

Software versions

• OS: MacOS 10.15.6 (19G73)
• ArchiveBox version: ArchiveBox v0.4.21
• Python version: Python 3.8.5

[lylebrown]

That was the hint I needed, thanks! I may not be doing things in the ideal way with my smb share, but I’m using file_mode=0777,dir_mode=0777 as options in my fstab, and I needed to add the noperm option for it to be able to respond to chmod calls with success (even if they aren’t actually making changes). I’m open to suggestions to better manage permissions, but I don’t think I have many options over smb.

[blackberryoctopus]

@pirate Thanks for your help with the investigation.

I have one suggestion/question: Is it possible to improve the initialization routine to output more descriptive debug info when a non-compatible drive destination is attempted for use?