Data loss when interrupting import

I am importing my music collection for the first time. At a certain moment I had to interrupt the proces. That took a while, so I hit Ctrl-c. Now, if I restart, it resumes, but after a while a song is not found in the new location where beets expects it, and it is also no longer in the old location. Data loss!

Problem

Running this command in verbose (-vv) mode:

$ beet import /media/me/external-usb/music to be sorted

It is still running in verbose mode…

Led to this problem:

This album is already in the library!
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Kryptonite.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Loser.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Duck and Run.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Not Enough.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Be Like That.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Life of My Own.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 The Better Life.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Down Poison.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 By My Side.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Smack.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 So I Need You.flac'
Old: 11 items, FLAC, 1631kbps, 44.1kHz/24 bit, 40:51, 0.0 B
Old: 11 items, FLAC, 1631kbps, 44.1kHz/24 bit, 40:51, 476.3 MiB
New: 1 items, FLAC, 1644kbps, 44.1kHz/24 bit, 3:07, 36.8 MiB
[S]kip new, Keep all, Remove old, Merge all? m
Traceback (most recent call last):
  File "/home/hendrik2/.local/bin/beet", line 8, in <module>
    sys.exit(main())
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/ui/__init__.py", line 1285, in main
    _raw_main(args)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/ui/__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/ui/commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/ui/commands.py", line 943, in import_files
    session.run()
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/util/pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/util/pipeline.py", line 314, in run
    for msg in _allmsgs(out):
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/util/pipeline.py", line 467, in pull
    out = coro.send(msg)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/util/pipeline.py", line 193, in coro
    func(*(args + (task,)))
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/importer.py", line 1369, in lookup_candidates
    plugins.send('import_task_start', session=session, task=task)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/plugins.py", line 488, in send
    result = handler(**arguments)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/plugins.py", line 145, in wrapper
    return func(*args, **kwargs)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beetsplug/chroma.py", line 180, in fingerprint_task
    return fingerprint_task(self._log, task, session)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beetsplug/chroma.py", line 249, in fingerprint_task
    acoustid_match(log, item.path)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beetsplug/chroma.py", line 89, in acoustid_match
    duration, fp = acoustid.fingerprint_file(util.syspath(path))
  File "/home/hendrik2/.local/lib/python3.8/site-packages/acoustid.py", line 340, in fingerprint_file
    return _fingerprint_file_audioread(path, maxlength)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/acoustid.py", line 280, in _fingerprint_file_audioread
    with audioread.audio_open(path) as f:
  File "/home/hendrik2/.local/lib/python3.8/site-packages/audioread/__init__.py", line 111, in audio_open
    return BackendClass(path)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/audioread/rawread.py", line 62, in __init__
    self._fh = open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Kryptonite.flac'

and then command line.

Here’s a link to the music files that trigger the bug (if relevant): it is on a local external usb drive

Setup

• OS: Linux Mint Mate edition 20.3 64 bit
• Python version: $ python2 --version Python 2.7.18
• beets version: $ beet --version beets version 1.6.0 Python version 3.8.10 plugins: chroma, fetchart, fromfilename, lastgenre (different python versions, that’s interesting)
• Turning off plugins made problem go away (yes/no):

My configuration (output of beet config) is:

$ beet config
directory: /media/hendrik2/Scharnier/Muziek
library: ~/.beets/musiclibrary.db

import:
    move: yes
log: beetslog.txt
art_filename: "$albumartist \u2013 $album"

plugins: fetchart lastgenre chroma fromfilename

paths:
    default: "$albumartist/$album%aunique{}/$artist \u2013 $title"
    singleton: "$artist/$artist \u2013 $title"
    comp: "Compilations/$album%aunique{}/$artist \u2013 $title"
acoustid:
    apikey: REDACTED
fetchart:
    auto: yes
    minwidth: 0
    maxwidth: 0
    quality: 0
    max_filesize: 0
    enforce_ratio: no
    cautious: no
    cover_names:
    - cover
    - front
    - art
    - album
    - folder
    sources:
    - filesystem
    - coverart
    - itunes
    - amazon
    - albumart
    google_key: REDACTED
    google_engine: 001442825323518660753:hrh5ch1gjzm
    fanarttv_key: REDACTED
    lastfm_key: REDACTED
    store_source: no
    high_resolution: no
    deinterlace: no
    cover_format:
lastgenre:
    whitelist: yes
    min_weight: 10
    count: 1
    fallback:
    canonical: no
    source: album
    force: yes
    auto: yes
    separator: ', '
    prefer_specific: no
    title_case: yes
chroma:
    auto: yes

I tried removing .beets/musiclibrary.db, but the error seems to persist.

[wisp3rwind]

We might still consider making this a safer experience: Merge probably isn’t the most appropriate action to choose in this case, but maybe it shouldn’t crash the importer. Things we could do

• If all files of one of the duplicate albums are missing, don’t even offer to merge,
• If only some of the files are missing, inform the user and remove the corresponding items from the library and the import task. The current problem is that duplicate merging involves a re-import (see the user_query import stage), which would then involve an import task with partially missing files. Our importer is probably in general not expecting this. This manifests here for the chroma plugin, since it runs very early. However, I strongly suspect that other plugins and even core beets would lead to a crash due to missing files, even if chroma is disabled.

Note that in any case, this is about using beets out of spec: Beets for the most part assumes that it’s in exclusive control of the library (including associated files). So we might decide that none of the above is really worth the effort (I won’t be the one to implement this anyway…).

[wisp3rwind]

Closing in favor of #4342, which is probably the same.