Re-entrancy isn't achieved because of db-backend

Description

I have a little app built on doit, here the task generator: https://github.com/tenuki/amd2pdf/blob/main/amd2pdf/core.py#L70 . That works ok.

After that, I was interested in use that tool from another script in a make-like fashion using doit, this way:

import sys

srcs = [x[:-4] for x in sys.argv if x.lower().endswith('.pdf')]

def task_make():
    for src in srcs:
        yield {
            'name': 'making-'+src,
            'file_dep': [x+'.md' for x in srcs],
            'targets': ['%s.pdf' % src],
            'actions': ['python -m amd2pdf %s.md' % src],
        }

The output then is:

> doit -f make.py xxx.pdf
.  make:making-xxx
Traceback (most recent call last):
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\doit_cmd.py", line 190, in run
    return command.parse_execute(args)
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\cmd_base.py", line 150, in parse_execute
    return self.execute(params, args)
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\cmd_base.py", line 601, in execute
    return self._execute(**exec_params)
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\cmd_run.py", line 264, in _execute
    return runner.run_all(self.control.task_dispatcher())
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\runner.py", line 256, in run_all
    self.run_tasks(task_dispatcher)
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\runner.py", line 219, in run_tasks
    if not self.select_task(node, task_dispatcher.tasks):
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\runner.py", line 119, in select_task
    if node.ignored_deps or self.dep_manager.status_is_ignore(task):
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\dependency.py", line 587, in status_is_ignore
    return self._get(task.name, "ignore:")
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\dependency.py", line 214, in get
    self._db[task_id] = self.codec.decode(task_data.decode('utf-8'))
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\dependency.py", line 60, in decode
    return self.decoder.decode(data)
  File "C:\Program Files\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\Python38\lib\json\decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 279 (char 278)
TaskFailed - taskid:make:making-xxx
Command failed: 'python -m amd2pdf  xxx.md' returned 3

Also, if I try to execute the automated command python -m amd2pdf xxx.md, by hand, we notice we start to get kind of the same error (and in this case the “make.py” is not involved:

λ python -m amd2pdf xxx.md
[..]
Traceback (most recent call last):
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\doit_cmd.py", line 190, in run
    return command.parse_execute(args)
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\cmd_base.py", line 150, in parse_execute
    return self.execute(params, args)
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\cmd_base.py", line 601, in execute
    return self._execute(**exec_params)
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\cmd_run.py", line 264, in _execute
    return runner.run_all(self.control.task_dispatcher())
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\runner.py", line 256, in run_all
    self.run_tasks(task_dispatcher)
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\runner.py", line 219, in run_tasks
    if not self.select_task(node, task_dispatcher.tasks):
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\runner.py", line 119, in select_task
    if node.ignored_deps or self.dep_manager.status_is_ignore(task):
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\dependency.py", line 587, in status_is_ignore
    return self._get(task.name, "ignore:")
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\dependency.py", line 214, in get
    self._db[task_id] = self.codec.decode(task_data.decode('utf-8'))
  File "C:\Users\aweil\AppData\Roaming\Python\Python38\site-packages\doit\dependency.py", line 60, in decode
    return self.decoder.decode(data)
  File "C:\Program Files\Python38\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\Python38\lib\json\decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 279 (char 278)

Then, dropping the doit/dependency database and making “make.py” use a different backend with:

DOIT_CONFIG = {
    'backend': 'json',
    'dep_file': 'doit-db.json',
}

seems to work without any problems.

In my case, I can setup a different backend and/or db location on each execution, however you maybe interested in adding support for this kind of use case.

Environment

1. OS: windows 10
2. python version: Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
3. doit version: 0.32.0

[schettino72]

I have not looked at amd2pdf. But since it is a tool I would recommend it to NOT use the default name for its DB file (maybe .amd2pdf.doit.db instead of .doit.db) .

[schettino72]

Thanks for report. Yes, as you figured out this is a limitation of the backend.

I am re-opening because I guess there are 2 improvements to be made:

• clear docs saying re-entrancy/concurrency is not supported on all backends (works GDBM?). make sure the word “re-entrancy” is mentioned somewhere. Maybe add this to FAQ…
• Exception message could be more clear, saying that DB is corrupted (maybe because of concurrent access)