dsub 0.4.3 crashes on ubuntu 20.04

When I run the hello world example

dsub \
   --provider local \
   --logging "${TMPDIR:-/tmp}/dsub-test/logging/" \
   --output OUT="${TMPDIR:-/tmp}/dsub-test/output/out.txt" \
   --command 'echo "Hello World" > "${OUT}"' \
   --wait

it crashes as follows:

***WARNING: No Docker image specified. The default, `ubuntu:14.04` will be used.
***WARNING: For reproducible pipelines, specify an image with the `--image` flag.
Job properties:
  job-id: echo--hylke--201220-165144-45
  job-name: echo
  user-id: hylke
Launched job-id: echo--hylke--201220-165144-45
To check the status, run:
  dstat --provider local --jobs 'echo--hylke--201220-165144-45' --users 'hylke' --status '*'
To cancel the job, run:
  ddel --provider local --jobs 'echo--hylke--201220-165144-45' --users 'hylke'
Waiting for job to complete...
Waiting for: echo--hylke--201220-165144-45.
Traceback (most recent call last):
  File "/home/hylke/.local/bin/dsub", line 8, in <module>
    sys.exit(main())
  File "/home/hylke/.local/lib/python3.8/site-packages/dsub/commands/dsub.py", line 1106, in main
    dsub_main(prog, argv)
  File "/home/hylke/.local/lib/python3.8/site-packages/dsub/commands/dsub.py", line 1091, in dsub_main
    launched_job = run_main(args)
  File "/home/hylke/.local/lib/python3.8/site-packages/dsub/commands/dsub.py", line 1168, in run_main
    return run(
  File "/home/hylke/.local/lib/python3.8/site-packages/dsub/commands/dsub.py", line 1322, in run
    error_messages = _wait_after(provider, [job_metadata['job-id']],
  File "/home/hylke/.local/lib/python3.8/site-packages/dsub/commands/dsub.py", line 786, in _wait_after
    jobs_left = _wait_for_any_job(provider, job_ids_to_check, poll_interval,
  File "/home/hylke/.local/lib/python3.8/site-packages/dsub/commands/dsub.py", line 1015, in _wait_for_any_job
    tasks = provider.lookup_job_tasks({'*'}, job_ids=job_ids)
  File "/home/hylke/.local/lib/python3.8/site-packages/dsub/providers/local.py", line 519, in lookup_job_tasks
    task = self._get_task_from_task_dir(j, u, task_id, task_attempt)
  File "/home/hylke/.local/lib/python3.8/site-packages/dsub/providers/local.py", line 669, in _get_task_from_task_dir
    end_time = self._get_end_time_from_task_dir(task_dir)
  File "/home/hylke/.local/lib/python3.8/site-packages/dsub/providers/local.py", line 583, in _get_end_time_from_task_dir
    datetime.datetime.strptime(f.readline().strip(),
  File "/usr/lib/python3.8/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/lib/python3.8/_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '' does not match format '%Y-%m-%d %H:%M:%S.%f

Any ideas how to get the Hello World example to work?

Versions: dsub: 0.4.3 Ubuntu: 20.04 Python: 3.8.5 Docker: 19.03.11

Thanks,

Hylke

[wnojopra]

I’m happy to see you were able to get it working. I am not familiar with snap, but it looks like the way it containerizes docker prevents it from accessing /tmp. I can’t think of a workaround with dsub other than to use the apt package of docker as you have done.

On our end we’ll update the documentation to make this clear, and provider better error handling in the local provider.

[hylkedonker]

It seems that the problem is indeed related to Docker, which does not like the /tmp folder. For example, this does not work

$ docker run \
   --env-file=/tmp/docker.env \
   "ubuntu:14.04"
docker: open /tmp/docker.env: no such file or directory.
See 'docker run --help'.

But if I copy /tmp/docker.env to my $HOME folder, it works okay:

$ docker run \
   --env-file=$HOME/docker.env \
   "ubuntu:14.04"
$

Solution: I removed the snap Docker package

sudo snap remove docker

and installed the apt Docker package instead (after adding the Docker apt repository)

sudo apt-get install docker-ce docker-ce-cli containerd.io

Now it seems to be working as expected (I am now running Docker version 20.10.1, build 831ebea):

$ dsub \                
   --provider local \
   --logging "${TMPDIR:-/tmp}/dsub-test/logging/" \
   --output OUT="${TMPDIR:-/tmp}/dsub-test/output/out.txt" \
   --command 'echo "Hello World" > "${OUT}"' \
   --wait
***WARNING: No Docker image specified. The default, `ubuntu:14.04` will be used.
***WARNING: For reproducible pipelines, specify an image with the `--image` flag.
Job properties:
  job-id: echo--hylke--201230-204230-26
  job-name: echo
  user-id: hylke
Launched job-id: echo--hylke--201230-204230-26
To check the status, run:
  dstat --provider local --jobs 'echo--hylke--201230-204230-26' --users 'hylke' --status '*'
To cancel the job, run:
  ddel --provider local --jobs 'echo--hylke--201230-204230-26' --users 'hylke'
Waiting for job to complete...
Waiting for: echo--hylke--201230-204230-26.
  echo--hylke--201230-204230-26: SUCCESS
echo--hylke--201230-204230-26