FileNotFoundError occurs in python operator in docker

HI. I am running the digdag server in the docker container with the following version.

# docker --version
Docker version 1.12.6, build 78d1802

# java -version
openjdk version "1.8.0_141"
OpenJDK Runtime Environment (build 1.8.0_141-8u141-b15-1~deb9u1-b15)
OpenJDK 64-Bit Server VM (build 25.141-b15, mixed mode)

# digdag --version
0.9.16

The digdag server mounts docker.sock and makes sibling containers possible.

Attempting to execute the python operator in the sibling container with the following dig file will cause FileNotFoundError.

python-with-docker.dig

timezone: UTC

_export:
  docker:
    image: python:3
    
+print_python_version:
  sh>: python --version

+hello:
  py>: scripts.myclass.MyClass.hello

myclass.py

class MyClass(object):
    def __init__(self):
        pass

    def hello(self):
        print("hello world")

log

2017-09-22 09:28:28.673 +0000 [INFO] (101622@[0:python]+python-with-docker+print_python_version) io.digdag.core.agent.OperatorManager: sh>: python --version
Python 3.6.2
2017-09-22 09:28:31.106 +0000 [INFO] (101622@[0:python]+python-with-docker+hello) io.digdag.core.agent.OperatorManager: py>: scripts.myclass.MyClass.hello
Traceback (most recent call last):
  File "<stdin>", line 12, in <module>
FileNotFoundError: [Errno 2] No such file or directory: '.digdag/tmp/digdag-py-in-6591803356036202388.tmp'
2017-09-22 09:28:32.193 +0000 [ERROR] (101622@[0:python]+python-with-docker+hello) io.digdag.core.agent.OperatorManager: Task failed with unexpected error: Python command failed with code 1
java.lang.RuntimeException: Python command failed with code 1
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runCode(PyOperatorFactory.java:153)
	at io.digdag.standards.operator.PyOperatorFactory$PyOperator.runTask(PyOperatorFactory.java:91)
	at io.digdag.util.BaseOperator.run(BaseOperator.java:35)
	at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:312)
	at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:254)
	at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:137)
	at io.digdag.core.agent.ExtractArchiveWorkspaceManager.withExtractedArchive(ExtractArchiveWorkspaceManager.java:36)
	at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:135)
	at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:119)
	at io.digdag.core.agent.MultiThreadAgent.lambda$null$0(MultiThreadAgent.java:127)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2017-09-22 09:28:33.833 +0000 [INFO] (101622@[0:python]+python-with-docker^failure-alert) io.digdag.core.agent.OperatorManager: type: notify

Since the shell operator is working well, I guess it’s a problem in running the python operator inside docker.

[yoyama]

I think this is docker in docker problem and not issue on Digdag itself. If we have document on Digdag server on docker, this information may be useful.

[kanga333]

Thank you for your advice!! By mounting /tmp it was possible to avoid errors.

As for me, this workaround has no problem. However, since it is not a fundamental solution, I thought that the issue should remain open. What do you think?