[Question] Environment of Python Task is different from that of real worker container

My DS system is setup by Docker. When I ran like this in a PythonTask, I found that env is different from what i got in the docker container of worker. This is what env showed by the Python Task. And this is what env showed by execute os.system("env") or os.environ directly in the container of worker.

I don’t know why they are totally different, especially the Variable ‘PATH’. I thought that docker container is exactly where python executes script. So i am confused

Which version of DolphinScheduler: -[1.3.6-release]

[chengshiwen]

@wangdazhong001 Under different user, the result of env is different. Regardless of whether it is in the docker container, the same thing happens in linux on the host machine. The richer environment variables directly in the container of worker occur under roor user. While the environment variables from the python task occur under the user whose id is 2.

Since the python script does not execute source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh, the python task cannot read the environment variables in this file.

If you want to access the environment variables from /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh, there are four solutions:

1. open('/opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh', 'r') and parse it
2. Use python-dotenv
3. Use the following code:

import os
import pprint
import shlex
import subprocess

command = shlex.split("env -i bash -c 'source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh && env'")
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
for line in proc.stdout:
  (key, _, value) = line.partition("=")
  os.environ[key] = value
proc.communicate()

pprint.pprint(dict(os.environ))

1. Use shell task instead of python task

source /opt/dolphinscheduler/conf/env/dolphinscheduler_env.sh
python /path/to/your_python_script.py

[chengshiwen]

Close since it has been replied