python functions cannot find python3 interpreter

Describe the bug python functions do not start if the system has “python3” but not “python” (which is the case for modern Ubuntu systems - “python” is python2 and is not installed by default)

To Reproduce Write a simple function func1.py

from pulsar import Function

class FirstFunction(Function):
    def process(self, item, context):
        log = context.get_logger()
        log.info("Got %r with properties %r" % (item, context.get_message_properties()))

Deploy it:

apache-pulsar-2.4.1/bin/pulsar-admin functions create --name womble --inputs my-topic \
  --py /home/ubuntu/func1.py --classname FirstFunction --log-topic pulsar-log

(side note: have to give absolute path to function, pulsar-admin changes the working directory)

Check its status:

$ apache-pulsar-2.4.1/bin/pulsar-admin functions status --name womble
{
  "numInstances" : 1,
  "numRunning" : 0,
  "instances" : [ {
    "instanceId" : 0,
    "status" : {
      "running" : false,
      "error" : "Cannot run program \"python\": error=2, No such file or directory",
      "numRestarts" : 16,
      "numReceived" : 0,
      "numSuccessfullyProcessed" : 0,
      "numUserExceptions" : 0,
      "latestUserExceptions" : [ ],
      "numSystemExceptions" : 0,
      "latestSystemExceptions" : [ ],
      "averageLatency" : 0.0,
      "lastInvocationTime" : 0,
      "workerId" : "c-standalone-fw-ldex-pulsar.int.example.net-8080"
    }
  } ]
}

Note error: Cannot run program “python”: error=2, No such file or directory

Expected behavior I would expect the function to deploy.

Screenshots N/A

Desktop (please complete the following information): N/A

Additional context This may be the same problem as #5369 - however in that case, it’s the shebang line in pulsar’s own python code, whereas this one is my own function which is not starting.

I cannot find it stated in Pulsar documentation whether Pulsar supports python2, python3 or both. However python2 is very shortly going to be end-of-life.

Workarounds

Install python2 and write my code in python2. I really don’t want to do that, with python2 being end-of-life in 2 months’ time.

Symlink python to python3. However in Ubuntu systems, symlinking python to python3 has the potential to cause all sorts of problems. Many applications and scripts depend on python meaning python2 (in which case they will depend on the python-XXX packages rather than python3-XXX). More content here.

Ideally pulsar would try python3 and fallback to python. If pulsar were to be hard-coded to run “python3” instead of “python” that would also be fine IMO, given the impending EOL of python2.

[candlerb]

How about adding a setting to broker.conf / standalone.conf? e.g.

python_interpreter=/usr/bin/python

However I notice at least one place where searching in $PATH is currently used:

./pulsar-client-cpp/docker/Dockerfile:ENV PATH="/opt/python/${PYTHON_SPEC}/bin:${PATH}"

So I suggest:

1. Default is python_interpreter=python
2. If the setting starts with slash, it’s an absolute path (avoids some edge-case security issues)
3. Otherwise search for it in $PATH as today

If you don’t want a conf setting, then an environment variable would work for me too (PYTHON=/usr/bin/python3)

[sbourkeostk]

@srkukarni, you got me thinking. How about reusing customRuntimeOptions? Eg: 3a2b312c4ded8cd78ca61e3877add394f01b8fe4. Then setting custom-runtime-options to {"PythonInterpreter": "python3"} or even {"PythonInterpreter": "/opt/python-3.6.5/bin/python3"} would set the interpreter. The reason for using json is to maintain compatibility with KubernetesManifestCustomizer which I think is the only other use of the parameter.

By the way, the protobuf python definitions also need to be recompiled for this to work. Probably should be done in any case. The one in master seems to be old and doesn’t include customRuntimeOptions (added with 320cebe09f80116a2df81a5df1bcfd518ecf8c08) (The python definitions don’t get built by maven - they are been generated manually and tracked by git).