Pulsar branch-2.9 build failure in auditwheel running build-wheel-file-within-docker.sh

Description Docker build fails in pulsar-client-cpp/docker/build-wheel-file-within-docker.sh with error on + auditwheel repair dist/pulsar_client-2.8.2-cp38-cp38-linux_x86_64.whl dist/pulsar_client-2.9.1-cp38-cp38-linux_x86_64.whl

To Reproduce Steps to reproduce the behavior: git clone https://github.com/apache/pulsar.git git checkout branch-2.9 Follow steps here:

mvn clean install -DskipTests
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests

Expected behavior Pulsar successfully builds docker images on latest version with Log4J 2.17+ to be tagged and pushed to a custom or local docker registry.

Trace

[100%] Built target _pulsar
make[2]: Leaving directory `/pulsar/pulsar-client-cpp'
/usr/local/bin/cmake -E cmake_progress_start /pulsar/pulsar-client-cpp/CMakeFiles 0
make[1]: Leaving directory `/pulsar/pulsar-client-cpp'
+ cd python
+ python setup.py bdist_wheel
2.9.1
pulsar-client
running bdist_wheel
running build
running build_py
copying pulsar/schema/definition.py -> build/lib.linux-x86_64-3.8/pulsar/schema
copying pulsar/schema/schema.py -> build/lib.linux-x86_64-3.8/pulsar/schema
running build_ext
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
copying build/lib.linux-x86_64-3.8/_pulsar.cpython-38-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/pulsar
copying build/lib.linux-x86_64-3.8/pulsar/__init__.py -> build/bdist.linux-x86_64/wheel/pulsar
copying build/lib.linux-x86_64-3.8/pulsar/exceptions.py -> build/bdist.linux-x86_64/wheel/pulsar
creating build/bdist.linux-x86_64/wheel/pulsar/functions
copying build/lib.linux-x86_64-3.8/pulsar/functions/__init__.py -> build/bdist.linux-x86_64/wheel/pulsar/functions
copying build/lib.linux-x86_64-3.8/pulsar/functions/context.py -> build/bdist.linux-x86_64/wheel/pulsar/functions
copying build/lib.linux-x86_64-3.8/pulsar/functions/function.py -> build/bdist.linux-x86_64/wheel/pulsar/functions
copying build/lib.linux-x86_64-3.8/pulsar/functions/serde.py -> build/bdist.linux-x86_64/wheel/pulsar/functions
creating build/bdist.linux-x86_64/wheel/pulsar/schema
copying build/lib.linux-x86_64-3.8/pulsar/schema/__init__.py -> build/bdist.linux-x86_64/wheel/pulsar/schema
copying build/lib.linux-x86_64-3.8/pulsar/schema/definition.py -> build/bdist.linux-x86_64/wheel/pulsar/schema
copying build/lib.linux-x86_64-3.8/pulsar/schema/schema.py -> build/bdist.linux-x86_64/wheel/pulsar/schema
copying build/lib.linux-x86_64-3.8/pulsar/schema/schema_avro.py -> build/bdist.linux-x86_64/wheel/pulsar/schema
running install_egg_info
running egg_info
writing pulsar_client.egg-info/PKG-INFO
writing dependency_links to pulsar_client.egg-info/dependency_links.txt
writing requirements to pulsar_client.egg-info/requires.txt
writing top-level names to pulsar_client.egg-info/top_level.txt
reading manifest file 'pulsar_client.egg-info/SOURCES.txt'
writing manifest file 'pulsar_client.egg-info/SOURCES.txt'
Copying pulsar_client.egg-info to build/bdist.linux-x86_64/wheel/pulsar_client-2.9.1-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/pulsar_client-2.9.1.dist-info/WHEEL
creating 'dist/pulsar_client-2.9.1-cp38-cp38-linux_x86_64.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding '_pulsar.cpython-38-x86_64-linux-gnu.so'
adding 'pulsar/__init__.py'
adding 'pulsar/exceptions.py'
adding 'pulsar/functions/__init__.py'
adding 'pulsar/functions/context.py'
adding 'pulsar/functions/function.py'
adding 'pulsar/functions/serde.py'
adding 'pulsar/schema/__init__.py'
adding 'pulsar/schema/definition.py'
adding 'pulsar/schema/schema.py'
adding 'pulsar/schema/schema_avro.py'
adding 'pulsar_client-2.9.1.dist-info/METADATA'
adding 'pulsar_client-2.9.1.dist-info/WHEEL'
adding 'pulsar_client-2.9.1.dist-info/top_level.txt'
adding 'pulsar_client-2.9.1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
+ auditwheel repair dist/pulsar_client-2.8.2-cp38-cp38-linux_x86_64.whl dist/pulsar_client-2.9.1-cp38-cp38-linux_x86_64.whl
usage: auditwheel [-h] [-V] [-v] command ...
auditwheel: error: unrecognized arguments: dist/pulsar_client-2.9.1-cp38-cp38-linux_x86_64.whl
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Pulsar 2.9.1:
[INFO]
[INFO] Pulsar ............................................. SUCCESS [ 24.345 s]
[INFO] Apache Pulsar :: Docker Images ..................... SUCCESS [  0.630 s]
[INFO] Apache Pulsar :: Docker Images :: Pulsar Latest Version FAILURE [27:17 min]
[INFO] Apache Pulsar :: Docker Images :: Pulsar Latest Version (Include All Components) SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  27:43 min
[INFO] Finished at: 2021-12-22T09:50:54-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:exec (build-pulsar-clients-python-35) on project pulsar-docker-image: Command execution failed.: Process exited with an error: 2 (Exit value: 2) -> [Help 1]

Desktop:

• OS: MacOS 12.0 Monterey Darwin <hostname removed>.local 21.0.1 Darwin Kernel Version 21.0.1: Tue Sep 14 20:56:24 PDT 2021; root:xnu-8019.30.61~4/RELEASE_ARM64_T6000 arm64 However, the command is failing inside a container it seems.

Additional context This error occurs on branch 2.9. I’ve also tried to build master and branch-2.8 with no success. I’ve also tried building following the steps to reproduce on a clean install of Linux Mint with openjdk-11 and maven 3.8.4, but that gives a different error about Text file busy: 'build/bdist.linux-x86_64/wheel/pulsar_client-2.9.1-py3.8.egg-info' - so should be a separate issue.

Maven:

➜ mvn --version
Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: /opt/homebrew/Cellar/maven/3.8.4/libexec
Java version: 11, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk-11.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"

Java:

➜  java --version
openjdk 11 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

Python (although it seems to be using a bundled version so this probably doesn’t matter): Python 3.9.7

Side note: Is there any timeline for when the Log4J patched (2.17) images will be released to the Docker Registry? I feel like there are a lot of people struggling with issues like this and I’m surprised the patch has been completed in here but hasn’t been released. There’s still a lot of vulnerable Pulsar instances because of this. It’s blocking development for us as we also can’t seem to build pulsar images across different machines.

[eolivelli]

Can you try to ‘git clean -xdf’ your repo? IIRC that errors occurs if you have multiple builds locally

I have built the official images for 2.9.1 and 2.9.0 and it worked well

[tisonkun]

Closed as stale. The development of the Python client has been permanently moved to http://github.com/apache/pulsar-client-python. Please open an issue there if it’s still relevant.