onnx requirement lists protobuf >=3.12.2,<=3.20.1 but does not work on 3.19 and below

Bug Report

Is the issue related to model conversion?

No.

Describe the bug

When onnx is installed with protobuf 3.19 or less it fails on load on import despite that being valid given onnx requirements.

This is not a problem with protobuf 3.20.0.

System information

• OS Platform and Distribution (e.g. Linux Ubuntu 16.04): MacOS 12.5.1 (arm64)
• ONNX version (e.g. 1.7): 1.12.0
• Python version: 3.10.6
• Protobuf version: 3.19.4

Reproduction instructions

$ pipenv install protobuf==3.19.4 onnx
...
$ pipenv graph
onnx==1.12.0
  - numpy [required: >=1.16.6, installed: 1.23.2]
  - protobuf [required: >=3.12.2,<=3.20.1, installed: 3.19.4]
  - typing-extensions [required: >=3.6.2.1, installed: 4.3.0]
$ pipenv run python -c "import onnx"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/damon/Downloads/test-onnx/.venv/lib/python3.10/site-packages/onnx/__init__.py", line 6, in <module>
    from onnx.external_data_helper import load_external_data_for_model, write_external_data_tensors, convert_model_to_external_data
  File "/Users/damon/Downloads/test-onnx/.venv/lib/python3.10/site-packages/onnx/external_data_helper.py", line 9, in <module>
    from .onnx_pb import TensorProto, ModelProto, AttributeProto, GraphProto
  File "/Users/damon/Downloads/test-onnx/.venv/lib/python3.10/site-packages/onnx/onnx_pb.py", line 4, in <module>
    from .onnx_ml_pb2 import *  # noqa
  File "/Users/damon/Downloads/test-onnx/.venv/lib/python3.10/site-packages/onnx/onnx_ml_pb2.py", line 5, in <module>
    from google.protobuf.internal import builder as _builder
ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/Users/damon/Downloads/test-onnx/.venv/lib/python3.10/site-packages/google/protobuf/internal/__init__.py)

Expected behavior

If onnx is incompatible with protobuf versions that should be accurately reflected in requirements.

[damonmaria]

@jcwchen Ahhhh… using -vv when installing with pip I can see it’s not using a wheel and so builds it from source. This explains it.

Looking through the install I can see there are no _arm64 or _universal2 wheels. I had been following an onnx (or onnxruntime) issue about M1 support and saw it had been merged in before v1.12. I had assumed this would mean there would be wheels in v1.12 but now I see from #3129 that that is not the case.

I will close this as #3129 will solve it.

Thanks for your time.

[Digital2Slave]

I can’t install onnx from python3 -m pip install numpy onnx

• OS: Ubuntu 18.04.6 LTS
• Python: 3.6.9
• protobuf: 3.19.4

error message:

ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-c6x8es8r/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/tianzx/.local/include/python3.6m/onnx
         cwd: /tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/
    Complete output (117 lines):
    fatal: not a git repository (or any of the parent directories): .git
    running install
    running build
    running build_py
    running create_version
    running cmake_build
    Using cmake args: ['/usr/bin/cmake', '-DPYTHON_INCLUDE_DIR=/usr/include/python3.6m', '-DPYTHON_EXECUTABLE=/usr/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-36m-x86_64-linux-gnu.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2']
    Generated: /tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/.setuptools-cmake-build/onnx/onnx-ml.proto
    Generated: /tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/.setuptools-cmake-build/onnx/onnx-operators-ml.proto
    Generated: /tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/.setuptools-cmake-build/onnx/onnx-data.proto
    -- Found pybind11: /usr/local/include (found version "2.6.2" )
    --
    -- ******** Summary ********
    --   CMake version             : 3.10.2
    --   CMake command             : /usr/bin/cmake
    --   System                    : Linux
    --   C++ compiler              : /usr/bin/c++
    --   C++ compiler version      : 7.5.0
    --   CXX flags                 :  -Wnon-virtual-dtor
    --   Build type                : Release
    --   Compile definitions       : __STDC_FORMAT_MACROS
    --   CMAKE_PREFIX_PATH         :
    --   CMAKE_INSTALL_PREFIX      : /usr/local
    --   CMAKE_MODULE_PATH         :
    --
    --   ONNX version              : 1.12.0
    --   ONNX NAMESPACE            : onnx
    --   ONNX_USE_LITE_PROTO       : OFF
    --   USE_PROTOBUF_SHARED_LIBS  : OFF
    --   Protobuf_USE_STATIC_LIBS  : ON
    --   ONNX_DISABLE_EXCEPTIONS   : OFF
    --   ONNX_WERROR               : OFF
    --   ONNX_BUILD_TESTS          : OFF
    --   ONNX_BUILD_BENCHMARKS     : OFF
    --   ONNXIFI_DUMMY_BACKEND     : OFF
    --   ONNXIFI_ENABLE_EXT        : OFF
    --
    --   Protobuf compiler         : /usr/bin/protoc
    --   Protobuf includes         : /usr/include
    --   Protobuf libraries        : /usr/lib/x86_64-linux-gnu/libprotobuf.a;-lpthread
    --   BUILD_ONNX_PYTHON         : ON
    --     Python version        :
    --     Python executable     : /usr/bin/python3
    --     Python includes       : /usr/include/python3.6m
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/.setuptools-cmake-build
    [  8%] Built target onnxifi_loader
    [  8%] Built target onnxifi_dummy
    [  8%] Built target gen_onnx_proto
    [ 14%] Built target gen_onnx_data_proto
    [ 14%] Built target gen_onnx_operators_proto
    [ 19%] Built target onnxifi_wrapper
    [ 30%] Built target onnx_proto
    [ 33%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/traditionalml/old.cc.o
    [ 33%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/sequence/defs.cc.o
    [ 35%] Building CXX object CMakeFiles/onnx.dir/onnx/onnxifi_utils.cc.o
    [ 36%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/tensor_util.cc.o
    [ 38%] Building CXX object CMakeFiles/onnx.dir/onnx/version_converter/convert.cc.o
    [ 39%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/traditionalml/defs.cc.o
    [ 40%] Building CXX object CMakeFiles/onnx.dir/onnx/defs/training/defs.cc.o
    [ 42%] Building CXX object CMakeFiles/onnx.dir/onnx/version_converter/helper.cc.o
    [ 43%] Building CXX object CMakeFiles/onnx.dir/onnx/shape_inference/implementation.cc.o
    /tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/onnx/defs/sequence/defs.cc: In function ‘bool onnx::BuildSequenceMapBodyFunc(const onnx::FunctionBodyBuildContext&, const onnx::OpSchema&, onnx::FunctionProto&)’:
    /tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/onnx/defs/sequence/defs.cc:675:40: error: no match for ‘operator[]’ (operand types are ‘google::protobuf::RepeatedPtrField<onnx::ValueInfoProto>’ and ‘int’)
             seq_at_node.add_output(g_inputs[inputIndex].name());
                                            ^
    /tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/onnx/defs/sequence/defs.cc:683:37: error: no match for ‘operator[]’ (operand types are ‘google::protobuf::RepeatedPtrField<onnx::ValueInfoProto>’ and ‘int’)
             identity.add_output(g_inputs[inputIndex].name());
                                         ^
    /tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/onnx/defs/sequence/defs.cc:741:30: error: no match for ‘operator[]’ (operand types are ‘google::protobuf::RepeatedPtrField<onnx::ValueInfoProto>’ and ‘int’)
         int64_t dtype = g_outputs[outputIndex].type().tensor_type().elem_type();
                                  ^
    CMakeFiles/onnx.dir/build.make:830: recipe for target 'CMakeFiles/onnx.dir/onnx/defs/sequence/defs.cc.o' failed
    make[2]: *** [CMakeFiles/onnx.dir/onnx/defs/sequence/defs.cc.o] Error 1
    make[2]: *** Waiting for unfinished jobs....
    CMakeFiles/Makefile2:313: recipe for target 'CMakeFiles/onnx.dir/all' failed
    make[1]: *** [CMakeFiles/onnx.dir/all] Error 2
    Makefile:129: recipe for target 'all' failed
    make: *** [all] Error 2
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/setup.py", line 354, in <module>
        "backend-test-tools = onnx.backend.test.cmd_tools:main",
      File "/usr/local/lib/python3.6/dist-packages/setuptools/__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/local/lib/python3.6/dist-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.6/distutils/command/install.py", line 589, in run
        self.run_command('build')
      File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/setup.py", line 223, in run
        self.run_command("cmake_build")
      File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/setup.py", line 217, in run
        subprocess.check_call(build_args)
      File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/usr/bin/cmake', '--build', '.', '--', '-j', '12']' returned non-zero exit status 2.
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-gcgyk_b5/onnx_06f0dabac4754e0d844b7a8934aacda2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-c6x8es8r/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/tianzx/.local/include/python3.6m/onnx Check the logs for full command output.