cannot `pip3 install orjson` in Docker currently (alpine linux)

I get this:

Collecting orjson
  Downloading https://files.pythonhosted.org/packages/9c/6e/14a622d6b2615573bbd1c404dc16f60a153474d20e91ab13ab49dde0867c/orjson-3.0.2.tar.gz (649kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  Complete output from command /usr/bin/python3.6 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-vugd6617 --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- maturin>=0.8.1,<0.9:
  Collecting maturin<0.9,>=0.8.1
    Downloading https://files.pythonhosted.org/packages/a7/df/a9060d8e4639feec0e8ab98081e5b9e7bd302db29b666cc79c2c6baf4397/maturin-0.8.1.tar.gz (78kB)
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
  Collecting toml~=0.10.0 (from maturin<0.9,>=0.8.1)
    Using cached https://files.pythonhosted.org/packages/9f/e1/1b40b80f2e1663a6b9f497123c11d7d988c0919abbf3c3f2688e448c5363/toml-0.10.1-py2.py3-none-any.whl
  Installing collected packages: toml, maturin
    Running setup.py install for maturin: started
      Running setup.py install for maturin: finished with status 'error'
      Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-pyhjev5y/maturin/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-89n4jgda/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-vugd6617 --compile:
      running install
      warning: unused manifest key: package.edition
      error: failed to parse lock file at: /tmp/pip-install-pyhjev5y/maturin/Cargo.lock

      Caused by:
        invalid serialized PackageId for key `package.dependencies`
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/tmp/pip-install-pyhjev5y/maturin/setup.py", line 101, in <module>
          zip_safe=False,
        File "/tmp/pip-build-env-5zcgcduk/lib/python3.6/site-packages/setuptools/__init__.py", line 145, in setup
          return distutils.core.setup(**attrs)
        File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
          self.run_command(cmd)
        File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
          cmd_obj.run()
        File "/tmp/pip-install-pyhjev5y/maturin/setup.py", line 59, in run
          ["cargo", "rustc", "--bin", "maturin", "--", "-C", "link-arg=-s"]
        File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cargo', 'rustc', '--bin', 'maturin', '--', '-C', 'link-arg=-s']' returned non-zero exit status 101.

When called from docker-compose build with alpine linux 3.7

Not possible to distribute orjson without the need to compile?

If I try on latest alpine linux:

...
Collecting orjson
  Downloading orjson-3.0.2.tar.gz (649 kB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 /usr/lib/python3.8/site-packages/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpv96ooomy
         cwd: /tmp/pip-install-se40dy4c/orjson
    Complete output (14 lines):
    💥 maturin failed
      Caused by: Cargo metadata failed. Do you have cargo in your PATH?
      Caused by: Error during execution of `cargo metadata`: error: failed to run `rustc` to learn about target-specific information

    Caused by:
      process didn't exit successfully: `rustc - --crate-name ___ --print=file-names -Z mutable-noalias --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit code: 1)
    --- stderr
    error: the option `Z` is only accepted on the nightly compiler



    Checking for Rust toolchain....
    Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-lpe0im7j --manylinux=off --strip=on`
    Error: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/pip-modern-metadata-lpe0im7j', '--manylinux=off', '--strip=on']' returned non-zero exit status 1.
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.8/site-packages/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpv96ooomy Check the logs for full command output.

[bluebrown]

Just want to mention that if you copy the stuff compiled into site-packages and sometimes also binaries from /user/local/bin into a second stage alpine image, the resulting image won’t be bigger since you just dropped all compile dependencies, again. You just need to make sure you install the slim version of the dependencies again in the final image.

So, result can in fact be around 100 MB. Wile to compile stage, image can rack up to 1 Gb+. Yes, the build time will be significantly higher, but we have layer caching. It is OK for a production image, IMO.

FROM python:3.9-alpine as builder

RUN apk add --no-cache gcc g++ musl-dev rust cargo patchelf

RUN python -m pip install orjson


FROM python:3.9-alpine

COPY --from=builder /usr/local/lib/python3.9/site-packages /usr/local/lib/python3.9/site-packages

RUN apk add --no-cache libstdc++

CMD ["python", "-c", "import orjson; print(orjson.dumps({'number': 1}))"]

$ docker build -t test .
$ docker run test
b'{"number":1}'

With this Dockerfile the builder stage is as big as 1.28 GB, but the final image is only 58 MB. The first build took around 10 minutes on my machine. But after caching, when changing lets say only the program to run, it only takes seconds.

Edit: since recently, we seem to need to add patchelf ourselves. See https://github.com/ijl/orjson/issues/227. I have added it to the dockerfile.

[itamarst]

Alpine Linux cannot use wheels from PyPI. This isn’t an orjson-specific issue, any binary wheel will fail to install. You’re better off just not using Alpine for this reason (https://pythonspeed.com/articles/alpine-docker-python/ talks about this in more detail.)