Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Cannot install Transformers version >2.3.0 with pip on CentOS

See original GitHub issue

🐛 Bug

I cannot install pip install transformers for a release newer than 2.3.0. The install errors out when trying to install tokenizers. This is similar to another issue, except I have a Rust Compiler in my environment so I do not see: "error: can not find Rust Compiler".

Information

Model I am using (Bert, XLNet …):

N/A

Language I am using the model on (English, Chinese …):

N/A

The problem arises when using:

  • the official example scripts: (give details below)
  • my own modified scripts: (give details below)

The tasks I am working on is:

  • an official GLUE/SQUaD task: (give the name)
  • my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

pip install transformers

which leads to the following error:

Building wheels for collected packages: tokenizers
  Building wheel for tokenizers (PEP 517) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/johnmg/t2t/bin/python /home/johnmg/t2t/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpcvv0fpj6
       cwd: /tmp/pip-install-d2wcoxbe/tokenizers
  Complete output (221 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/tokenizers
  copying tokenizers/__init__.py -> build/lib/tokenizers
  creating build/lib/tokenizers/models
  copying tokenizers/models/__init__.py -> build/lib/tokenizers/models
  creating build/lib/tokenizers/decoders
  copying tokenizers/decoders/__init__.py -> build/lib/tokenizers/decoders
  creating build/lib/tokenizers/normalizers
  copying tokenizers/normalizers/__init__.py -> build/lib/tokenizers/normalizers
  creating build/lib/tokenizers/pre_tokenizers
  copying tokenizers/pre_tokenizers/__init__.py -> build/lib/tokenizers/pre_tokenizers
  creating build/lib/tokenizers/processors
  copying tokenizers/processors/__init__.py -> build/lib/tokenizers/processors
  creating build/lib/tokenizers/trainers
  copying tokenizers/trainers/__init__.py -> build/lib/tokenizers/trainers
  creating build/lib/tokenizers/implementations
  copying tokenizers/implementations/sentencepiece_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/char_level_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/byte_level_bpe.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/bert_wordpiece.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/base_tokenizer.py -> build/lib/tokenizers/implementations
  copying tokenizers/implementations/__init__.py -> build/lib/tokenizers/implementations
  copying tokenizers/__init__.pyi -> build/lib/tokenizers
  copying tokenizers/models/__init__.pyi -> build/lib/tokenizers/models
  copying tokenizers/decoders/__init__.pyi -> build/lib/tokenizers/decoders
  copying tokenizers/normalizers/__init__.pyi -> build/lib/tokenizers/normalizers
  copying tokenizers/pre_tokenizers/__init__.pyi -> build/lib/tokenizers/pre_tokenizers
  copying tokenizers/processors/__init__.pyi -> build/lib/tokenizers/processors
  copying tokenizers/trainers/__init__.pyi -> build/lib/tokenizers/trainers
  running build_ext
  running build_rust
      Updating crates.io index
  warning: unused manifest key: target.x86_64-apple-darwin.rustflags
     Compiling proc-macro2 v1.0.8
     Compiling unicode-xid v0.2.0
     Compiling syn v1.0.15
     Compiling libc v0.2.67
     Compiling autocfg v1.0.0
     Compiling lazy_static v1.4.0
     Compiling cfg-if v0.1.10
     Compiling semver-parser v0.7.0
     Compiling memchr v2.3.3
     Compiling serde v1.0.104
     Compiling maybe-uninit v2.0.0
     Compiling ryu v1.0.2
     Compiling regex-syntax v0.6.14
     Compiling getrandom v0.1.14
     Compiling scopeguard v1.1.0
     Compiling unicode-width v0.1.7
     Compiling itoa v0.4.5
     Compiling bitflags v1.2.1
       Running `rustc --crate-name unicode_xid /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-xid-0.2.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=d0c8361b0afb9c55 -C extra-filename=-d0c8361b0afb9c55 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.8/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=97f59661e87a2bff -C extra-filename=-97f59661e87a2bff --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/proc-macro2-97f59661e87a2bff -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.15/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' --cfg 'feature="visit"' -C metadata=be4245bf41be9154 -C extra-filename=-be4245bf41be9154 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/syn-be4245bf41be9154 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.67/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=4aa17b314a9f9392 -C extra-filename=-4aa17b314a9f9392 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/libc-4aa17b314a9f9392 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name autocfg /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=0b99959d54eb5a43 -C extra-filename=-0b99959d54eb5a43 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name lazy_static /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.4.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=7fe90463f0542b89 -C extra-filename=-7fe90463f0542b89 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name semver_parser /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/semver-parser-0.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=ce71380f50d590b6 -C extra-filename=-ce71380f50d590b6 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name cfg_if /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=1b04fa8f4baea64e -C extra-filename=-1b04fa8f4baea64e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.3.3/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' --cfg 'feature="use_std"' -C metadata=682166ccfd58c578 -C extra-filename=-682166ccfd58c578 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memchr-682166ccfd58c578 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.104/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=e3191056f1858817 -C extra-filename=-e3191056f1858817 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/serde-e3191056f1858817 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/maybe-uninit-2.0.0/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=99bcd9a60d46382c -C extra-filename=-99bcd9a60d46382c --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/maybe-uninit-99bcd9a60d46382c -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ryu-1.0.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=e704a6b7a71f3d7a -C extra-filename=-e704a6b7a71f3d7a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/ryu-e704a6b7a71f3d7a -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name regex_syntax /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-syntax-0.6.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=feb44197369905d4 -C extra-filename=-feb44197369905d4 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.14/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="std"' -C metadata=c2394b8b43d330b2 -C extra-filename=-c2394b8b43d330b2 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/getrandom-c2394b8b43d330b2 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name scopeguard /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/scopeguard-1.1.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=8a63ed9d96488c18 -C extra-filename=-8a63ed9d96488c18 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unicode_width /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-width-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=397b6227577b65ae -C extra-filename=-397b6227577b65ae --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name itoa /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/itoa-0.4.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=b9ca519a13df71bf -C extra-filename=-b9ca519a13df71bf --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
     Compiling ppv-lite86 v0.2.6
     Compiling rayon-core v1.7.0
     Compiling unindent v0.1.5
     Compiling version_check v0.9.1
     Compiling strsim v0.8.0
     Compiling vec_map v0.8.1
     Compiling either v1.5.3
     Compiling number_prefix v0.3.0
     Compiling smallvec v1.2.0
     Compiling ansi_term v0.11.0
     Compiling unicode_categories v0.1.1
     Compiling spin v0.5.2
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.2.1/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' -C metadata=74dee2c088f4fdf7 -C extra-filename=-74dee2c088f4fdf7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/bitflags-74dee2c088f4fdf7 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name ppv_lite86 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ppv-lite86-0.2.6/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=a66cdba604de2e91 -C extra-filename=-a66cdba604de2e91 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.7.0/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=01d02e6ac28bd2a9 -C extra-filename=-01d02e6ac28bd2a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/rayon-core-01d02e6ac28bd2a9 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name version_check /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/version_check-0.9.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=cb4ea0451d56bc0d -C extra-filename=-cb4ea0451d56bc0d --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name unindent /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unindent-0.1.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=54ee88ab0038e61b -C extra-filename=-54ee88ab0038e61b --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name strsim /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/strsim-0.8.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=97f24fc34d9d28cd -C extra-filename=-97f24fc34d9d28cd --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name vec_map /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/vec_map-0.8.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=135a62f8a977656a -C extra-filename=-135a62f8a977656a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name either /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/either-1.5.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6fcca30f288e7f70 -C extra-filename=-6fcca30f288e7f70 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name number_prefix /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/number_prefix-0.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=c617a5b231fd33f2 -C extra-filename=-c617a5b231fd33f2 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --edition=2018 --crate-name smallvec /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/smallvec-1.2.0/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=5a1b93c8a07d924a -C extra-filename=-5a1b93c8a07d924a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name ansi_term /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ansi_term-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=f9c541f6d3ce7af3 -C extra-filename=-f9c541f6d3ce7af3 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name unicode_categories /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode_categories-0.1.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=3376acf28b9791fe -C extra-filename=-3376acf28b9791fe --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
       Running `rustc --crate-name spin /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/spin-0.5.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6c5053f023d06140 -C extra-filename=-6c5053f023d06140 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow`
     Compiling thread_local v1.0.1
       Running `rustc --crate-name thread_local /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/thread_local-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=7cdd46b26d4f9805 -C extra-filename=-7cdd46b26d4f9805 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --cap-lints allow`
     Compiling textwrap v0.11.0
       Running `rustc --crate-name textwrap /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/textwrap-0.11.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=0f312c8508ee8e9d -C extra-filename=-0f312c8508ee8e9d --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern unicode_width=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_width-397b6227577b65ae.rmeta --cap-lints allow`
     Compiling semver v0.9.0
       Running `rustc --crate-name semver /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/semver-0.9.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=21f81775743ad422 -C extra-filename=-21f81775743ad422 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern semver_parser=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsemver_parser-ce71380f50d590b6.rmeta --cap-lints allow`
     Compiling unicode-normalization-alignments v0.1.12
       Running `rustc --crate-name unicode_normalization_alignments /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/unicode-normalization-alignments-0.1.12/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=2f95bcef5d770e35 -C extra-filename=-2f95bcef5d770e35 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern smallvec=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsmallvec-5a1b93c8a07d924a.rmeta --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/rayon-core-01d02e6ac28bd2a9/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memchr-682166ccfd58c578/build-script-build`
       Running `rustc --crate-name memchr /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-2.3.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' --cfg 'feature="use_std"' -C metadata=916f9f60d041f29f -C extra-filename=-916f9f60d041f29f --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg memchr_runtime_simd --cfg memchr_runtime_sse2 --cfg memchr_runtime_sse42 --cfg memchr_runtime_avx`
     Compiling rustc_version v0.2.3
       Running `rustc --crate-name rustc_version /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rustc_version-0.2.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=63af08b6d5f0b1a9 -C extra-filename=-63af08b6d5f0b1a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern semver=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsemver-21f81775743ad422.rmeta --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/getrandom-c2394b8b43d330b2/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/bitflags-74dee2c088f4fdf7/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/maybe-uninit-99bcd9a60d46382c/build-script-build`
     Compiling crossbeam-utils v0.7.2
     Compiling crossbeam-epoch v0.8.2
     Compiling num-traits v0.2.11
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.7.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=5b24f08aed575110 -C extra-filename=-5b24f08aed575110 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-utils-5b24f08aed575110 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libautocfg-0b99959d54eb5a43.rlib --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=515968de6557b6b7 -C extra-filename=-515968de6557b6b7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-epoch-515968de6557b6b7 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libautocfg-0b99959d54eb5a43.rlib --cap-lints allow`
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.11/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=618c6828911959ab -C extra-filename=-618c6828911959ab --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/num-traits-618c6828911959ab -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern autocfg=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libautocfg-0b99959d54eb5a43.rlib --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/ryu-e704a6b7a71f3d7a/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/serde-e3191056f1858817/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/libc-4aa17b314a9f9392/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/syn-be4245bf41be9154/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/proc-macro2-97f59661e87a2bff/build-script-build`
       Running `rustc --crate-name bitflags /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/bitflags-1.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=95683541c46a6653 -C extra-filename=-95683541c46a6653 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg bitflags_const_fn`
       Running `rustc --crate-name maybe_uninit /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/maybe-uninit-2.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=169701ffcef7a104 -C extra-filename=-169701ffcef7a104 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg derive_copy --cfg repr_transparent --cfg native_uninit`
     Compiling c2-chacha v0.2.3
       Running `rustc --edition=2018 --crate-name c2_chacha /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/c2-chacha-0.2.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="simd"' --cfg 'feature="std"' -C metadata=db6d7fc899faf453 -C extra-filename=-db6d7fc899faf453 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern ppv_lite86=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libppv_lite86-a66cdba604de2e91.rmeta --cap-lints allow`
       Running `rustc --crate-name ryu /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ryu-1.0.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=81c16096c65f1d25 -C extra-filename=-81c16096c65f1d25 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg integer128 --cfg must_use_return --cfg maybe_uninit`
       Running `rustc --edition=2018 --crate-name proc_macro2 /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.8/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=ce62abe820ec95ab -C extra-filename=-ce62abe820ec95ab --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern unicode_xid=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_xid-d0c8361b0afb9c55.rmeta --cap-lints allow --cfg use_proc_macro --cfg wrap_proc_macro`
       Running `rustc --crate-name libc /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.67/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5d9b252ed56b1945 -C extra-filename=-5d9b252ed56b1945 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg freebsd11 --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN`
     Compiling aho-corasick v0.7.8
       Running `rustc --crate-name aho_corasick /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.8/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5f8df0f6460a66c2 -C extra-filename=-5f8df0f6460a66c2 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern memchr=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmemchr-916f9f60d041f29f.rmeta --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/num-traits-618c6828911959ab/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-utils-5b24f08aed575110/build-script-build`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/crossbeam-epoch-515968de6557b6b7/build-script-build`
     Compiling memoffset v0.5.3
       Running `rustc --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.5.3/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C metadata=8ebd67a7766256e7 -C extra-filename=-8ebd67a7766256e7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memoffset-8ebd67a7766256e7 -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern rustc_version=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librustc_version-63af08b6d5f0b1a9.rlib --cap-lints allow`
     Compiling quote v1.0.2
       Running `rustc --edition=2018 --crate-name quote /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/quote-1.0.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=5dd3b63b3c37ba50 -C extra-filename=-5dd3b63b3c37ba50 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rmeta --cap-lints allow`
       Running `rustc --crate-name num_traits /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.2.11/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=682c148c69950086 -C extra-filename=-682c148c69950086 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg has_i128`
     Compiling num_cpus v1.12.0
     Compiling termios v0.3.1
     Compiling clicolors-control v1.0.1
     Compiling atty v0.2.14
       Running `rustc --edition=2018 --crate-name getrandom /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.1.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="std"' -C metadata=3bcce62cba29d0a1 -C extra-filename=-3bcce62cba29d0a1 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
       Running `rustc --crate-name num_cpus /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/num_cpus-1.12.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=5375d111fec819b4 -C extra-filename=-5375d111fec819b4 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
       Running `rustc --crate-name termios /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/termios-0.3.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=d64097ba20dddbc5 -C extra-filename=-d64097ba20dddbc5 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
       Running `rustc --crate-name clicolors_control /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/clicolors-control-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="terminal_autoconfig"' -C metadata=f95aedfd36305d68 -C extra-filename=-f95aedfd36305d68 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
       Running `rustc --crate-name atty /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/atty-0.2.14/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=61c34de20facc8fb -C extra-filename=-61c34de20facc8fb --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --cap-lints allow`
       Running `rustc --edition=2018 --crate-name syn /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/syn-1.0.15/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="clone-impls"' --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="extra-traits"' --cfg 'feature="full"' --cfg 'feature="parsing"' --cfg 'feature="printing"' --cfg 'feature="proc-macro"' --cfg 'feature="quote"' --cfg 'feature="visit"' -C metadata=fb7a652ed3ecc931 -C extra-filename=-fb7a652ed3ecc931 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rmeta --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rmeta --extern unicode_xid=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_xid-d0c8361b0afb9c55.rmeta --cap-lints allow --cfg syn_disable_nightly_tests`
     Compiling clap v2.33.0
       Running `rustc --crate-name clap /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/clap-2.33.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="ansi_term"' --cfg 'feature="atty"' --cfg 'feature="color"' --cfg 'feature="default"' --cfg 'feature="strsim"' --cfg 'feature="suggestions"' --cfg 'feature="vec_map"' -C metadata=4d1679758f5cc3c5 -C extra-filename=-4d1679758f5cc3c5 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern ansi_term=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libansi_term-f9c541f6d3ce7af3.rmeta --extern atty=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libatty-61c34de20facc8fb.rmeta --extern bitflags=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libbitflags-95683541c46a6653.rmeta --extern strsim=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libstrsim-97f24fc34d9d28cd.rmeta --extern textwrap=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libtextwrap-0f312c8508ee8e9d.rmeta --extern unicode_width=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_width-397b6227577b65ae.rmeta --extern vec_map=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libvec_map-135a62f8a977656a.rmeta --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/memoffset-8ebd67a7766256e7/build-script-build`
     Compiling rand_core v0.5.1
       Running `rustc --edition=2018 --crate-name rand_core /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_core-0.5.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="alloc"' --cfg 'feature="getrandom"' --cfg 'feature="std"' -C metadata=4adb25904fdd70df -C extra-filename=-4adb25904fdd70df --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern getrandom=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libgetrandom-3bcce62cba29d0a1.rmeta --cap-lints allow`
       Running `rustc --crate-name memoffset /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/memoffset-0.5.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=ee652fbed0600815 -C extra-filename=-ee652fbed0600815 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --cap-lints allow --cfg memoffset_maybe_uninit --cfg memoffset_doctests`
       Running `rustc --crate-name crossbeam_utils /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-utils-0.7.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=71c95db82240db48 -C extra-filename=-71c95db82240db48 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --cap-lints allow --cfg has_min_const_fn --cfg has_atomic_u8 --cfg has_atomic_u16 --cfg has_atomic_u32 --cfg has_atomic_u64`
     Compiling rand_chacha v0.2.1
       Running `rustc --edition=2018 --crate-name rand_chacha /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand_chacha-0.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="std"' -C metadata=164a44df65235912 -C extra-filename=-164a44df65235912 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern c2_chacha=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libc2_chacha-db6d7fc899faf453.rmeta --extern rand_core=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand_core-4adb25904fdd70df.rmeta --cap-lints allow`
     Compiling rand v0.7.3
       Running `rustc --edition=2018 --crate-name rand /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.7.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="alloc"' --cfg 'feature="default"' --cfg 'feature="getrandom"' --cfg 'feature="getrandom_package"' --cfg 'feature="libc"' --cfg 'feature="std"' -C metadata=102d035e4ca6c699 -C extra-filename=-102d035e4ca6c699 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern getrandom_package=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libgetrandom-3bcce62cba29d0a1.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --extern rand_chacha=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand_chacha-164a44df65235912.rmeta --extern rand_core=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand_core-4adb25904fdd70df.rmeta --cap-lints allow`
     Compiling crossbeam-queue v0.2.1
       Running `rustc --crate-name crossbeam_epoch /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-epoch-0.8.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="lazy_static"' --cfg 'feature="std"' -C metadata=4cd0c2190306aa4a -C extra-filename=-4cd0c2190306aa4a --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern maybe_uninit=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmaybe_uninit-169701ffcef7a104.rmeta --extern memoffset=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmemoffset-ee652fbed0600815.rmeta --extern scopeguard=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libscopeguard-8a63ed9d96488c18.rmeta --cap-lints allow --cfg has_min_const_fn`
       Running `rustc --crate-name crossbeam_queue /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-queue-0.2.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=e5afea70501509a9 -C extra-filename=-e5afea70501509a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern cfg_if=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcfg_if-1b04fa8f4baea64e.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --cap-lints allow`
     Compiling regex v1.3.4
       Running `rustc --crate-name regex /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-1.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="aho-corasick"' --cfg 'feature="default"' --cfg 'feature="memchr"' --cfg 'feature="perf"' --cfg 'feature="perf-cache"' --cfg 'feature="perf-dfa"' --cfg 'feature="perf-inline"' --cfg 'feature="perf-literal"' --cfg 'feature="std"' --cfg 'feature="thread_local"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=40c5630aef8afe3e -C extra-filename=-40c5630aef8afe3e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern aho_corasick=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libaho_corasick-5f8df0f6460a66c2.rmeta --extern memchr=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmemchr-916f9f60d041f29f.rmeta --extern regex_syntax=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex_syntax-feb44197369905d4.rmeta --extern thread_local=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libthread_local-7cdd46b26d4f9805.rmeta --cap-lints allow`
     Compiling crossbeam-deque v0.7.3
       Running `rustc --crate-name crossbeam_deque /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/crossbeam-deque-0.7.3/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=6fff16ed40375025 -C extra-filename=-6fff16ed40375025 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern crossbeam_epoch=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_epoch-4cd0c2190306aa4a.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --extern maybe_uninit=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libmaybe_uninit-169701ffcef7a104.rmeta --cap-lints allow`
       Running `rustc --edition=2018 --crate-name rayon_core /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.7.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=5e812e4a0a947026 -C extra-filename=-5e812e4a0a947026 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern crossbeam_deque=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_deque-6fff16ed40375025.rmeta --extern crossbeam_queue=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_queue-e5afea70501509a9.rmeta --extern crossbeam_utils=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_utils-71c95db82240db48.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern num_cpus=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libnum_cpus-5375d111fec819b4.rmeta --cap-lints allow`
     Compiling rayon v1.3.0
       Running `rustc --edition=2018 --crate-name rayon /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.3.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=35b9846c719edc99 -C extra-filename=-35b9846c719edc99 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern crossbeam_deque=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libcrossbeam_deque-6fff16ed40375025.rmeta --extern either=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libeither-6fcca30f288e7f70.rmeta --extern rayon_core=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librayon_core-5e812e4a0a947026.rmeta --cap-lints allow`
     Compiling console v0.9.2
       Running `rustc --edition=2018 --crate-name console /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/console-0.9.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="unicode-width"' -C metadata=a0e410a25b05d297 -C extra-filename=-a0e410a25b05d297 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern clicolors_control=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libclicolors_control-f95aedfd36305d68.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern libc=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblibc-5d9b252ed56b1945.rmeta --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rmeta --extern termios=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libtermios-d64097ba20dddbc5.rmeta --extern unicode_width=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_width-397b6227577b65ae.rmeta --cap-lints allow`
     Compiling indicatif v0.14.0
       Running `rustc --edition=2018 --crate-name indicatif /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indicatif-0.14.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' -C metadata=af81cb79d58cbea4 -C extra-filename=-af81cb79d58cbea4 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern console=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libconsole-a0e410a25b05d297.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern number_prefix=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libnumber_prefix-c617a5b231fd33f2.rmeta --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rmeta --cap-lints allow`
     Compiling pyo3-derive-backend v0.8.5
       Running `rustc --edition=2018 --crate-name pyo3_derive_backend /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-derive-backend-0.8.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=4e1d8a522a8e0abc -C extra-filename=-4e1d8a522a8e0abc --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rmeta --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rmeta --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rmeta --cap-lints allow`
     Compiling serde_derive v1.0.104
     Compiling proc-macro-hack v0.5.11
     Compiling ctor v0.1.12
     Compiling ghost v0.1.1
     Compiling inventory-impl v0.1.5
     Compiling pyo3cls v0.8.5
       Running `rustc --crate-name serde_derive /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_derive-1.0.104/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 --cfg 'feature="default"' -C metadata=c97a5ca23329a0e7 -C extra-filename=-c97a5ca23329a0e7 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name proc_macro_hack /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.5.11/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=6c301fa525410f51 -C extra-filename=-6c301fa525410f51 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name ctor /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ctor-0.1.12/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=6d15aecbd8ecf9a9 -C extra-filename=-6d15aecbd8ecf9a9 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name ghost /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/ghost-0.1.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=a7fa8d8cb581322e -C extra-filename=-a7fa8d8cb581322e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name inventory_impl /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/inventory-impl-0.1.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=07295296bec98a10 -C extra-filename=-07295296bec98a10 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name pyo3cls /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3cls-0.8.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=4e128ea32e108d4e -C extra-filename=-4e128ea32e108d4e --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern pyo3_derive_backend=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libpyo3_derive_backend-4e1d8a522a8e0abc.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
     Compiling paste-impl v0.1.7
     Compiling indoc-impl v0.3.4
       Running `rustc --edition=2018 --crate-name paste_impl /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-impl-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=0100b2f59cf859eb -C extra-filename=-0100b2f59cf859eb --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --cap-lints allow`
       Running `rustc --edition=2018 --crate-name indoc_impl /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indoc-impl-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=b6ef86cd971e397d -C extra-filename=-b6ef86cd971e397d --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --extern proc_macro2=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro2-ce62abe820ec95ab.rlib --extern quote=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libquote-5dd3b63b3c37ba50.rlib --extern syn=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libsyn-fb7a652ed3ecc931.rlib --extern unindent=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunindent-54ee88ab0038e61b.rlib --cap-lints allow`
     Compiling inventory v0.1.5
       Running `rustc --edition=2018 --crate-name inventory /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/inventory-0.1.5/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=81f198a01c0ec25f -C extra-filename=-81f198a01c0ec25f --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern ctor=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libctor-6d15aecbd8ecf9a9.so --extern ghost=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libghost-a7fa8d8cb581322e.so --extern inventory_impl=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libinventory_impl-07295296bec98a10.so --cap-lints allow`
     Compiling indoc v0.3.4
       Running `rustc --edition=2018 --crate-name indoc /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/indoc-0.3.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=3e8e6670af4a3851 -C extra-filename=-3e8e6670af4a3851 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern indoc_impl=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libindoc_impl-b6ef86cd971e397d.so --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --cap-lints allow`
     Compiling paste v0.1.7
       Running `rustc --edition=2018 --crate-name paste /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/paste-0.1.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=331a39d39ec573ee -C extra-filename=-331a39d39ec573ee --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern paste_impl=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libpaste_impl-0100b2f59cf859eb.so --extern proc_macro_hack=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libproc_macro_hack-6c301fa525410f51.so --cap-lints allow`
       Running `rustc --crate-name serde /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde-1.0.104/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="derive"' --cfg 'feature="serde_derive"' --cfg 'feature="std"' -C metadata=ef69b68005b70c62 -C extra-filename=-ef69b68005b70c62 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern serde_derive=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde_derive-c97a5ca23329a0e7.so --cap-lints allow --cfg ops_bound --cfg core_reverse --cfg de_boxed_c_str --cfg de_boxed_path --cfg de_rc_dst --cfg core_duration --cfg integer128 --cfg range_inclusive --cfg num_nonzero --cfg core_try_from --cfg num_nonzero_signed --cfg std_atomic64 --cfg std_atomic`
     Compiling serde_json v1.0.48
       Running `rustc --edition=2018 --crate-name serde_json /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/serde_json-1.0.48/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=5007cd62ba6989a5 -C extra-filename=-5007cd62ba6989a5 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern itoa=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libitoa-b9ca519a13df71bf.rmeta --extern ryu=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libryu-81c16096c65f1d25.rmeta --extern serde=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde-ef69b68005b70c62.rmeta --cap-lints allow`
     Compiling tokenizers v0.7.0 (/tmp/pip-install-d2wcoxbe/tokenizers/tokenizers-lib)
       Running `rustc --edition=2018 --crate-name tokenizers tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C metadata=02f2af0a4056c877 -C extra-filename=-02f2af0a4056c877 --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern clap=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libclap-4d1679758f5cc3c5.rmeta --extern indicatif=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libindicatif-af81cb79d58cbea4.rmeta --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rmeta --extern rand=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librand-102d035e4ca6c699.rmeta --extern rayon=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/librayon-35b9846c719edc99.rmeta --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rmeta --extern regex_syntax=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex_syntax-feb44197369905d4.rmeta --extern serde=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde-ef69b68005b70c62.rmeta --extern serde_json=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde_json-5007cd62ba6989a5.rmeta --extern unicode_normalization_alignments=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_normalization_alignments-2f95bcef5d770e35.rmeta --extern unicode_categories=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libunicode_categories-3376acf28b9791fe.rmeta`
     Compiling pyo3 v0.8.5
       Running `rustc --edition=2018 --crate-name build_script_build /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.5/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 --cfg 'feature="default"' --cfg 'feature="extension-module"' --cfg 'feature="python3"' -C metadata=7ff152acc5305eee -C extra-filename=-7ff152acc5305eee --out-dir /tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/pyo3-7ff152acc5305eee -L dependency=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps --extern lazy_static=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/liblazy_static-7fe90463f0542b89.rlib --extern regex=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libregex-40c5630aef8afe3e.rlib --extern serde=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde-ef69b68005b70c62.rlib --extern serde_json=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libserde_json-5007cd62ba6989a5.rlib --extern version_check=/tmp/pip-install-d2wcoxbe/tokenizers/target/release/deps/libversion_check-cb4ea0451d56bc0d.rlib --cap-lints allow`
       Running `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/pyo3-7ff152acc5305eee/build-script-build`
  error: failed to run custom build command for `pyo3 v0.8.5`

  Caused by:
    process didn't exit successfully: `/tmp/pip-install-d2wcoxbe/tokenizers/target/release/build/pyo3-7ff152acc5305eee/build-script-build` (exit code: 101)
  --- stderr
  thread 'main' panicked at 'Error: pyo3 requires a nightly or dev version of Rust.', /home/johnmg/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.5/build.rs:542:9
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

  warning: build failed, waiting for other jobs to finish...
  error: build failed
  cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module pyo3/python3 --release --verbose -- --crate-type cdylib
  error: cargo failed with code: 101

  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly

Expected behavior

A successful install of transformers.

Environment info

  • transformers version: >2.3.0
  • Platform:
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Python version: 3.7.4
  • PyTorch version (GPU?): 1.4.0
  • Tensorflow version (GPU?): N/A.
  • Using GPU in script?: N/A.
  • Using distributed or parallel set-up in script?: N/A.

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:14 (5 by maintainers)

github_iconTop GitHub Comments

6reactions
ExeCuteRunrunruncommented, Apr 30, 2020

Hey guys, @JohnGiorgi @Shane-Neeley , I think I figured out what was happening to you (to me also). At the time of your question, I think there was a high possibility that in the “setup.py” of transformers’ source code, there was a line writing “tokenizers=0.5.2” ; but in fact this old version will bother the update of transformers. But when I install the newest tokenizers by pip (pip install -U tokenizers), you will get a tokenizers==0.7.0. That’s why at the time of your question, there were always conflits and bugs about this tokenizers (I also installed Rust and setuptool-rust, it was always the same error).

And now they just corrected this line in the setup.py. So I just suggest you to

  1. uninstall your old version transformers (very important!)
  2. pip install -U tokenizers (so that it becomes tokenizers==0.7.0)
  3. install transformers from source !

Then voilà ! You’ll get a brand new smoothy transformers.

0reactions
DarrenZhang01commented, Feb 22, 2021

Hey guys, @JohnGiorgi @Shane-Neeley , I think I figured out what was happening to you (to me also). At the time of your question, I think there was a high possibility that in the “setup.py” of transformers’ source code, there was a line writing “tokenizers=0.5.2” ; but in fact this old version will bother the update of transformers. But when I install the newest tokenizers by pip (pip install -U tokenizers), you will get a tokenizers==0.7.0. That’s why at the time of your question, there were always conflits and bugs about this tokenizers (I also installed Rust and setuptool-rust, it was always the same error).

And now they just corrected this line in the setup.py. So I just suggest you to

  1. uninstall your old version transformers (very important!)
  2. pip install -U tokenizers (so that it becomes tokenizers==0.7.0)
  3. install transformers from source !

Then voilà ! You’ll get a brand new smoothy transformers.

I am installing transformers from source but I am getting the error.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Installation - Hugging Face
TensorFlow 2.0 installation instructions. Flax installation instructions. Install with pip. You should install Transformers in a virtual environment.
Read more >
transformers · PyPI
1. pip install transformers. Copy PIP instructions. Latest version. Released: Dec 1, 2022.
Read more >
Package List — Spack 0.20.0.dev0 documentation
This is a list of things you can install using Spack. It is automatically generated based on the packages in this Spack version....
Read more >
Install spaCy · spaCy Usage Documentation
It's recommended to run the command with python -m to make sure you're executing the correct version of spaCy. pip install -U spacy...
Read more >
Huggingface Transformers 安装报错ImportError: /lib64/libc.so ...
项目场景：近期安装transformers的时候发现了这个报错，之前transformers 3.5.0 ... Cannot install Transformers version >2.3.0 with pip on CentOS.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

How to generate BERT/Roberta word/sentence embedding?

Next page

XLMRobertaTokenizer vocab size