PyCall 1.92.0 breaks sysimage build?

I see PyCall 1.92.0 was just released.

Now my system image build, of the pyjulia system image on top of a custom sysimage, is failing. See the output.

I’ve tried under both Julia 1.4.2 and 1.5.2, Python 3.8.5, Pyjulia 0.5.6.

Actually building on top of the standard julia sysimage breaks also, with the same errors produced, using build_sysimage("lib/test.so", compiler_env=".") and build_sysimage("lib/test.so").

When PyCall 1.91.4 was the latest version, that was used, and the build succeeded, although with warnings about “incremental precompilation of this module may be fatally broken” or similar when I was building the pyjulia sysimage on top of the custom sysimage.

Can I control the version of PyCall that is pulled in to make a sysimage? Can I use an existing PyCall installation rather than always downloading a version to the temporary environment?

At worst I can simply not do the pyjulia sysimage build on top of the custom sysimage, since using the custom sysimage directly with e.g. jl = Julia(sysimage="lib/rr_sys.so") seems to work, just giving a moderately slower startup time.

> python
Python 3.8.5 (default, Sep  5 2020, 10:50:12) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia.sysimage import build_sysimage
>>> build_sysimage("lib/pj_rr_sys.so", base_sysimage="lib/rr_sys.so", compiler_env=".") 
 Activating new environment at `/tmp/tmp.pyjulia.sysimage.uny9ajpf/Project.toml`
[ Info: Loading PackageCompiler...
[ Info: Installing PyCall...
 Activating new environment at `/tmp/tmp.pyjulia.sysimage.uny9ajpf/Project.toml`
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
  Resolving package versions...
Updating `/tmp/tmp.pyjulia.sysimage.uny9ajpf/Project.toml`
  [438e738f] + PyCall v1.92.0
Updating `/tmp/tmp.pyjulia.sysimage.uny9ajpf/Manifest.toml`
  [8f4d0f93] + Conda v1.4.1
  [682c06a0] + JSON v0.21.1
  [1914dd2f] + MacroTools v0.5.5
  [69de0a69] + Parsers v1.0.10
  [438e738f] + PyCall v1.92.0
  [81def892] + VersionParsing v1.2.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [de0858da] + Printf
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [8dfed614] + Test
  [4ec0a83e] + Unicode
[ Info: Compiling a temporary system image without `PyCall`...
[ Info: PackageCompiler: creating system image object file, this might take a while...
[ Info: Compiling system image...
ERROR: LoadError: LoadError: LoadError: UndefVarError: pyversion_build not defined
Stacktrace:
 [1] top-level scope at none:1
 [2] eval(::Module, ::Any) at ./boot.jl:331
 [3] @static(::LineNumberNode, ::Module, ::Any) at ./osutils.jl:19
 [4] #macroexpand#36 at ./expr.jl:108 [inlined]
 [5] macroexpand at ./expr.jl:107 [inlined]
 [6] docm(::LineNumberNode, ::Module, ::Any, ::Any, ::Bool) at ./docs/Docs.jl:521 (repeats 2 times)
 [7] @doc(::LineNumberNode, ::Module, ::String, ::Vararg{Any,N} where N) at ./boot.jl:464
 [8] include(::Function, ::Module, ::String) at ./Base.jl:380
 [9] include(::Module, ::String) at ./Base.jl:368
 [10] top-level scope at none:2
 [11] eval at ./boot.jl:331 [inlined]
 [12] eval(::Expr) at ./client.jl:467
 [13] top-level scope at ./none:3
in expression starting at /home/graham/.julia/packages/PyCall/l64G5/src/PyCall.jl:924
in expression starting at /home/graham/.julia/packages/PyCall/l64G5/src/PyCall.jl:913
in expression starting at /home/graham/.julia/packages/PyCall/l64G5/src/PyCall.jl:913
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
error in running finalizer: UndefVarError(var=:_finalized)
ERROR: LoadError: Failed to precompile PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0] to /home/graham/.julia/compiled/v1.5/PyCall/mmQEj_RLR2M.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1305
 [3] _require(::Base.PkgId) at ./loading.jl:1030
 [4] require(::Base.PkgId) at ./loading.jl:928
 [5] require(::Module, ::Symbol) at ./loading.jl:923
 [6] include(::Function, ::Module, ::String) at ./Base.jl:380
 [7] include(::Module, ::String) at ./Base.jl:368
 [8] exec_options(::Base.JLOptions) at ./client.jl:296
 [9] _start() at ./client.jl:506
 [10] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [11] invokelatest(::Any) at ./essentials.jl:709
 [12] top-level scope at none:1
in expression starting at /home/graham/.local/lib/python3.8/site-packages/julia/precompile.jl:1
ERROR: LoadError: failed process: Process(`/home/graham/.local/bin/julia-py --color=yes --startup-file=no --sysimage=/tmp/jl_ezKvE4/pj_rr_sys.so --project=. --compile=all --trace-compile=/tmp/jl_xRTRH4 /home/graham/.local/lib/python3.8/site-packages/julia/precompile.jl`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error at ./process.jl:525 [inlined]
 [2] read(::Cmd) at ./process.jl:412
 [3] run_precompilation_script(::String, ::String, ::String) at /home/graham/.julia/packages/PackageCompiler/vsMJE/src/PackageCompiler.jl:177
 [4] create_sysimg_object_file(::String, ::Array{String,1}; project::String, base_sysimage::String, precompile_execution_file::Array{String,1}, precompile_statements_file::Array{String,1}, cpu_target::String, script::Nothing, isapp::Bool) at /home/graham/.julia/packages/PackageCompiler/vsMJE/src/PackageCompiler.jl:204
 [5] create_sysimage(::Array{Symbol,1}; sysimage_path::String, project::String, precompile_execution_file::String, precompile_statements_file::Array{String,1}, incremental::Bool, filter_stdlibs::Bool, replace_default::Bool, cpu_target::String, script::Nothing, base_sysimage::String, isapp::Bool) at /home/graham/.julia/packages/PackageCompiler/vsMJE/src/PackageCompiler.jl:423
 [6] (::var"#7#8")(::String) at /home/graham/.local/lib/python3.8/site-packages/julia/compile.jl:34
 [7] mktempdir(::var"#7#8", ::String; prefix::String) at ./file.jl:682
 [8] mktempdir(::Function, ::String) at ./file.jl:680 (repeats 2 times)
 [9] top-level scope at /home/graham/.local/lib/python3.8/site-packages/julia/compile.jl:24
 [10] include(::Function, ::Module, ::String) at ./Base.jl:380
 [11] include(::Module, ::String) at ./Base.jl:368
 [12] exec_options(::Base.JLOptions) at ./client.jl:296
 [13] _start() at ./client.jl:506
 [14] #invokelatest#1 at ./essentials.jl:710 [inlined]
 [15] invokelatest(::Any) at ./essentials.jl:709
 [16] top-level scope at none:1
in expression starting at /home/graham/.local/lib/python3.8/site-packages/julia/compile.jl:23
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/graham/.local/lib/python3.8/site-packages/julia/sysimage.py", line 110, in build_sysimage
    check_call(build_sysimage_cmd(julia_py, julia, compile_args), cwd=path)
  File "/home/graham/.local/lib/python3.8/site-packages/julia/sysimage.py", line 64, in check_call
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/graham/.local/bin/julia-py', '--julia', 'julia', '--compiled-modules=no', '--color=yes', '/home/graham/.local/lib/python3.8/site-packages/julia/compile.jl', '.', '/home/graham/.local/lib/python3.8/site-packages/julia/precompile.jl', '/home/graham/projects/julia/RecommendRebalance/lib/pj_rr_sys.so', '/home/graham/projects/julia/RecommendRebalance/lib/rr_sys.so']' returned non-zero exit status 1.
>>> 

[tkf]

Hmm… I thought I suppressed these warnings (but I’m not sure). These warnings are expected. I think they should be harmless.

[tkf]

Yeah, I agree that’s a great idea. We’ll have it once #422 is merged.