Add support for packagcompiler for julia support

cc: @davidanthoff @arnavs This is a specific implementation of the discussion in #601

We have now discussed how best to annotate project files for packagecompiler, which is a prerequisite for getting immediate support in repo2docker

Background Right now, repo2docker doesn’t support the package compiler but:

• @arnavs has a proof of concept in https://github.com/arnavs/compiled-binder-example which relies on the fancy https://github.com/arnavs/compiled-binder-example/blob/master/Dockerfile to do the package compilation. It is not possible to do this with a postBuild
• In practice, it is not possible or desireable to package compile every package, so you need a list of settings and whitelisted packages to compile into the system image.
• The Project.toml and Manifest.toml file need to be the only sources of setup for the repository

Extension to the Project File After discussion with maintainers of Julia and the package manager (cc: @KristofferC @aviks) the proposal was to standardize on a section such as the following inside of the Project.toml files:

[deps]
Markdown = "XXXXXXXXXXX"
Random = "XXXXXXX"

[packagecompiler]
compilerflags = "-O3"
packages = ["Markdown", "Random"]

Note that this is not repo2docker specific, and could be used by other backend systems for packagecompile’ing based on a Project and Manifest. It would be hand-modified in the Project.toml.

@SimonDanisch any thoughts on this including the naming or additional options to be able to pass to the compile_incremental?

Implementation of PackageCompiler Support

Now, to implement support repo2docker, we can produce a PR which does the following:

• Around https://github.com/jupyter/repo2docker/blob/master/repo2docker/buildpacks/julia/julia_project.py#L30 where the Project is loaded, we would check if 'packagecompiler' in project_toml: and if so, load up all of the packages inside of it into a list for later. Similarly, if compilerflags was given, we would store it.
• In https://github.com/jupyter/repo2docker/blob/master/repo2docker/buildpacks/julia/julia_project.py#L126 we would check to see if any packagecompiler packages were given. If so, then we do a variation of the logic in https://github.com/jupyter/repo2docker/blob/master/repo2docker/buildpacks/julia/julia_project.py#L130-L131 where we packagecompile the passed in list, and pass in the compilerflags as an option.
• Adds additional documentation comment to https://mybinder.readthedocs.io/en/latest/config_files.html#project-toml-install-a-julia-environment about package compiling

[davidanthoff]

One more thought: I think we should maybe not yet give up on postBuild.

I think, now that I understand the current state of PackageCompiler.jl better, it is actually not too surprising that we couldn’t get that to work: PackageCompiler.jl currently really doesn’t seem to work with anything but the default julia environment, and of course here in the repo2docker story we use a custom project at the core of the design. So that can’t work out.

But, if PackageCompiler.jl worked properly with julia environments, it really should all just work, or at least I don’t really see a reason why not. So maybe it is worth trying to fix the julia env story in PackageCompiler.jl, and then try the postBuild approach again?

[davidanthoff]

Yes, thanks @aviks, this is incredibly helpful!

Did juliabox at some point have a custom sysimage? Or did I completely mix that up?

I was not aware that with a custom sysimage, one can’t update packages anymore. I had thought that if I include a package in a custom sysimage, and then do a pkg> up that updates that package, I would lose the benefit of the precompiled stuff, but that my environment would then load the new, updated version of the package. But if I understand things right now, the sysimage version of the package would then essentially take precedence over the version of package in the environment, even if the environment has a newer package version… If that is so, it is a real bummer…

So maybe all of that does speak in favor of an opt-in approach, at least once PackageCompiler.jl is more stable…

I have to admit that I’m not super happy with any of the options here (including not doing anything)…