Cache conda packages for Windows builds

It should be possible to cache the conda packages which are always installed on AppVeyor to speed up the build. AppVeyor provides documentation on using a build cache.

My current plan on how to implements this is to cache all the packages which are downloaded and install by the following lines of the appveyor.yml file

    - cmd: conda update --yes --quiet conda
    - cmd: set PYTHONUNBUFFERED=1
    - cmd: conda config --add channels conda-forge
    - cmd: conda install --yes --quiet obvious-ci conda-build-all

These packages come out to ~30 MB for both win-32 and win-64, so ~60 MB total which is under the 100 MB max cache recommendation from AppVeyor. On though would be to add a file backed channel to the conda configuration and then copy the bz2 files from the Miniconda package cache into this directory after installing conda-build-all. The packages would still need to be extracted from this directory.

AppVeyor allows cache-invalidation by watching for changes to a specified files in the repository. Adding two files for invalidating the win-32 and win-64 caches would be prudent. We could roll a cache number in these files whenever new packages are released which are installed by these commands. This would require someone to watch for this to occur but it may be possible to automate this process.

I’ll try to put together a example PR on this topic in the coming week.

[jjhelmus]

@jakirkham I love the idea of using constructor to create a conda-forge installer which includes all the packages that would normally be install prior to a build. Caching this installer on each platform will probably be easier than caching the packages themselves. A few questions on the idea that should be fleshed out:

• Should we only use packages from the conda-forge channel or would including packages from the defaults channel be acceptable.
• Where would we host such an installer? One option would be a GitHub repo with each installer being a released. Where-ever this is hosted it will need to be reliable as the CI services will be downloading from it quite regularly, although caching would reduce this burden.
• How often would this installer need to be updated with new packages? Would newer packages be conda install-ed on top of this installer or would this also pin a particular set of packages.

I see that this proposal is included in the agenda for tomorrow’s call, perhapes we can work some of these issues out then.

[jjhelmus]

After a bit more thinking and googling it should be possible sync the the bz2 files between the two folders so that new packages are automatically copied over to the cache and unused packages are deleted. To manual cache invalidation should be required! robocopy and this SO question should help with the syncing. Also, the documentation on custom conda channels will be helpful.