Build conda packages for ARM

@show0k commented on Fri Sep 02 2016

Hi, Did you ever think about building conda package for ARM architecture ? Since August 2015, conda is officially supporting ARMv7l architecture. It is possible to start ARM Docker container from an intel-64 arch thanks to qemu-arm-static. It could be used to build every conda-forge packages for arm platform with travis-ci or circle-ci.

Now a day I’m building almost every conda packages I need for my ARM board because there few official builds, but I think that it could have a growing ARM/Raspberry Pi community if there were more official or at least conda-forge packets.

Do you think that there is too few ARM users for conda package that it does not worth it ?

---

@jakirkham commented on Mon Sep 05 2016

There’s a good chance that one will need to run such a Docker container in privileged mode. Not sure that CircleCI or other CIs support that.

---

@jakirkham commented on Mon Sep 05 2016

This may be a good tool for containerizing VMs.

---

@show0k commented on Mon Sep 12 2016

There is no need to run container in privileged mode. Usualy to run an ARM docker image on a x86_64 host it needs a kernel with binfmt_misc support, which is the case for travis-ci but not for CircleCI.

But, with a minor modification on qemu, it is possible to embed it on the image and run the ARM image without anything special on the host.

I made a POC image on show0k/miniconda-armv7.

---

@groutr commented on Sat Sep 24 2016

I wonder if, in this case, using the linaro tool chain to cross compile would be a good a option.

---

@show0k commented on Mon Sep 26 2016

I think that the Docker idea is better than the cross-compliation, it emulates the whole system, which is easier to integrate with the actual conda & conda-forge scripts. In background, they both use Qemu for the ARM emulation, so the overhead is the same.

---

@pelson commented on Thu Sep 29 2016

I’m totally up for this. I’ve suffered conda build on the RaspberryPi and it was a terrible experience (conda build ... … wait… wait some more… go to sleep fail. conda build...… wait … wait some more … go to work … fail). My recipes lived in https://github.com/pelson/raspberrypi-conda-recipes and I got quite a way into the stack (in fact, many of those recipes were the basis for conda-forge recipes).

The major things that I’d want to consider though:

• how we reduce the impact on build time for the existing conda-forge system
• whether we can improve on the existing noarch conda concept to save us having duplicate conda distributions for non-compiled things
• how we integrate this into conda-forge without breaking existing feedstocks (that may not yet compile on ARM)

---

@jakirkham commented on Thu Sep 29 2016

One idea I was toying with earlier was actually having another org for this. It may sound a bit extreme, but this is something that one of the Homebrew devs did when they wanted to support Mac OS 10.4 (Tiger). At least worth keeping in the back of our minds if nothing else.

xref: https://github.com/mistydemeo/tigerbrew

---

@show0k commented on Thu Sep 29 2016

The problem of build time is really an issue we have to deal with. Throw Qemu the build time on my i7-5557U@3Ghz laptop is two time slower than on Raspberry Pi 3. I mainly use a Odroid XU4 which is far faster than a Raspberry Pi for testing new builds rules.

I think that we can use another free CI provider, in the same way you separate Linux and OSX build with CircleCi and Travis CI. This would solve the impact on build time for x64_86 build, and avoid labelling the existing build as faild.

Another solution could be to use a paid farm of ARM hardware which run Jenkins or Drone on it. Scaleway have great and cheap hardware, but I suppose that managing money and donations for build is out of the scope of conda-forge.

---

@pelson commented on Thu Sep 29 2016

Managing money is achievable (e.g. NumFOCUS), managing hardware is harder (and the software infrastructure around it) - it would ultimately take time away from other conda-forge activities. I have long wanted a CI service where we can attach donated systems (e.g. personal laptop cycles, corporately donated AWS instances etc.), but we aren’t there yet 😉 .

Your Qemu experience doesn’t sound like it was much fun 😄

---

@jakirkham commented on Tue Nov 15 2016

I’m going to move this over to the webpage repo for a few reasons.

1. This is an organization discussion.
2. It will get more visibility.

Hope that is ok. Thanks everyone for bring this up and providing interesting suggestions. Looking forward to seeing what others might contribute.

[jakirkham]

Just learned recently that Shippable supports ARMv8-A, which would provide us a way to build and upload packages for ARM. Not seeing anything about support for older ARM processors, but it’s possible I’m missing something.

[jjhelmus]

I’m been building a bunch of conda packages for the Raspberry Pi 2/3 (armv7l) and 1/0 (armv6l). They are available on the rpi channel on Anaconda.org.

The recipes were mostly derived from conda-forge (thanks all for such a great resources) and are available in the jjhelmus/rpi-recipes repo. I’ll be adding docs, a README, proper license, etc and likely renaming this repo in the next week but wanted to give conda-forge folks a heads up.