Rebuilding for longer path length (conda-build 2.0.0)

The build prefix is going to get longer in conda-build 2.0.0. ( https://github.com/conda/conda-build/pull/877 ) There has been some discussion about what this will affect and what needs to get rebuilt. I have moved this from a different thread so the discussion can see the light of day. 😄 An excerpt of it is below. Gist is we need to rebuild a few things some of which we know. Some we may not have. These include curl, fftw, pkg-config, and tk. If swig ever gets added, we have to watch out for that too. I suspect libtool and git will also be affected.

---

@msarahan commented on Sun Jun 05 2016

@stuarteberg - 2.0.0beta tagged: https://github.com/conda/conda-build/releases/tag/2.0.0beta

---

@stuarteberg commented on Sun Jun 05 2016

OHHH yeahhhh…

… wait, there’s no emoji for the kool-aid man? W. T. F.

I’ll try to test this out this week. Thanks for the heads-up.

---

@stuarteberg commented on Mon Jun 06 2016

2.0.0beta tagged

I’ll try to test this out this week.

Whoa now. I clearly wasn’t paying enough attention to conda/conda-build#877. 😃

Sounds like the right decision was made. But it will take time to test – I need to rebuild my entire stack and it will be a few days before I can do that. When do plan on turning the ‘beta’ release into a real release?

---

@msarahan commented on Mon Jun 06 2016

Sometime before mid-June. It can stew for a week or two.

On Mon, Jun 6, 2016 at 3:05 PM Stuart Berg [e-mail] wrote:

2.0.0beta tagged

I’ll try to test this out this week.

Whoa now. I clearly wasn’t paying enough attention to conda/conda-build#877 https://github.com/conda/conda-build/pull/877. 😃

Sound like the right decision was made. But it will take time to test – I need to rebuild my entire stack and it will be a few days before I can do that. When do plan on turning the ‘beta’ release into a real release?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/conda-forge/jpeg-feedstock/pull/2#issuecomment-224071792, or mute the thread https://github.com/notifications/unsubscribe/AACV-e8K4EbpfEzQyKN_toW3zajyP8mcks5qJH1xgaJpZM4Ifozb .

---

@msarahan commented on Mon Jun 06 2016

Please keep me posted on your findings. I’m especially interested in how compatible new packages are with old ones. I think they should be interchangeable, but only the new ones will work on systems with long prefixes.

---

@stuarteberg commented on Mon Jun 06 2016

My very first attempt at building a package failed immediately (on OS X). The _build... prefix was really long, and apparently I have dependency that contains binary files which include the prefix.

It looks like many of my recipes include detect_binary_files_with_prefix: true, which causes many (all?) of the dylibs to be listed in the package’s info/has_prefix metadata. Was it a mistake to use that detect_binary_files_with_prefix setting in the first place?

---

@msarahan commented on Mon Jun 06 2016

I don’t think it was a mistake to use that, but it does imply the long prefix. What were the error messages? Are you able to build something with no dependencies?

---

@stuarteberg commented on Mon Jun 06 2016

What were the error messages?

I’m building a package that depends on fftw. My fftw package is from my own channel (ilastik - for reference, recipe here: https://github.com/ilastik/ilastik-build-conda/blob/master/fftw/meta.yaml)

So while it was creating the _build environment, the linking step failed when it tried to “link” fftw:

ERROR: placeholder '/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho' too short in: ilastik::fftw-3.3.4-1

Are you able to build something with no dependencies?

Yes. For example, this recipe builds: https://github.com/ilastik/ilastik-build-conda/blob/master/lz4/meta.yaml

I’ll do some more digging (and thinking) about this later this week. If I have to rebuild my whole stack, that’s no big deal. But it would be nice if we can come up with clear guidance for people who run into the same issue I’m seeing.

---

@msarahan commented on Mon Jun 06 2016

Agreed. Thanks for being my guinea pig.

---

@jakirkham commented on Mon Jun 06 2016

Well, the placeholder size changed in 2.0.0beta. Not sure if you caught that or not, @stuarteberg, but that could be causing you some pain.

---

@msarahan commented on Tue Jun 07 2016

@stuarteberg before you get too far, I’m going to tag a 1.21.0 release that has everything but the prefix length change. We are working on a new Anaconda release, and this change is simply too disruptive so close to a release.

Is it true to say that new builds can not use old builds, but old builds can use new builds?

---

@stuarteberg commented on Tue Jun 07 2016

I’m going to tag a 1.21.0 release

Sounds like a good idea.

Is it true to say that new builds can not use old builds, but old builds can use new builds?

I think that’s right, if I understand the problem correctly.

---

@stuarteberg commented on Tue Jun 07 2016

Continuing to investigate the example from above (my problematic fftw package), here’s what I see.

(Reminder: this package was built with detect_binary_files_with_prefix: true)

$ cat /miniconda/pkgs/fftw-3.3.4-1/info/has_prefix
/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho binary lib/libfftw3.3.dylib
/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho binary lib/libfftw3.dylib
/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho binary lib/libfftw3f.3.dylib
/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho binary lib/libfftw3f.dylib
/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho binary lib/libfftw3l.3.dylib
/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho binary lib/libfftw3l.dylib
/opt/anaconda1anaconda2anaconda3 text lib/libfftw3.la
/opt/anaconda1anaconda2anaconda3 text lib/libfftw3_threads.la
/opt/anaconda1anaconda2anaconda3 text lib/libfftw3f.la
/opt/anaconda1anaconda2anaconda3 text lib/libfftw3f_threads.la
/opt/anaconda1anaconda2anaconda3 text lib/libfftw3l.la
/opt/anaconda1anaconda2anaconda3 text lib/libfftw3l_threads.la
/opt/anaconda1anaconda2anaconda3 text lib/pkgconfig/fftw3.pc
/opt/anaconda1anaconda2anaconda3 text lib/pkgconfig/fftw3f.pc
/opt/anaconda1anaconda2anaconda3 text lib/pkgconfig/fftw3l.pc

OK, so for some reason all of the .dylib files have the prefix embedded in them. But, wait, they use relative RPATHs and whatnot. Why do they contain the prefix?

Here’s an ugly command that identifies the files containing _build and prints out the guilty strings:

$ for f in $(find /miniconda/pkgs/fftw-3.3.4-1/lib/ -type f | xargs grep -l _build); do echo "$f:"; strings $f | grep _build; done
/miniconda/pkgs/fftw-3.3.4-1/lib//libfftw3.3.dylib:
gcc -arch x86_64 -I/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho/include
/miniconda/pkgs/fftw-3.3.4-1/lib//libfftw3.a:
gcc -arch x86_64 -I/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho/include
/miniconda/pkgs/fftw-3.3.4-1/lib//libfftw3f.3.dylib:
gcc -arch x86_64 -I/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho/include
/miniconda/pkgs/fftw-3.3.4-1/lib//libfftw3f.a:
gcc -arch x86_64 -I/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho/include
/miniconda/pkgs/fftw-3.3.4-1/lib//libfftw3l.3.dylib:
gcc -arch x86_64 -I/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho/include
/miniconda/pkgs/fftw-3.3.4-1/lib//libfftw3l.a:
gcc -arch x86_64 -I/miniconda/envs/_build_placehold_placehold_placehold_placehold_placehold_placeho/include

OK, so apparently the gcc command is stored within the binaries for some reason? BTW, I checked on Linux, and it’s the same. Not sure why this is the case. Do you know?

---

@msarahan commented on Tue Jun 07 2016

I don’t know, but I guess I get to learn.

---

@msarahan commented on Tue Jun 07 2016

1.21.0beta tagged: https://github.com/conda/conda-build/releases

---

@stuarteberg commented on Tue Jun 07 2016

I don’t know, but I guess I get to learn.

It might not be worth learning: Looking through my packages, there are several dylib files that DO include the _build_placeholder... prefix, but for different reasons (i.e. not the gcc commands as shown above). I don’t know if those uses of it are important (I suspect most aren’t), but it’s probably not worth investigating each one.

Even so, I’m attempting to get some explanation for the embedded build commands, just for curiosity’s sake: http://stackoverflow.com/questions/37684320/what-causes-a-compiled-library-to-store-its-build-command-internally

---

@stuarteberg commented on Tue Jun 07 2016

OK, I did some more digging (with the help of a stackoverflow user), and it turns out that fftw is the only package on my machine that includes it’s own build command in the binary itself. (For the record, it’s in a variable named FFTW_CC, which makes it’s way into the binary by way of api/version.c)

Anyway, whatever, it doesn’t matter. I think there’s no way around it: People whose detect_binary_files_with_prefix: true may have to rebuild some of their packages.

The good news is that – as far as I can tell – this applies to very few of the packages in the default anaconda distribution.

But there are other packages from the defaults channel that will need updating, such as:

• curl (The latest version needs to be rebuilt, but the version pinned in anaconda doesn’t for some reason.)
• tk (ditto)
• gcc
• pkg-config
• swig

… and probably more.

---

@msarahan commented on Tue Jun 07 2016

Good to know. Thanks! Should we be trying to clear that information? Any idea why people put it there? Posterity’s sake?

---

@stuarteberg commented on Tue Jun 07 2016

Should we be trying to clear that information?

I don’t see the harm in leaving it there. Does conda even provide fftw? I don’t think it does. This was in one of my own packages. I guess now that you guys ship mklfft, there’s no need for me to use fftw anyway…

---

@jakirkham commented on Tue Jun 07 2016

Does conda even provide fftw?

conda-forge does.

---

@stuarteberg commented on Tue Jun 07 2016

conda-forge does.

Then make sure you rebuild it when conda-build 2.0 comes out! 😉

---

@jakirkham commented on Tue Jun 07 2016

Duly noted.

curl (The latest version needs to be rebuilt, but the version pinned in anaconda doesn’t for some reason.)

So, curl hardcodes the path to where the certificates live. I’m guessing old versions of curl used these from the system. So they were not affected by the prefix length during build time. Newer versions of curl (from defaults) use certificates that are provided in the openssl package. So, they are affected by the prefix during build time. At conda-forge we have a separate certificates package that ends up living in the same location as where the openssl one from defaults place them for compatibility reasons. So, it will probably be affected. Though we already knew about this as we had to fix it before. 😄

tk (ditto)

Not sure where this hardcodes things. Did you see this anywhere in it? Probably points to $PREFIX/lib/tkX.Y/ and $PREFIX/lib/tclX.Y/.

gcc

Not surprised. Any chance we could get a rebuild of gcc before that conda-build release, @msarahan? I think only a few packages are affected by this (ones using Fortran or OpenMP), but we should probably get the compiler fixed for them.

pkg-config

Have not inspected this, but I’m guessing it hardcodes the path to $PREFIX/lib/pkgconfig somewhere. Hence unsurprising it needs a rebuild.

Surprised you didn’t mention libtool, which adds .la files to $PREFIX/lib/. Would figure it has the same problem.

swig

I don’t really use SWIG. So, I trust your judgement. Probably hardcoding some path to some provided .i files.

---

@jakirkham commented on Tue Jun 07 2016

Did you try git? I think that might be affected too. At least, I suspect ours will be. Guessing defaults is similar.

---

@jakirkham commented on Tue Jun 07 2016

Alright, this conversation needs to see the light of day (not a closed unrelated PR), I’m going to try using ZenHub’s move issue feature, but this could go horribly wrong. So, please fasten your seatbelts. 😁

[jjhelmus]

Updated my original gist so that the script now downloads and checks a package from each platform (osx, linux, and win). This might still miss a few corner cases but I think it is good enough. The list of packages which use a binary prefix that this script finds are:

$ python binary_prefix_check_file.py
bison uses a binary prefix
curl uses a binary prefix
cycamore uses a binary prefix
cyclus uses a binary prefix
dbus uses a binary prefix
eccodes uses a binary prefix
ecmwf_grib uses a binary prefix
flex uses a binary prefix
fontconfig uses a binary prefix
git uses a binary prefix
glib uses a binary prefix
graphviz uses a binary prefix
harfbuzz uses a binary prefix
hdf5 uses a binary prefix
iverilog uses a binary prefix
lua uses a binary prefix
mpi4py uses a binary prefix
ncurses uses a binary prefix
obspy uses a binary prefix
openmpi uses a binary prefix
openssl uses a binary prefix
openturns uses a binary prefix
pango uses a binary prefix
pkg-config uses a binary prefix
python-eccodes uses a binary prefix
python-ecmwf_grib uses a binary prefix
python-spams uses a binary prefix
simbody uses a binary prefix
swig uses a binary prefix
texlive-core uses a binary prefix
tk uses a binary prefix
udunits uses a binary prefix
udunits2 uses a binary prefix
uwsgi uses a binary prefix
vigra uses a binary prefix
Uses a binary prefix: 35
Does NOT use a binary prefix: 1168
Total: 1203

[msarahan]

so, try to use long first, but if that fails, fall back to short? That’s probably a good idea. It might make the transition less jarring.