conda fails to install python=36 (verification error)

On an Ubuntu 16.04 system which had previously had litex-buildenv installed, updating to the latest commit (52f75717ac6) which switches to python-36 results in a system that:

• Fails scripts/enter-env.sh because python 3.6 is missing; and

• Fails scripts/download-env.sh because conda fails to install python 3.6 due to CondaVerificationError on seemingly every single python file, eg

  CondaVerificationError: The package for chardet located at /src/fpga-miniconf/litex-buildenv/build/conda/pkgs/chardet-3.0.4-py36h0f667ec_1 appears to be corrupted. The path ‘lib/python3.6/site-packages/chardet/pycache/init.cpython-36.pyc’ specified in the package manifest cannot be found.

(see also gist with full output).

From some reading (eg, https://github.com/conda/conda-build/issues/1301, https://github.com/conda/conda/issues/6033, https://github.com/conda-forge/python-feedstock/issues/70) my current guess is that perhaps this was an issue with an older version of conda, and then the bug got fixed, and everyone moved on. However it appears that scripts/download-env.sh will only ever update conda on first installation (from miniconda) and then after that assumes that the installed version will work perfectly; see

https://github.com/timvideos/litex-buildenv/blob/52f75717ac6e7fee1db8d206dd77d4a542904bc7/scripts/download-env.sh#L192-L208

(and note the conda ... update is guarded by if no bin/conda exists, so it’ll only happen once).

I’m about to try manually doing conda update conda and see if that is sufficient to fix the issue. But am making an issue now as a record in case others run into the same issue…

Ewen

[ewenmcneill]

After concluding that reinstalling the conda environment was going to be quicker, I did:

mv build/conda build/conda-3.5
# set environment variables
scripts/download-env.sh

and it seemed to work perfectly. I assume the python=3.5 to python=3.6 upgrade process isn’t very well tested. I did briefly try to find a combination of conda packages I could remove to avoid the .pyc complaints… but a bunch that I tried were conda dependencies, and unsurprisingly conda is unwilling to remove itself, so I gave up on that approach.

FTR, output of reinstalling after mv build/conda build/conda-3.5. At least with a fast network connection it took only a few minutes – way less time than I spent trying to make it work without doing that 😃 (And it might have been faster if I had, eg, found a way to save conda/pkgs/ or similar.)

Ewen

[ewenmcneill]

In case it helps, I have conda 4.4.7 installed, using Python 3.5, along with these packages:

(LX P=arty C=or1k.linux F=linux R=in-tftpd-fix) ewen@parthenon:/src/fpga-minicon
f/litex-buildenv$ conda --version
conda 4.4.7
(LX P=arty C=or1k.linux F=linux R=in-tftpd-fix) ewen@parthenon:/src/fpga-minicon
f/litex-buildenv$ conda list
# packages in environment at /src/fpga-miniconf/litex-buildenv/build/conda:
#
asn1crypto                0.24.0                   py35_0  
binutils-lm32-elf         2.28.0          20180115_065451    timvideos
binutils-or1k-elf         2.28.0          20180108_070246    timvideos
ca-certificates           2017.08.26           h1d4fec5_0  
certifi                   2017.11.5        py35h9749603_0  
cffi                      1.11.2           py35hc7b2db7_0  
chardet                   3.0.4            py35hb6e9ddf_1  
cloog                     0.18.0                        0  
colorama                  0.3.9                     <pip>
conda                     4.4.7                    py35_0  
conda-env                 2.6.0                h36134e3_1  
cryptography              2.1.4            py35hbeb2da1_0  
decorator                 4.1.2            py35h3a268aa_0  
flterm                    2.4_24_gee93960               0    timvideos
gcc-lm32-elf-newlib       5.4.0           20171225_070346    timvideos
gcc-lm32-elf-nostdc       5.4.0           20171211_070518    timvideos
gcc-or1k-elf-newlib       5.4.0_4334_g9310fdc97ee 20171211_071240    timvideos
gcc-or1k-elf-nostdc       5.4.0_4334_g9310fdc97ee 20171211_071023    timvideos
gmp                       6.1.2                h6c8ec71_1  
hdmi2usb.modeswitch       0.0.1                     <pip>
hexfile                   0.1.1                     <pip>
idna                      2.6              py35h8605a33_1  
ipython                   6.2.1            py35hd850d2a_1  
ipython_genutils          0.2.0            py35hc9e07d0_0  
isl                       0.12.2                        0  
jedi                      0.11.0                   py35_2  
libedit                   3.1                  heed3624_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 7.2.0                h7cc24e2_2  
libstdcxx-ng              7.2.0                h7a57d05_2  
litedram                  0.1                       <pip>
liteeth                   0.1                       <pip>
litepcie                  1.0                       <pip>
litesata                  1.0                       <pip>
litescope                 0.1                       <pip>
liteusb                   0.1                       <pip>
litevideo                 0.1                       <pip>
litex                     0.1                       <pip>
mpc                       1.0.3                hec55b23_5  
mpfr                      3.1.5                h11a74b3_2  
ncurses                   6.0                  h9df7e31_2  
openocd                   0.10.0_267_gf7836bbc7   20180115_0740    timvideos
openssl                   1.0.2n               hb7f436b_0  
parso                     0.1.1            py35h1b200a3_0  
pexpect                   4.3.1                    py35_0  
pickleshare               0.7.4            py35hd57304d_0  
pip                       9.0.1            py35h7e7da9d_4  
progressbar2              3.34.3                    <pip>
prompt_toolkit            1.0.15           py35hc09de7a_0  
ptyprocess                0.5.2            py35h38ce0a3_0  
pycosat                   0.6.3            py35h6b6bb97_0  
pycparser                 2.18             py35h61b3040_1  
pygments                  2.2.0            py35h0f41973_0  
pyopenssl                 17.5.0           py35h4f8b8c8_0  
pyserial                  3.4              py35h84edd1e_0  
pysocks                   1.6.7            py35h6aefbb0_1  
python                    3.5.4               h417fded_24  
python-utils              2.2.0                     <pip>
readline                  7.0                  ha6073c6_4  
requests                  2.18.4           py35hb9e6ad1_1  
ruamel_yaml               0.11.14          py35h8e2c16b_2  
setuptools                36.5.0           py35ha8c1747_0  
simplegeneric             0.8.1            py35h2ec4104_0  
six                       1.11.0           py35h423b573_1  
sqlite                    3.20.1               hb898158_2  
tk                        8.6.7                hc745277_3  
traitlets                 4.3.2            py35ha522a97_0  
urllib3                   1.22             py35h2ab6e29_0  
wcwidth                   0.1.7            py35hcd08066_0  
wheel                     0.30.0           py35hd3883cf_1  
xz                        5.2.3                h55aa19d_2  
yaml                      0.1.7                had09818_2  
zlib                      1.2.11               ha838bed_2  
(LX P=arty C=or1k.linux F=linux R=in-tftpd-fix) ewen@parthenon:/src/fpga-minicon
f/litex-buildenv$ 

all of which are notably the py35 versions rather than the py36 versions…

Ewen