Build does not use proper Python path (Following instructions in DEVELOPMENT.md)

This may be an issue with bazel (2.28.1, the current version), but either way it’s a problem for tensorboard build that should at least be documented. I tried the installing 2.23.2, the version found in package.json, but it was too old and another package required a newer version.

I’m trying to build Tensorboard locally using the instructions found in DEVELOPMENT.md.

When I run bazel build tensorboard, I get an error that it’s trying to use my system Python2.7 (/usr/bin/python) instead of my virtual environment.

In the error, it says it’s using this $PATH: /usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.

Here’s what it should be:

$ echo $PATH
/anaconda3/bin:/anaconda3/condabin:/anaconda3/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/bsteers/bin

# and other info
$ which python
/Users/bsteers/Documents/GitHub/tensorboard/tf/bin/python
$ python --version
Python 3.7.3
$ bazel version
Build label: 0.28.1
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Jul 19 15:22:50 2019 (1563549770)
Build timestamp: 1563549770
Build timestamp as int: 1563549770

I found out from here that if I run this I get the same path so it seems like it’s clearing the environmental variables or something?

$ exec env - /bin/bash -c 'echo $PATH'
/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.

I tried adding build --action_env=PATH to my ~/.bazelrc with no luck.

Environment information (required)

Please run diagnose_tensorboard.py (link below) in the same environment from which you normally run TensorFlow/TensorBoard, and paste the output here:

Diagnostics

Diagnostics output

--- check: autoidentify
INFO: diagnose_tensorboard.py version 393931f9685bd7e0f3898d7dcdf28819fef54c43

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Darwin', nodename='10-18-151-131.dynapool.wireless.nyu.edu', release='18.0.0', version='Darwin Kernel Version 18.0.0: Wed Aug 22 20:13:40 PDT 2018; root:xnu-4903.201.2~1/RELEASE_X86_64', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: '/Users/bsteers/Documents/GitHub/tensorboard/tf'

--- check: installed_packages
INFO: installed: tb-nightly==1.15.0a20190806
INFO: installed: tf-nightly==1.15.0.dev20190809
INFO: installed: tf-estimator-nightly==1.14.0.dev2019080901

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '1.15.0a20190806'

--- check: tensorflow_python_version
WARNING: 

  TensorFlow's `tf-nightly` package will soon be updated to TensorFlow 2.0.

  Please upgrade your code to TensorFlow 2.0:
    * https://www.tensorflow.org/beta/guide/migration_guide

  Or install the latest stable TensorFlow 1.X release:
    * `pip install -U "tensorflow==1.*"`

  Otherwise your code may be broken by the change.

  
INFO: tensorflow.__version__: '1.15.0-dev20190809'
INFO: tensorflow.__git_version__: 'v1.12.1-8422-g0d1a1413c8'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/Users/bsteers/Documents/GitHub/tensorboard/tf/bin/tensorboard\n'

--- check: readable_fqdn
INFO: socket.getfqdn(): '10-18-151-131.dynapool.wireless.nyu.edu'

--- check: stat_tensorboardinfo
INFO: directory: /var/folders/nc/30stjm_n07gcpz6djh26cf8m0000gn/T/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=31416359, st_dev=16777221, st_nlink=2, st_uid=501, st_gid=20, st_size=64, st_atime=1564953516, st_mtime=1565356530, st_ctime=1565356530)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/Users/bsteers/Documents/GitHub/tensorboard/tf/lib/python3.7/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.7.1
astor==0.8.0
gast==0.2.2
google-pasta==0.1.7
grpcio==1.22.0
h5py==2.9.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
Markdown==3.1.1
numpy==1.17.0
opt-einsum==2.3.2
pip==19.2.1
protobuf==3.9.1
setuptools==41.0.1
six==1.12.0
tb-nightly==1.15.0a20190806
termcolor==1.1.0
tf-estimator-nightly==1.14.0.dev2019080901
tf-nightly==1.15.0.dev20190809
Werkzeug==0.15.5
wheel==0.33.4
wrapt==1.11.2

Next steps

No action items identified. Please copy ALL of the above output, including the lines containing only backticks, into your GitHub issue or comment. Be sure to redact any sensitive information.

Steps to reproduce (required)

Please provide explicit commands to reproduce the problem in a fresh environment (virtualenv, Conda environment, …). Include any necessary configuration, such as Conda environment.yml files.

First, I installed bazel:

brew tap bazelbuild/tap
brew install bazelbuild/tap/bazel

Then I setup a virtualenv just as explained (I tried conda as well with no luck)

virtualenv tf
source tf/bin/activate
pip install --upgrade pip
pip install tf-nightly

Then to setup and run bazel I did:

cd tensorboard # the repo
bazel build tensorboard

ERROR: /Users/bsteers/Documents/GitHub/tensorboard/tensorboard/plugins/profile/pod_viewer/pod_viewer_common/BUILD:7:1: Compiling 4 TypeScript files //tensorboard/plugins/profile/pod_viewer/pod_viewer_common:pod_viewer_common failed (Exit 1) execrooter failed: error executing command bazel-out/host/bin/tensorboard/scripts/execrooter bazel-out/darwin-fastbuild/bin/tensorboard/plugins/profile/pod_viewer/pod_viewer_common/pod_viewer_common-tsc-execroot.json

Use --sandbox_debug to see verbose messages from the sandbox
Error occurred while attempting to use the default Python toolchain (@bazel_tools//tools/python:autodetecting_toolchain).
According to '/usr/bin/python -V', version is 'Python 2.7.10', but we need version 3. PATH is:

/usr/gnu/bin:/usr/local/bin:/bin:/usr/bin:.

[wchargin]

Understood. In the style of docker I suppose.

Yep, exactly.

Would adding a virtualenv to the build work?

Not sure whether you mean “creating a virtualenv as part of the Bazel build” or “referring to an existing virtualenv from the Bazel build”. The former is possible, and is kind of what the rules_python rules do. But for that approach you have to explicitly specify all the packages and versions (by design), so when I want to test against

• today’s tf-nightly,
• today’s tf-nightly-2.0-preview,
• today’s tf-nightly-gpu-2.0-preview,
• stable tensorflow==1.14,
• tf-nightly from a week ago because something changed in the last week and I’m trying to bisect it, and
• no TensorFlow at all

then in the simplest approach I’d need to manually patch the build file for each different environment. Also, we’d need some kind of bot to bump the tf-nightly version every day.

It’s possible that we could rig up something more principled by using Bazel’s config_setting and select functionality, but that would be more involved, and we just haven’t looked into it yet.

Honestly, I’m not sure how it’s building using the system python 2 when tf-nightly was only installed in the virtualenv

With the autodetecting toolchain, the “Python interpreter” is actually a wrapper provided by Bazel, defined in pywrapper_template.txt and expanded in tools/python/toolchain.bzl. It just shells out to which; as long as your virtualenv is active, that will resolve to the virtualenv’s Python.

I think it would be important to note all of this in DEVELOPMENT.md though.

Agreed, and thank you for writing this up! Instead of asking developers to write their own personal .bazelrcs, I’ve gone ahead and added a workspace-level .bazelrc to TensorBoard itself, so you should be able to pull in latest master as of #2535 and build without any additional configuration (i.e., you can remove your user-level .bazelrc if you like).

Anyways, thanks for the help!

Sure thing.

[beasteers]

Understood. In the style of docker I suppose. I understand clearing $PATH in that respect. Would adding a virtualenv to the build work? Honestly, I’m not sure how it’s building using the system python 2 when tf-nightly was only installed in the virtualenv 🤷‍♀️

I think it would be important to note all of this in DEVELOPMENT.md though. i.e.

---

You have to install Bazel==0.26.1 in order for the build to run successfully.

Due to changes in Bazel, in order to get versions >=0.27 working you will have to add a flag to the build command. This will work temporarily, but at some point Bazel may remove the flag.

bazel run //tensorboard \
    --incompatible_use_python_toolchains=false \
    -- --logdir=/path/to/logs

# or alternatively
bazel run //tensorboard \
    --extra_toolchains=@bazel_tools//tools/python:autodetecting_toolchain_nonstrict \
    -- --logdir=/path/to/logs

This must be run for every build call. For convenience, it can be added to your ~/.bazelrc file (or equivalent, see docs) which will add it as a flag automatically.

---

Anyways, thanks for the help!