Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
No train metrics show up in scalar plots
See original GitHub issueConsider Stack Overflow for getting support using TensorBoard—they have a larger community with better searchability:
https://stackoverflow.com/questions/tagged/tensorboard
Do not use this template for for setup, installation, or configuration issues. Instead, use the “installation problem” issue template:
https://github.com/tensorflow/tensorboard/issues/new?template=installation_problem.md
To report a problem with TensorBoard itself, please fill out the remainder of this template.
Environment information (required)
Please run diagnose_tensorboard.py (link below) in the same
environment from which you normally run TensorFlow/TensorBoard, and
paste the output here:
Diagnostics
Diagnostics output
--- check: autoidentify
INFO: diagnose_tensorboard.py version 4725c70c7ed724e2d1b9ba5618d7c30b957ee8a4
--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=6, micro=8, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='master1', release='4.15.0-66-generic', version='#75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019', machine='x86_64')
INFO: sys.getwindowsversion(): N/A
--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: None
--- check: installed_packages
INFO: installed: tensorboard==2.0.1
INFO: installed: tensorflow==2.0.0
INFO: installed: tensorflow-estimator==2.0.1
--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.0.1'
--- check: tensorflow_python_version
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.25.3) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
INFO: tensorflow.__version__: '2.0.0'
INFO: tensorflow.__git_version__: 'v2.0.0-rc2-26-g64c3d38'
--- check: tensorboard_binary_path
INFO: which tensorboard: b'/home/bz/.local/bin/tensorboard\n'
--- check: readable_fqdn
INFO: socket.getfqdn(): 'master1.bz'
--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=14562209, st_dev=2430, st_nlink=2, st_uid=1000, st_gid=1000, st_size=4096, st_atime=1572563590, st_mtime=1573247067, st_ctime=1573247
067)
INFO: mode: 0o40777
--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/home/bz/.local/lib/python3.6/site-packages']; bad_roots (0): []
--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.8.1
asn1crypto==0.24.0
astor==0.8.0
attrs==17.4.0
Automat==0.6.0
bleach==2.1.2
cachetools==3.1.1
certifi==2018.1.18
chardet==3.0.4
click==6.7
colorama==0.3.7
command-not-found==0.3
configobj==5.0.6
constantly==15.1.0
cryptography==2.1.4
decorator==4.1.2
distro-info===0.18ubuntu0.18.04.1
entrypoints==0.2.3.post1
eventkit==0.8.5
gast==0.2.2
google-auth==1.7.0
google-auth-oauthlib==0.4.1
google-pasta==0.1.8
grpcio==1.25.0
h5py==2.10.0
html5lib==0.999999999
httplib2==0.9.2
hyperlink==17.3.1
ib-insync==0.9.53
idna==2.6
incremental==16.10.1
ipykernel==4.8.2
ipython==5.5.0
ipython-genutils==0.2.0
ipywidgets==6.0.0
Jinja2==2.10
joblib==0.14.0
jsonschema==2.6.0
jupyter-client==5.2.2
jupyter-console==5.2.0
jupyter-core==4.4.0
Keras==2.3.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
keyring==10.6.0
keyrings.alt==3.0
language-selector==0.1
Markdown==3.1.1
MarkupSafe==1.0
mistune==0.8.3
nbconvert==5.3.1
nbformat==4.4.0
nest-asyncio==1.0.0
netifaces==0.10.4
notebook==5.2.2
numpy==1.17.3
oauthlib==3.1.0
opt-einsum==3.1.0
PAM==0.4.2
pandas==0.24.2
pandocfilters==1.4.2
pexpect==4.2.1
pickleshare==0.7.4
pip==19.3.1
prompt-toolkit==1.0.15
protobuf==3.10.0
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycrypto==2.6.1
Pygments==2.2.0
pygobject==3.26.1
pyOpenSSL==17.5.0
pyserial==3.4
python-apt==1.6.4
python-dateutil==2.8.0
python-debian==0.1.32
pytz==2019.1
pyxdg==0.25
PyYAML==3.12
pyzmq==16.0.2
requests==2.18.4
requests-oauthlib==1.3.0
requests-unixsocket==0.1.5
rsa==4.0
scikit-learn==0.21.3
scipy==1.3.1
SecretStorage==2.3.1
selenium==3.141.0
service-identity==16.0.0
setuptools==41.6.0
simplegeneric==0.8.1
six==1.13.0
sklearn==0.0
ssh-import-id==5.7
systemd-python==234
tdameritrade==0.0.7
tensorboard==2.0.1
tensorflow==2.0.0
tensorflow-estimator==2.0.1
termcolor==1.1.0
terminado==0.7
testpath==0.3.1
tornado==4.5.3
tqdm==4.32.2
traitlets==4.3.2
Twisted==17.9.0
ufw==0.36
unattended-upgrades==0.1
urllib3==1.25.3
vboxapi==1.0
wcwidth==0.1.7
webencodings==0.5
Werkzeug==0.16.0
wheel==0.33.6
wrapt==1.11.2
zope.interface==4.3.2
For browser-related issues, please additionally specify:
- Browser type and version (e.g., Chrome 64.0.3282.140):
- Screenshot, if it’s a visual issue:
Issue description
I just upgraded tensorflow to 2.0. In training, I noticed tensorboard now has two runs for each experiment, including train and validation. However, only validation has scalar value curves. Train metric plots are always empty.
I can reproduce this issue by using the script in tensorboard get started guide: https://www.tensorflow.org/tensorboard/get_started. The script prints out reasonable train and val metrics as it should, but I’m just not getting the right plots.
Issue Analytics
- State:
- Created 4 years ago
- Reactions:8
- Comments:10 (5 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
I tried this a few more times. It looks like tensorboard 2.0 has trouble updating the train metrics by itself. If I kill tensorboard and restart it, it will then show both train and validation metrics. If the training is still ongoing, the validation metrics will be updated where as the train metrics are stuck.
Adding profile_batch=0 to the keras callback resolves it.