Initializing client with sync_tensorboard=True crashes
See original GitHub issuewandb --version && python --version && uname
- Weights and Biases version: 0.8.18
- Python version: Python 3.5.3
- Operating System: Linux
Description
Trying to syncronize tensorboard to weights and biases client, and the program keeps crashing with the following log:
wandb.init(sync_tensorboard=True)
File "/home/giacaglia/tacotron2/venv/lib/python3.5/site-packages/wandb/__init__.py", line 889, in init
util.get_module("wandb.tensorboard").patch()
AttributeError: 'NoneType' object has no attribute 'patch'
The following are the libraries installed in our machine:
absl-py==0.8.1
anykeystore==0.2
apex==0.1
argh==0.26.2
astor==0.8.0
audioread==2.1.8
cachetools==3.1.1
certifi==2019.11.28
chardet==3.0.4
Click==7.0
configparser==4.0.2
cryptacular==1.5.5
cycler==0.10.0
decorator==4.4.1
defusedxml==0.6.0
docker-pycreds==0.4.0
gast==0.2.2
gitdb2==2.0.6
GitPython==3.0.5
google-auth==1.7.1
google-auth-oauthlib==0.4.1
google-pasta==0.1.8
gql==0.1.0
graphql-core==2.2.1
grpcio==1.25.0
h5py==2.10.0
hupper==1.9.1
idna==2.8
inflect==0.2.5
joblib==0.14.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
librosa==0.6.0
llvmlite==0.30.0
Markdown==3.1.1
matplotlib==2.1.0
numba==0.46.0
numpy==1.13.3
nvidia-ml-py3==7.352.0
oauthlib==3.1.0
opt-einsum==3.1.0
PasteDeploy==2.0.1
pathtools==0.1.2
pbkdf2==1.3
Pillow==6.2.1
plaster==1.0
plaster-pastedeploy==0.7
promise==2.2.1
protobuf==3.10.0
psutil==5.6.7
pyasn1==0.4.7
pyasn1-modules==0.2.7
pyparsing==2.4.5
pyramid==1.10.4
pyramid-mailer==0.15.1
python-dateutil==2.8.1
python3-openid==3.1.0
pytz==2019.3
PyYAML==5.2
repoze.sendmail==4.4.1
requests==2.22.0
requests-oauthlib==1.3.0
resampy==0.2.2
rsa==4.0
Rx==1.6.1
scikit-learn==0.21.3
scipy==1.0.0
sentry-sdk==0.13.5
shortuuid==0.5.0
six==1.13.0
smmap2==2.0.5
SQLAlchemy==1.3.11
subprocess32==3.5.4
tensorboard==1.10.0
tensorboardX==1.1
tensorflow-estimator==1.14.0
tensorflow-gpu==1.10.0
termcolor==1.1.0
torch==1.0.0
transaction==2.4.0
translationstring==1.3
Unidecode==1.0.22
urllib3==1.25.7
velruse==1.1.1
venusian==3.0.0
wandb==0.8.18
watchdog==0.9.0
WebOb==1.8.5
Werkzeug==0.16.0
wrapt==1.11.2
WTForms==2.2.1
wtforms-recaptcha==0.3.2
zope.deprecation==4.4.0
zope.interface==4.7.1
zope.sqlalchemy==1.2
What I Did
Just running a simple training of our neural network. When running wandb.init(project=‘project_name’) it works, but not with sync_tensorboard=True
Issue Analytics
- State:
- Created 4 years ago
- Comments:5 (3 by maintainers)
Top Results From Across the Web
Initializing client with sync_tensorboard=True crashes #709
I think the solution is to change the relevant line to util.get_module("wandb.tensorboard", required=True) , so that it fails fast and shows the ...
Read more >Launch Experiments with wandb.init - Documentation
I have to run TensorBoard locally on a master machine and sync logs between computers to visualize real-time results. With wandb, this is...
Read more >Troubleshoot Dataflow errors - Google Cloud
If you run into problems with your Dataflow pipeline or job, this page lists error messages that you might see and provides suggestions...
Read more >Configuring Ray — Ray 2.2.0 - the Ray documentation
This page discusses the various way to configure Ray, both from the Python API and from the command line. Take a look at...
Read more >tf.io.gfile.GFile | TensorFlow v2.11.0
This means that the data would survive an application crash but not ... Returns True as FileIO supports random access ops of seek()/tell() ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
This is fixed in the 0.10.x series of our library.
It looks like this was triggered by the change in 72b8ef982d2, using
util.get_module("wandb.tensorboard")
, which very aggressively eats import errors: https://github.com/wandb/client/blob/4c46d8d093eb3a52748fb28bb1d7386efd9375e3/wandb/util.py#L122-L126However, probably most relevant, @giacaglia, is I saw
tensorboardX==1.1
in your freeze output. I can reproduce your error withtensorboardX==1.4
, but nottensorboardX==1.5
(that was in my original repro, but I have now edited it).So it looks like this is generally resolved by upgrading to
tensorboardX>=1.5
, which is now explicitly called via merge of #1098I think the solution is to change the relevant line to
util.get_module("wandb.tensorboard", required=True)
, so that it fails fast and shows the true error.