question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Initializing client with sync_tensorboard=True crashes

See original GitHub issue

wandb --version && python --version && uname

  • Weights and Biases version: 0.8.18
  • Python version: Python 3.5.3
  • Operating System: Linux

Description

Trying to syncronize tensorboard to weights and biases client, and the program keeps crashing with the following log:

    wandb.init(sync_tensorboard=True)
  File "/home/giacaglia/tacotron2/venv/lib/python3.5/site-packages/wandb/__init__.py", line 889, in init
    util.get_module("wandb.tensorboard").patch()
AttributeError: 'NoneType' object has no attribute 'patch'

The following are the libraries installed in our machine:

absl-py==0.8.1
anykeystore==0.2
apex==0.1
argh==0.26.2
astor==0.8.0
audioread==2.1.8
cachetools==3.1.1
certifi==2019.11.28
chardet==3.0.4
Click==7.0
configparser==4.0.2
cryptacular==1.5.5
cycler==0.10.0
decorator==4.4.1
defusedxml==0.6.0
docker-pycreds==0.4.0
gast==0.2.2
gitdb2==2.0.6
GitPython==3.0.5
google-auth==1.7.1
google-auth-oauthlib==0.4.1
google-pasta==0.1.8
gql==0.1.0
graphql-core==2.2.1
grpcio==1.25.0
h5py==2.10.0
hupper==1.9.1
idna==2.8
inflect==0.2.5
joblib==0.14.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
librosa==0.6.0
llvmlite==0.30.0
Markdown==3.1.1
matplotlib==2.1.0
numba==0.46.0
numpy==1.13.3
nvidia-ml-py3==7.352.0
oauthlib==3.1.0
opt-einsum==3.1.0
PasteDeploy==2.0.1
pathtools==0.1.2
pbkdf2==1.3
Pillow==6.2.1
plaster==1.0
plaster-pastedeploy==0.7
promise==2.2.1
protobuf==3.10.0
psutil==5.6.7
pyasn1==0.4.7
pyasn1-modules==0.2.7
pyparsing==2.4.5
pyramid==1.10.4
pyramid-mailer==0.15.1
python-dateutil==2.8.1
python3-openid==3.1.0
pytz==2019.3
PyYAML==5.2
repoze.sendmail==4.4.1
requests==2.22.0
requests-oauthlib==1.3.0
resampy==0.2.2
rsa==4.0
Rx==1.6.1
scikit-learn==0.21.3
scipy==1.0.0
sentry-sdk==0.13.5
shortuuid==0.5.0
six==1.13.0
smmap2==2.0.5
SQLAlchemy==1.3.11
subprocess32==3.5.4
tensorboard==1.10.0
tensorboardX==1.1
tensorflow-estimator==1.14.0
tensorflow-gpu==1.10.0
termcolor==1.1.0
torch==1.0.0
transaction==2.4.0
translationstring==1.3
Unidecode==1.0.22
urllib3==1.25.7
velruse==1.1.1
venusian==3.0.0
wandb==0.8.18
watchdog==0.9.0
WebOb==1.8.5
Werkzeug==0.16.0
wrapt==1.11.2
WTForms==2.2.1
wtforms-recaptcha==0.3.2
zope.deprecation==4.4.0
zope.interface==4.7.1
zope.sqlalchemy==1.2

What I Did

Just running a simple training of our neural network. When running wandb.init(project=‘project_name’) it works, but not with sync_tensorboard=True

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:5 (3 by maintainers)

github_iconTop GitHub Comments

2reactions
vanpeltcommented, Dec 20, 2020

This is fixed in the 0.10.x series of our library.

1reaction
EricCousineau-TRIcommented, Jun 12, 2020

It looks like this was triggered by the change in 72b8ef982d2, using util.get_module("wandb.tensorboard"), which very aggressively eats import errors: https://github.com/wandb/client/blob/4c46d8d093eb3a52748fb28bb1d7386efd9375e3/wandb/util.py#L122-L126

However, probably most relevant, @giacaglia, is I saw tensorboardX==1.1 in your freeze output. I can reproduce your error with tensorboardX==1.4, but not tensorboardX==1.5 (that was in my original repro, but I have now edited it).

So it looks like this is generally resolved by upgrading to tensorboardX>=1.5, which is now explicitly called via merge of #1098

I think the solution is to change the relevant line to util.get_module("wandb.tensorboard", required=True), so that it fails fast and shows the true error.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Initializing client with sync_tensorboard=True crashes #709
I think the solution is to change the relevant line to util.get_module("wandb.tensorboard", required=True) , so that it fails fast and shows the ...
Read more >
Launch Experiments with wandb.init - Documentation
I have to run TensorBoard locally on a master machine and sync logs between computers to visualize real-time results. With wandb, this is...
Read more >
Troubleshoot Dataflow errors - Google Cloud
If you run into problems with your Dataflow pipeline or job, this page lists error messages that you might see and provides suggestions...
Read more >
Configuring Ray — Ray 2.2.0 - the Ray documentation
This page discusses the various way to configure Ray, both from the Python API and from the command line. Take a look at...
Read more >
tf.io.gfile.GFile | TensorFlow v2.11.0
This means that the data would survive an application crash but not ... Returns True as FileIO supports random access ops of seek()/tell() ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found