Could not sync with existing tensorboard event files
See original GitHub issuewandb --version && python --version && uname
- Weights and Biases version: 0.8.36
- Python version: Python 3.7.6
- Operating System: MSYS_NT-10.0-18362
Description
Describe what you were trying to get done. Tell us what happened, what went wrong, and what you expected to happen.
I am using Anaconda to manage my Python environment and I installed required dependencies for wandb and also pytorch >1.5.0 and tensorboard = 2.2.1. I followed the instructions to log in and sync using the command line interface, but first it requires me to install tensorflow (according to the documentation it should not be a hard requirement) which I did, but then when I tried to sync, it returns an internal error.
What I Did
wandb sync log_dir
wandb: Syncing log_dir to:
wandb: revived-sun-1 https://app.wandb.ai/username/project_name/runs/2cswv2su
wandb: Found tfevents file, converting...
2020-05-21 00:05:17.302868: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-05-21 00:05:17.310526: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
WARNING:tensorflow:From h:\anaconda3\envs\conda_env_name\lib\site-packages\tensorflow\python\summary\summary_iterator.py:68: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
Traceback (most recent call last):
File "h:\anaconda3\envs\conda_env_name\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "h:\anaconda3\envs\conda_env_name\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "H:\Anaconda3\envs\conda_env_name\Scripts\wandb.exe\__main__.py", line 7, in <module>
File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "h:\anaconda3\envs\conda_env_name\lib\site-packages\wandb\cli.py", line 154, in wrapper
return func(*args, **kwargs)
File "h:\anaconda3\envs\conda_env_name\lib\site-packages\wandb\cli.py", line 447, in sync
path, run_id=id, project=project, entity=entity, ignore_globs=globs)
File "h:\anaconda3\envs\conda_env_name\lib\site-packages\wandb\wandb_run.py", line 339, in from_directory
summary.update(wbtf.stream_tfevents(path, file_api, run, namespace=namespace))
File "h:\anaconda3\envs\conda_env_name\lib\site-packages\wandb\tensorflow\__init__.py", line 55, in stream_tfevents
parsed = tf_summary_to_dict(summary, namespace=namespace)
File "h:\anaconda3\envs\conda_env_name\lib\site-packages\wandb\tensorboard\__init__.py", line 339, in tf_summary_to_dict
if not wandb.run.config.get(key):
AttributeError: 'NoneType' object has no attribute 'config'
Issue Analytics
- State:
- Created 3 years ago
- Comments:6 (2 by maintainers)
Top Results From Across the Web
Fails to show new data when event files are replaced (e.g. w ...
Problem summary We have a problem that tensorboard (1.2.1) does not show new run data when we switched to syncing the data from...
Read more >TensorBoard - Documentation - Weights & Biases - Wandb
Syncing Previous TensorBoard Runs If you have existing tfevents files stored locally and you would like to import them into W&B, you can...
Read more >tensorboard can't find event files - Stack Overflow
Say your data is at 'X:\X\file.x' Go in command line to X:\ first. Then type: tensorboard --logdir=X/ NOT tensorboard --logdir='.X/'.
Read more >Use Arena to submit distributed TensorFlow training jobs
This topic describes how to use the Arena client to submit distributed TensorFlow training jobs that run on the parameter server (PS)-worker ...
Read more >What's new in TensorBoard (TF Dev Summit '19) - YouTube
TensorBoard provides the visualization needed for machine learning experimentation. This talk will cover some exciting new functionality on ...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Hi there, thanks for letting us know about this issue. We’d like to fix syncing problems in the new version of our client, and I’ll check with my colleague @raubitsj to see if this case is supported.
We’re working on a new beta of the Weights & Biases client that improves support for environments like Windows. Would you be interested in giving it a try?
Message us in the wandb-cling channel— we’d love to hear any questions or feedback.
"“Could not sync with existing tensorboard event files wandb/client#1051"” https://github.com/wandb/client-ng/issues/23#issuecomment-651258932 Above feature is not yet there,right ?