question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Could not sync with existing tensorboard event files

See original GitHub issue

wandb --version && python --version && uname

  • Weights and Biases version: 0.8.36
  • Python version: Python 3.7.6
  • Operating System: MSYS_NT-10.0-18362

Description

Describe what you were trying to get done. Tell us what happened, what went wrong, and what you expected to happen.

I am using Anaconda to manage my Python environment and I installed required dependencies for wandb and also pytorch >1.5.0 and tensorboard = 2.2.1. I followed the instructions to log in and sync using the command line interface, but first it requires me to install tensorflow (according to the documentation it should not be a hard requirement) which I did, but then when I tried to sync, it returns an internal error.

What I Did

wandb sync log_dir

wandb: Syncing log_dir to:
wandb: revived-sun-1 https://app.wandb.ai/username/project_name/runs/2cswv2su
wandb: Found tfevents file, converting...
2020-05-21 00:05:17.302868: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2020-05-21 00:05:17.310526: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
WARNING:tensorflow:From h:\anaconda3\envs\conda_env_name\lib\site-packages\tensorflow\python\summary\summary_iterator.py:68: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
Traceback (most recent call last):
  File "h:\anaconda3\envs\conda_env_name\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "h:\anaconda3\envs\conda_env_name\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "H:\Anaconda3\envs\conda_env_name\Scripts\wandb.exe\__main__.py", line 7, in <module>
  File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "h:\anaconda3\envs\conda_env_name\lib\site-packages\click\decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "h:\anaconda3\envs\conda_env_name\lib\site-packages\wandb\cli.py", line 154, in wrapper
    return func(*args, **kwargs)
  File "h:\anaconda3\envs\conda_env_name\lib\site-packages\wandb\cli.py", line 447, in sync
    path, run_id=id, project=project, entity=entity, ignore_globs=globs)
  File "h:\anaconda3\envs\conda_env_name\lib\site-packages\wandb\wandb_run.py", line 339, in from_directory
    summary.update(wbtf.stream_tfevents(path, file_api, run, namespace=namespace))
  File "h:\anaconda3\envs\conda_env_name\lib\site-packages\wandb\tensorflow\__init__.py", line 55, in stream_tfevents
    parsed = tf_summary_to_dict(summary, namespace=namespace)
  File "h:\anaconda3\envs\conda_env_name\lib\site-packages\wandb\tensorboard\__init__.py", line 339, in tf_summary_to_dict
    if not wandb.run.config.get(key):
AttributeError: 'NoneType' object has no attribute 'config'

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:6 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
cvphelpscommented, May 27, 2020

Hi there, thanks for letting us know about this issue. We’d like to fix syncing problems in the new version of our client, and I’ll check with my colleague @raubitsj to see if this case is supported.

We’re working on a new beta of the Weights & Biases client that improves support for environments like Windows. Would you be interested in giving it a try?

  1. Join the Slack forum and go to the wandb-cling channel.
  2. Install CLI-NG following directions from the GitHub repo.
  3. Try tracking a model with the new client, no code changes necessary.

Message us in the wandb-cling channel— we’d love to hear any questions or feedback.

0reactions
aj7teshcommented, Feb 15, 2021

"“Could not sync with existing tensorboard event files wandb/client#1051"” https://github.com/wandb/client-ng/issues/23#issuecomment-651258932 Above feature is not yet there,right ?

Read more comments on GitHub >

github_iconTop Results From Across the Web

Fails to show new data when event files are replaced (e.g. w ...
Problem summary We have a problem that tensorboard (1.2.1) does not show new run data when we switched to syncing the data from...
Read more >
TensorBoard - Documentation - Weights & Biases - Wandb
Syncing Previous TensorBoard Runs​​ If you have existing tfevents files stored locally and you would like to import them into W&B, you can...
Read more >
tensorboard can't find event files - Stack Overflow
Say your data is at 'X:\X\file.x' Go in command line to X:\ first. Then type: tensorboard --logdir=X/ NOT tensorboard --logdir='.X/'.
Read more >
Use Arena to submit distributed TensorFlow training jobs
This topic describes how to use the Arena client to submit distributed TensorFlow training jobs that run on the parameter server (PS)-worker ...
Read more >
What's new in TensorBoard (TF Dev Summit '19) - YouTube
TensorBoard provides the visualization needed for machine learning experimentation. This talk will cover some exciting new functionality on ...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found