Integration: Project renaming breaks TensorBoard loading

Describe the bug I am using TF 2.3.1 and wandb 0.10.10., and Safari.

I log the data via the TensorBoard callback for keras, and set sync_tensorboard to True. With this settings, everything worked fine the last days. I did not change anything. As of today, I can not visualize the logged data in TB, this is also for some older logs I did (weeks ago). It says no data, but I can recall using this visualization on this exact run with success.

On my two runs I did yesterday and this morning, I also log data, but the TensorBoard screen stays empty. I can’t seem to reproduce this behavior, for the following toy example it works fine:

from tensorflow import keras
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import itertools, io, sklearn.metrics, os

!pip install wandb

import wandb
wandb.login()

def do_stuff():
  import tensorflow as tf
  from tensorflow.keras.layers import Input, Dense, Reshape
  from tensorflow.keras import Sequential

  model = Sequential([
    Input(shape=(128)), 
    Dense(20, name="test1"),           
    Dense(3)
  ])

  wandb.init(sync_tensorboard=True)
  wand_cb = wandb.keras.WandbCallback('val_sparse_categorical_crossentropy')

  inp = tf.random.uniform([128, 128], dtype=tf.float32)
  labels = tf.random.uniform([128], 0, 3, dtype=tf.int32)
  labels = np.random.randint(0, 3, size=(128))
  model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=["sparse_categorical_accuracy"])
  #model.summary()

  tb = tf.keras.callbacks.TensorBoard(log_dir="test")


  model.fit(inp, labels, epochs=10, callbacks=[tb, wand_cb])
  print(wandb.run.name)
  wandb.finish()

I noticed that I exceed the storage limit by 8 GBs, can this infer? (Also suggesting to send users a notification if quota is reached).

[Yannik1337]

Update: Have it confirmed, renaming breaks the TensorBoard. Run below snippet in Colab once:

from tensorflow import keras
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import itertools, io, sklearn.metrics, os

!pip install wandb

import wandb
wandb.login()

def do_stuff():
  import tensorflow as tf
  from tensorflow.keras.layers import Input, Dense, Reshape
  from tensorflow.keras import Sequential

  model = Sequential([
    Input(shape=(128)), 
    Dense(20, name="test1"),           
    Dense(3)
  ])

  wandb.init(sync_tensorboard=True, project="test_a")
  wand_cb = wandb.keras.WandbCallback('val_sparse_categorical_crossentropy')

  inp = tf.random.uniform([128, 128], dtype=tf.float32)
  labels = tf.random.uniform([128], 0, 3, dtype=tf.int32)
  model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=["sparse_categorical_accuracy"])
  #model.summary()

  tb = tf.keras.callbacks.TensorBoard(log_dir="test")


  model.fit(inp, labels, epochs=10, callbacks=[tb, wand_cb])
  print(wandb.run.name)
  wandb.finish()

do_stuff()

Afterwards, rename project “test_a” to e.g. “test_b” in the GUI. Now, check the TensorBoard feature for this run again in the GUI. It does not work anymore.

As a second step, re-run the code snippet again, changing the project in init to “test_b”. This visualization will fail too, albeit it was logged to the already renamed project. This seems like an issue on the side of data handling, not on the client side.

[sydholl]

Hi @Yannik1337, this was fixed early this year. Please let us know if you’re still experiencing similar issues.