wandb.watch(model): unwatch and rewatch models with new parameters, using Hugging Face

wandb --version && python --version && uname

wandb, version 0.9.3 Python 3.6.8 :: Anaconda, Inc. Darwin

Description

I was trying to unwatch models and rewatch them with new parameters. I am using the huggingface library which uses wandb.

What I Did

wandb.unwatch()
self._setup_wandb()

self._setup_wandb() is defined basically as:

wandb.watch(
                   self.model, log=os.getenv("WANDB_WATCH", "gradients"), log_freq=max(100, self.args.logging_steps)
               )

Now, looking into unwatch/unhook in the wandb code:

 def unhook_all(self):
       for handle in self._hook_handles.values():
           handle.remove()
       self._hook_handles = []


   def unhook(self, name):
       handle = self._hook_handles.pop(name)
       handle.remove()

This is wrong, it seems that at some point self._hook_handles was changed to a dictionary, so unhook/unhook all should be something like:

 def unhook_all(self):
       for handle in self._hook_handles.values():
           handle.remove()
       self._hook_handles = {}


   def unhook(self, name):
       handle = self._hook_handles[name]
       handle.remove()

I havent tried this yet but I’ll get back to it if I do.

[borisdayma]

I can confirm issues with wandb.unwatch method. Maybe we should wait for the new CLI?

[cvphelps]

Hi Pedro, thanks for diving in to the wandb.watch() code, @borisdayma could you take a look?

As for un-watching and re-watching a model, it sounds like you’re editing the model during training, is that right? I’d recommend calling wandb.init() to start a new run each time you want to watch a model with new parameters