TensorBoard: gracefully handle deleted event files

as far as i can tell this is exactly the same as this issue https://github.com/tensorflow/tensorflow/issues/3267

if i delete files from the logdir while tensorboard is running i get things like

E0911 11:27:19.441699 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'
E0911 11:27:24.447706 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'
E0911 11:27:29.453577 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'
E0911 11:27:34.459517 139989077399296 plugin_event_multiplexer.py:226] Unable to reload accumulator 'srresnet_voc_2x': [Errno 2] No such file or directory: b'/home/maksim/data/tensorboard/srresnet_voc_2x/events.out.tfevents.1568215513.maksim-desktop.105092.0'

i’m using tb-nightly==1.15.0a20190911 through pytorch.

i’m not sure when reaping is supposed to happen e.g. as in https://github.com/tensorflow/tensorflow/issues/3267#issuecomment-292911229 or how to manually force

WARNING:tensorflow:Deleting accumulator 'run1/test'
WARNING:tensorflow:Deleting accumulator 'run1'
WARNING:tensorflow:Deleting accumulator 'run2/test'
WARNING:tensorflow:Deleting accumulator 'run2'

[thincal]

@stephanwlee one important thing forget to mention, I just run the tensorboard without tensorflow installed, you could try it again.

[thincal]

@stephanwlee

Raised exception causes the application “Reloader” thread exited, so that the graph won’t be updated anymore.

tensorflow.python.framework.errors_impl.NotFoundError: Could not find directory xxxx
tensorboard.backend.event_processing.directory_watcher.DirectoryDeletedError: Directory xxxx deleted

This seems like a bug, what I expect behavior would be:

• gracefully handle this DirectoryDeletedError exception without crash
• remove the graph of deleted run from the web

FYI, full callstack:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/abc/Library/Python/3.7/lib/python/site-packages/tensorboard/backend/application.py", line 502, in _reload
    multiplexer.AddRunsFromDirectory(path, name)
  File "/Users/abc/Library/Python/3.7/lib/python/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 193, in AddRunsFromDirectory
    self.AddRun(subdir, name=subname)
  File "/Users/abc/Library/Python/3.7/lib/python/site-packages/tensorboard/backend/event_processing/plugin_event_multiplexer.py", line 158, in AddRun
    accumulator.Reload()
  File "/Users/abc/Library/Python/3.7/lib/python/site-packages/tensorboard/backend/event_processing/plugin_event_accumulator.py", line 177, in Reload
    for event in self._generator.Load():
  File "/Users/abc/Library/Python/3.7/lib/python/site-packages/tensorboard/backend/event_processing/directory_watcher.py", line 94, in Load
    'Directory %s has been permanently deleted' % self._directory)
tensorboard.backend.event_processing.directory_watcher.DirectoryDeletedError: Directory /Users/abc/test/test_delete/run2 has been permanently deleted