Support multiple file writers writing at once.

Background

TensorBoard assigns one DirectoryWatcher per run. The DirectoryWatcher watches for new events being written to the directory and loads it into TensorBoard’s multiplexer. Today, the DirectoryWatcher iterates through events files (for a run) in lexicographic order. After it has finished reading from an events file, it never reads from it again (and moves on to a subsequent events file).

This behavior means that TensorBoard does not support multiple file writers writing to the same run directory: TensorBoard would move on to a different events file B once it finishes reading events file A and would never go back to reading A despite how A is updated.

In turn, that problem blocks TensorFlow+TensorBoard’s transition towards migrating Estimator, tflearn hooks, and other high-level TensorFlow constructs to using the tf.contrib.summary-style summaries. tf.contrib.summary-style summaries are written via a type of SummaryWriter different from the one that writes traditional-style (tf.summary.*) summaries, so users that intermingle the summaries will necessarily have to concurrently use 2 summary writers. Many users thoughout google3 intermingle the summaries, and the transition must happen soon.

[nfelt]

I’ve merged #1867 which introduces the experimental --reload_multifile=true option, which can now be used to poll all “active” files in a directory for new data, rather than the most recent one, and addresses the core concern of this issue. Note that a file is “active” as long as it received new data within --reload_multifile_inactive_secs seconds ago, a new configurable flag that defaults to 4000, aka just over an hour.

If you’ve been waiting on this functionality, thanks for your patience, and please try out the new flag in tb-nightly starting with tomorrow’s release and let us know how it works for you.

[nfelt]

This issue should stay open, since the flag is still experimental and not on by default yet.