What UI would you like to see for image/audio summaries?

Both the image and audio summaries include a batch_size, which allows you to record multiple images/clips at a single step. Currently, this works by creating (say) three separate tags foo/image/{0,1,2}, and so in TensorBoard you see three separate image/audio cards. Once we migrate to tensor summaries (see below for more details), we’ll instead store your three images/clips as three entries in a single tensor in a single tag instead of one entry each across three tags. This gives us more flexibility in how we display your data in the UI.

One option is to retain a functionally equivalent view to what’s currently present: we could, on the client side, silently explode the first dimension of the tensor, and yield three copies of the card.

Or, if you prefer, we could add a second slider below the existing step slider, allowing you to change the sample that you’re viewing. (I could see how this could have both advantages and disadvantages for images: on the one hand, you can quickly move back and forth to compare two samples, but on the other hand you wouldn’t have them available side-by-side.) Such a slider could alternatively be at the whole-dashboard level.

Perhaps for each tag we could have a similar slider that instead controls the number of samples that are shown at any given time.

Here are some more details about what this “migration to tensor summaries is” and why it’s related. As we continue to make TensorBoard’s plugin system more and more well-defined, we’re changing the format of newly created image and audio summaries. Currently, image and audio summaries each have a special field in the Summary.Value protobuf. This privileges these summaries over other summaries, so we’ll shortly be migrating them to use the generic tensor field instead, and—as mentioned above—storing all image/clips from one step within one tensor. For comparison, we recently did this for the histogram plugin; you can see the results in #246. Don’t worry: your old data will be compatible, and the old APIs will continue to work.

I’m open to suggestions—what do you think? (For now, I’ll implement the first option, retaining feature parity.)

[wchargin]

@jlazarow:

it seems a bit crazy to have to do that for every small complaint

I hear you, and I agree that forking the whole dashboard to add label information is not a long-term solution.

Like I mentioned, the problem is really in getting the label information from your TensorFlow session to the outside world. I’ve continued to think about it, and it seems like we might do the following. Currently, the image summary emits a rank-1 string tensor of the form [width_str, height_str, png_0, png_1, …], where the png_i are bytestrings with PNG-encoded image data and width_str and height_str are ASCII representations of the image width and height in decimal (e.g., "123"). We could backward-compatibly allow the operation to instead output a rank-2 string tensor of the form [[width_str, height_str, png_0, png_1, …], [ignored, ignored, markdown_label_0, markdown_label_1, …]], where [markdown_label_0, markdown_label_2, …] is a string tensor provided to the summary. I’ll see what other people on the team think about this, and then we can consider implementing it.

I guess I’m trying to use TensorBoard from an experimental/research aspect (perhaps wrongly?)

Not wrongly at all—our primary goal is to help researchers write visualizations tailored to their research.

The issue is that the interface between TF and TB is just too rigid

It is. In particular, the requirement that all summary output be a tensor, not something more struct-like, is very restrictive. (Things like “the images plugin emits a tensor the first two values of which are the stringified versions of the image dimensions” are already too hacky for my tastes…)

It’s already a hack to just change the names of summaries without introducing a new op

Happily, our new APIs support display_name and also summary_description parameters. The images demo (//tensorboard/plugins/image:images_demo) includes code that demonstrates how to use these parameters. You can see example screenshots in this PR: https://github.com/tensorflow/tensorboard/pull/314. Hope this helps a bit. 😃

[jlazarow]

Can you allow “arbitrary” tagging to the summaries outside of the actual summary name? I’d like to sometimes name the individual images and associate label information to them (perhaps there is already a way to do this?).