question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unable to use tensorboard viewer using S3 path

See original GitHub issue

What steps did you take

To view the output in tensorboard we have created the mlpipeline-ui-metadata.json file with following content (Followed the instructions from - https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/#tensorboard )

{"outputs": [{"type": "tensorboard", "source": "s3://<S3_path>"}]}

to view the training output in tensorboard.

What happened:

In the visualizations tab, Start Tensorboard button is appearing as expected.

image

Once Start Tensorboard is clicked, Open Tensorboard is appearing as expected.

image

But when the Start Tensorboard is clicked, an empty dashboard is shown.

image

What did you expect to happen:

Tensorboard should be opened as below

image

Environment:

  • KFP version: 1.4 To find the version number, See version number shows on bottom of KFP UI left sidenav. -->
  • KFP SDK version: 0.5.2

Anything else you would like to add:

To test the above behavious in local machine, we tried running

tensorboard --logdir s3://<S3_path>

Tensorboard is showing an empty dashboard.

However, the dashboard is working fine with the db argument as below

tensorboard --db s3://<S3_path>

From the https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/#tensorboard i couldnt figure out a way to change from logdir to db. Or is there any other way to achieve this in kubeflow mlpipeline-ui-metadata.json ?

Labels

/area frontend


Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

Issue Analytics

  • State:closed
  • Created 2 years ago
  • Comments:20 (19 by maintainers)

github_iconTop GitHub Comments

3reactions
shrinath-sureshcommented, Apr 16, 2021

That’s a very good point, do you want to contribute that?

Already occupied with deliverables. I can pick it up after 2 weeks. I will make a note of it.

1reaction
shrinath-sureshcommented, Apr 16, 2021

@Bobgy It will be more helpful if the documentation is updated about the supported systems - https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/#tensorboard . What do you think ?

Read more comments on GitHub >

github_iconTop Results From Across the Web

tensorboard logdir with s3 path - Stack Overflow
If you start a tensorboard by using AWS s3 file, you should do as follows: (1) add ENV VARIBLES export AWS_ACCESS_KEY_ID=****** export ...
Read more >
How Amazon SageMaker Provides Training Information
This section explains how SageMaker makes training information, such as training data, hyperparameters, and other configuration information, available to ...
Read more >
tf.keras.callbacks.TensorBoard | TensorFlow v2.11.0
TensorBoard is a visualization tool provided with TensorFlow. This callback logs events for TensorBoard, including: Metrics summary plots; Training graph ...
Read more >
Visualize Amazon SageMaker Training Jobs with TensorBoard
The notebook uses an external S3 Bucket to store TensorFlow logs. This allows data scientists to use TensorBoard to visualize all past training...
Read more >
How to Make Your TensorBoard Projects Easy to Share and ...
Although each engineer has their own piece of the project to work on, they still need to share their results with the team...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found