Tensorboard dev upload bug

Unable to upload my experiment on tensorboard.dev. Here is my code :

tensorboard dev upload --logdir gs://bioasq_clean --name “bioasq accuracy bs” --description "Simple comparison of batch sizes gs://bioasq_clean " And this is the output with error:

Data for the “graphs” plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the “–plugins” command line ar gument. Data for the “histograms” plugin is now uploaded to TensorBoard.dev! Note that uploaded data is public. If you do not want to upload data for this plugin, use the “–plugins” command lin e argument. Upload started and will continue reading any new data as it’s added to the logdir. To stop uploading, press Ctrl-C. View your TensorBoard live at: https://tensorboard.dev/experiment/MLRmwHmOTXCmhNkprCJZPw/ [2020-06-20T18:12:43] Uploader started. W0620 18:12:47.274621 140588233123648 uploader.py:1074] Blob too large; skipping. Size 18160588 exceeds limit of 10485760 bytes. W0620 18:12:49.654345 140588233123648 uploader.py:1074] Blob too large; skipping. Size 18502392 exceeds limit of 10485760 bytes. W0620 18:12:56.931489 140588233123648 uploader.py:1074] Blob too large; skipping. Size 36349153 exceeds limit of 10485760 bytes. W0620 18:13:02.599462 140588233123648 uploader.py:1074] Blob too large; skipping. Size 37015585 exceeds limit of 10485760 bytes. W0620 18:13:06.744498 140588233123648 uploader.py:1074] Blob too large; skipping. Size 18160588 exceeds limit of 10485760 bytes. W0620 18:13:09.085675 140588233123648 uploader.py:1074] Blob too large; skipping. Size 18502392 exceeds limit of 10485760 bytes. W0620 18:13:15.518504 140588233123648 uploader.py:1074] Blob too large; skipping. Size 36349153 exceeds limit of 10485760 bytes. W0620 18:13:20.855475 140588233123648 uploader.py:1074] Blob too large; skipping. Size 37015585 exceeds limit of 10485760 bytes. [2020-06-20T18:13:25] Total uploaded: 0 scalars, 0 tensors, 1 binary objects (9.1 MB) Total skipped: 8 binary objects (209.9 MB) Listening for new data in logdir… Done. View your TensorBoard at https://tensorboard.dev/experiment/MLRmwHmOTXCmhNkprCJZPw/ Traceback (most recent call last): File “/usr/local/bin/tensorboard”, line 8, in <module> sys.exit(run_main()) File “/usr/local/lib/python3.7/dist-packages/tensorboard/main.py”, line 75, in run_main app.run(tensorboard.main, flags_parser=tensorboard.configure) File “/usr/local/lib/python3.7/dist-packages/absl/app.py”, line 299, in run _run_main(main, args) File “/usr/local/lib/python3.7/dist-packages/absl/app.py”, line 250, in _run_main sys.exit(main(argv)) File “/usr/local/lib/python3.7/dist-packages/tensorboard/program.py”, line 290, in main return runner(self.flags) or 0 File “/usr/local/lib/python3.7/dist-packages/tensorboard/uploader/uploader_subcommand.py”, line 633, in run return _run(flags) File “/usr/local/lib/python3.7/dist-packages/tensorboard/uploader/uploader_subcommand.py”, line 140, in _run intent.execute(server_info, channel) File “/usr/local/lib/python3.7/dist-packages/tensorboard/uploader/uploader_subcommand.py”, line 447, in execute uploader.start_uploading() File “/usr/local/lib/python3.7/dist-packages/tensorboard/uploader/uploader.py”, line 199, in start_uploading self._upload_once() File “/usr/local/lib/python3.7/dist-packages/tensorboard/uploader/uploader.py”, line 219, in _upload_once self._request_sender.send_requests(run_to_events) File “/usr/local/lib/python3.7/dist-packages/tensorboard/uploader/uploader.py”, line 430, in send_requests run_name, event, value, metadata File “/usr/local/lib/python3.7/dist-packages/tensorboard/uploader/uploader.py”, line 989, in add_event self.flush() File “/usr/local/lib/python3.7/dist-packages/tensorboard/uploader/uploader.py”, line 1021, in flush blob_sequence_id, seq_index, blob File “/usr/local/lib/python3.7/dist-packages/tensorboard/uploader/uploader.py”, line 1086, in _send_blob for response in self._api.WriteBlob(request_iterator):

Ran the downloaded the diagnostic script from : https://raw.githubusercontent.com/tensorflow/tensorboard/master/tensorboard/tools/diagnose_tensorboard.py

I ran the diagnostic script and got the report as following: $ python3 diagnose_tensorboard.py

Diagnostics

Diagnostics output

`````` --- check: autoidentify INFO: diagnose_tensorboard.py version 724b56cee52e7d8eb89bbeec1f0d5ce3e38c9682 --- check: general INFO: sys.version_info: sys.version_info(major=3, minor=7, micro=3, releaselevel='final', serial=0) INFO: os.name: posix INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='for-tensorboard', release='4.19.0-9-cloud-amd64', version='#1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07)', machine='x86_64 ') INFO: sys.getwindowsversion(): N/A --- check: package_management INFO: has conda-meta: False INFO: $VIRTUAL_ENV: None --- check: installed_packages INFO: installed: tb-nightly==2.3.0a20200619 WARNING: no installation among: ['tensorflow', 'tensorflow-gpu', 'tf-nightly', 'tf-nightly-2.0-preview', 'tf-nightly-gpu', 'tf-nightly-gpu-2.0-preview'] INFO: installed: tf-estimator-nightly==2.3.0.dev2020062001 --- check: tensorboard_python_version INFO: tensorboard.version.VERSION: '2.3.0a20200619' --- check: tensorflow_python_version INFO: tensorflow.__version__: '2.3.0-dev20200620'

What should be my next steps to resolve this isssue. Thank You Regards Ishmeet Kaur

[bileschi]

It sounds like the uploader is choking on some very large graphs. Can you try uploading with just scalars by using the --plugins scalars flag?

For instance:

tensorboard dev upload \
  --logdir gs://bioasq_clean \
  --plugins scalars \
  --name "bioasq accuracy bs" \
  --description "Simple comparison of batch sizes gs://bioasq_clean "

If you want multiple plugins, you can specify comma separated values. E.g., --plugins=scalars,histograms

If that works, then let’s see if we can fix the issue with the graphs

[bileschi]

Resolved this out of band. Issue was that Google Cloud Storage access required installing TensorFlow as well as TenorBoard.

@ishmeetk please re-open if I’ve closed this prematurely.