question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Performance data broken in 1.3.0rc0/1

See original GitHub issue

I noticed that performance metrics such as compute time and memory usage are broken in TF 1.3.0 rc0 and rc1. When running the official MNIST example with summaries, tensorboard shows many unused nodes. Especially the ‘devices’ section on the left appears to contain no devices at all.

image

Here are the tensorflow logs

python .\mnist.py --log_dir logs --data_dir data --fake_data True
2017-07-30 08:30:58.271485: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-30 08:30:58.271549: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-30 08:30:58.907658: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1070
major: 6 minor: 1 memoryClockRate (GHz) 1.683
pciBusID 0000:01:00.0
Total memory: 8.00GiB
Free memory: 6.68GiB
2017-07-30 08:30:58.907767: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0
2017-07-30 08:30:58.908268: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:986] 0:   Y
2017-07-30 08:30:58.908320: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0)
Accuracy at step 0: 0.0
Accuracy at step 10: 1.0
Accuracy at step 20: 1.0
Accuracy at step 30: 1.0
Accuracy at step 40: 1.0
Accuracy at step 50: 1.0
Accuracy at step 60: 1.0
Accuracy at step 70: 1.0
Accuracy at step 80: 1.0
Accuracy at step 90: 1.0
2017-07-30 08:31:46.475993: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\dso_loader.cc:139] successfully opened CUDA library cupti64_80.dll locally
Adding run metadata for 99

System Setup

Tensorflow 1.3.0 rc0 / rc1 with GPU support GeForce GTX 1070 Windows 10 x64 Python 3.6 / 3.5 Chrome 59.0.3071.115

Issue Analytics

  • State:closed
  • Created 6 years ago
  • Comments:14 (4 by maintainers)

github_iconTop GitHub Comments

4reactions
hofingermarkuscommented, Aug 18, 2017

Seems still broken in now released v1.3 - I’m having the same issues (with mnist_with_summaries.py). I also provided a minimal example on Stackoverflow and now found this issue here…

https://stackoverflow.com/questions/45739917/tensorboard-profiling-compute-time-everything-is-a-unused-substructure-in

However, I also had similar problems already on v1.2.1 with my own graphs. Many Items would have stats, but some sub items inside Ops grouped with name scopes would be shown as unused-substructures without stats, yielding in false results for the sum of the whole group…

0reactions
pindinageshcommented, Oct 26, 2021

@cheind

Closing this issue as it is resolved, Please feel free to reopen if this still an issue. Thanks

Read more comments on GitHub >

github_iconTop Results From Across the Web

No results found

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found