Performance data broken in 1.3.0rc0/1
See original GitHub issueI noticed that performance metrics such as compute time and memory usage are broken in TF 1.3.0 rc0 and rc1. When running the official MNIST example with summaries, tensorboard shows many unused nodes. Especially the ‘devices’ section on the left appears to contain no devices at all.
Here are the tensorflow logs
python .\mnist.py --log_dir logs --data_dir data --fake_data True
2017-07-30 08:30:58.271485: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-07-30 08:30:58.271549: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-07-30 08:30:58.907658: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1070
major: 6 minor: 1 memoryClockRate (GHz) 1.683
pciBusID 0000:01:00.0
Total memory: 8.00GiB
Free memory: 6.68GiB
2017-07-30 08:30:58.907767: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0
2017-07-30 08:30:58.908268: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:986] 0: Y
2017-07-30 08:30:58.908320: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0)
Accuracy at step 0: 0.0
Accuracy at step 10: 1.0
Accuracy at step 20: 1.0
Accuracy at step 30: 1.0
Accuracy at step 40: 1.0
Accuracy at step 50: 1.0
Accuracy at step 60: 1.0
Accuracy at step 70: 1.0
Accuracy at step 80: 1.0
Accuracy at step 90: 1.0
2017-07-30 08:31:46.475993: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\dso_loader.cc:139] successfully opened CUDA library cupti64_80.dll locally
Adding run metadata for 99
System Setup
Tensorflow 1.3.0 rc0 / rc1 with GPU support GeForce GTX 1070 Windows 10 x64 Python 3.6 / 3.5 Chrome 59.0.3071.115
Issue Analytics
- State:
- Created 6 years ago
- Comments:14 (4 by maintainers)
Top Results From Across the Web
No results found
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Seems still broken in now released v1.3 - I’m having the same issues (with mnist_with_summaries.py). I also provided a minimal example on Stackoverflow and now found this issue here…
https://stackoverflow.com/questions/45739917/tensorboard-profiling-compute-time-everything-is-a-unused-substructure-in
However, I also had similar problems already on v1.2.1 with my own graphs. Many Items would have stats, but some sub items inside Ops grouped with name scopes would be shown as unused-substructures without stats, yielding in false results for the sum of the whole group…
@cheind
Closing this issue as it is resolved, Please feel free to reopen if this still an issue. Thanks