Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Tensorflow backend lower GPU Util compared to Theano

See original GitHub issue

Used a both a custom MLP model and the examples/mnist_mlp.py to test:

TF:

epoch time ~5s
real	1m20.077s
user	2m3.212s
sys	0m16.884s

Theano:

epoch time ~1s
real	0m38.972s
user	0m35.212s
sys	0m4.168s

On a Tesla K20c. The nvidia-smi gpu util is around 80% when using Theano. TF’s gpu util is around 10-20%. Using gpu device 0: Tesla K20c (CNMeM is enabled with initial size: 90.0% of memory, cuDNN 5005)

When I disable cnmem in .theanorc, theano performance becomes similar to TF: 20% gpu util, epoch time of ~4s:

real	1m51.123s
user	1m0.212s
sys	0m50.972s

Is there a way of customizing TF’s memory allocation?

Issue Analytics

State:
Created 7 years ago
Comments:8

Top GitHub Comments

1reaction

mrkentcommented, Dec 16, 2016

@CCXD Just checked and I believe that’s the problem. Importing tensorflow seems like it no longer uses cuda after installing keras. It might be caused by also installing Anaconda.

Python 2.7.12 |Anaconda 4.2.0 (64-bit)| (default, Jul  2 2016, 17:42:40)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import tensorflow as tf
>>>

0reactions

stale[bot]commented, Sep 13, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.