Different output results for Jupyter Notebook and other IDE for TensorFlow GPU

Hi all! System configuration: Linux Debian 11 Python 3.8.12 - compiled from source with Clang 12 (appeared also for Python installed from OS repo, compiled with gcc by pyenv and etc.) IPython: 7.28.0 IDEs: Spyder 5.1.5, Eclipse PyDev, PyCharm Tensorflow: 2.5.1 compiled from source Clang-12 (in my case bug appear for all edition TF - installed from PyPy or compiled from source myself with gcc/cliang also appear in the TF above and bellow versions instaled from pip) Hello everyone the essence of the problem: compiling locally the same code using TF above Jupiter Notebook in the IDE (locally) gives different results, although in all cases using the same version IPython backend. I opened the ticket in Google, but they play enterprise ping pong and closed it without doing anything. The output in Jupiter Notebook locally is the same as the output if you run the code remote in the Google colab.

Sample output console when run python script in the console or IDE (bellow script run on the Google support engineer PC, same when I run this script in the IDEs above locally):

(base) vishnuvardhanj-macbookpro:ch_4_IMDB_issue vishnuvardhanj$ python3 ch_4_IMDB_binary_classification.py
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz
17465344/17464789 [==============================] - 1s 0us/step
<__array_function__ internals>:5: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/keras/datasets/imdb.py:159: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tensorflow/python/keras/datasets/imdb.py:160: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json
1646592/1641221 [==============================] - 0s 0us/step
2021-09-20 17:23:25.962502: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-09-20 17:23:25.962739: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-09-20 17:23:27.823250: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
Epoch 1/20
30/30 [==============================] - 4s 94ms/step - loss: 0.6932 - acc: 0.4978 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 2/20
30/30 [==============================] - 1s 19ms/step - loss: 0.6931 - acc: 0.5095 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 3/20
30/30 [==============================] - 1s 19ms/step - loss: 0.6931 - acc: 0.5032 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 4/20
30/30 [==============================] - 1s 18ms/step - loss: 0.6931 - acc: 0.5041 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 5/20
30/30 [==============================] - 1s 18ms/step - loss: 0.6931 - acc: 0.5051 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 6/20
30/30 [==============================] - 1s 17ms/step - loss: 0.6931 - acc: 0.5078 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 7/20
30/30 [==============================] - 0s 16ms/step - loss: 0.6931 - acc: 0.5079 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 8/20
30/30 [==============================] - 0s 16ms/step - loss: 0.6931 - acc: 0.5086 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 9/20
30/30 [==============================] - 0s 15ms/step - loss: 0.6931 - acc: 0.5040 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 10/20
30/30 [==============================] - 0s 15ms/step - loss: 0.6931 - acc: 0.5026 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 11/20
30/30 [==============================] - 1s 17ms/step - loss: 0.6931 - acc: 0.5047 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 12/20
30/30 [==============================] - 0s 16ms/step - loss: 0.6931 - acc: 0.5060 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 13/20
30/30 [==============================] - 0s 15ms/step - loss: 0.6932 - acc: 0.4959 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 14/20
30/30 [==============================] - 0s 16ms/step - loss: 0.6931 - acc: 0.5047 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 15/20
30/30 [==============================] - 0s 17ms/step - loss: 0.6931 - acc: 0.5035 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 16/20
30/30 [==============================] - 0s 16ms/step - loss: 0.6931 - acc: 0.5054 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 17/20
30/30 [==============================] - 0s 17ms/step - loss: 0.6931 - acc: 0.5037 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 18/20
30/30 [==============================] - 0s 15ms/step - loss: 0.6931 - acc: 0.5050 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 19/20
30/30 [==============================] - 1s 18ms/step - loss: 0.6932 - acc: 0.5002 - val_loss: 0.6932 - val_acc: 0.4947
Epoch 20/20
30/30 [==============================] - 1s 17ms/step - loss: 0.6931 - acc: 0.5046 - val_loss: 0.6932 - val_acc: 0.4947
49/49 [==============================] - 2s 21ms/step - loss: 0.2500 - acc: 0.4964
782/782 [==============================] - 2s 2ms/step - loss: 0.2500 - acc: 0.5000

Sample console output for this code in the Jupyter Notebook locally (same with Google Colab remotely)

Epoch 1/20
2021-09-21 16:43:34.579784: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
 8/30 [=======>......................] - ETA: 0s - loss: 0.6436 - acc: 0.6494
2021-09-21 16:43:35.770346: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
30/30 [==============================] - 3s 41ms/step - loss: 0.5253 - acc: 0.7831 - val_loss: 0.3976 - val_acc: 0.8716
Epoch 2/20
30/30 [==============================] - 1s 22ms/step - loss: 0.3144 - acc: 0.9030 - val_loss: 0.3158 - val_acc: 0.8854
Epoch 3/20
30/30 [==============================] - 1s 22ms/step - loss: 0.2335 - acc: 0.9269 - val_loss: 0.2900 - val_acc: 0.8887
Epoch 4/20
30/30 [==============================] - 1s 22ms/step - loss: 0.1835 - acc: 0.9429 - val_loss: 0.2745 - val_acc: 0.8910
Epoch 5/20
30/30 [==============================] - 1s 22ms/step - loss: 0.1523 - acc: 0.9524 - val_loss: 0.2796 - val_acc: 0.8886
Epoch 6/20
30/30 [==============================] - 1s 22ms/step - loss: 0.1258 - acc: 0.9621 - val_loss: 0.3459 - val_acc: 0.8712
Epoch 7/20
30/30 [==============================] - 1s 22ms/step - loss: 0.1059 - acc: 0.9701 - val_loss: 0.3153 - val_acc: 0.8828
Epoch 8/20
30/30 [==============================] - 1s 22ms/step - loss: 0.0913 - acc: 0.9753 - val_loss: 0.3264 - val_acc: 0.8789
Epoch 9/20
30/30 [==============================] - 1s 22ms/step - loss: 0.0763 - acc: 0.9789 - val_loss: 0.3627 - val_acc: 0.8713
Epoch 10/20
30/30 [==============================] - 1s 22ms/step - loss: 0.0644 - acc: 0.9840 - val_loss: 0.3556 - val_acc: 0.8795
Epoch 11/20
30/30 [==============================] - 1s 21ms/step - loss: 0.0518 - acc: 0.9871 - val_loss: 0.3862 - val_acc: 0.8788
Epoch 12/20
30/30 [==============================] - 1s 21ms/step - loss: 0.0435 - acc: 0.9905 - val_loss: 0.4076 - val_acc: 0.8770
Epoch 13/20
30/30 [==============================] - 1s 22ms/step - loss: 0.0351 - acc: 0.9926 - val_loss: 0.4554 - val_acc: 0.8678
Epoch 14/20
30/30 [==============================] - 1s 21ms/step - loss: 0.0293 - acc: 0.9940 - val_loss: 0.4661 - val_acc: 0.8715
Epoch 15/20
30/30 [==============================] - 1s 22ms/step - loss: 0.0228 - acc: 0.9961 - val_loss: 0.4923 - val_acc: 0.8716
Epoch 16/20
30/30 [==============================] - 1s 22ms/step - loss: 0.0189 - acc: 0.9973 - val_loss: 0.5289 - val_acc: 0.8688
Epoch 17/20
30/30 [==============================] - 1s 21ms/step - loss: 0.0150 - acc: 0.9981 - val_loss: 0.5599 - val_acc: 0.8704
Epoch 18/20
30/30 [==============================] - 1s 22ms/step - loss: 0.0137 - acc: 0.9977 - val_loss: 0.5907 - val_acc: 0.8697
Epoch 19/20
30/30 [==============================] - 1s 23ms/step - loss: 0.0077 - acc: 0.9997 - val_loss: 0.6228 - val_acc: 0.8668
Epoch 20/20
30/30 [==============================] - 1s 20ms/step - loss: 0.0078 - acc: 0.9992 - val_loss: 0.6653 - val_acc: 0.8689

Attach files - script, Jupyter Notebook and theirs output for different cases. For example using script from the book “Deep Learning with Python Second Edition” by Francois Cholet MEAP 202 ch_4_IMDB_issue.tar.gz 0.

[Vadim-Maklakov]

@Vadim-Maklakov, what happens if you start Spyder from a system terminal (xterm, konsole, etc) and run your code? That should pick up all your environment variables and set them in the kernel of our IPython consoles correctly.

After solution patch for Matthias Bussonnier output become same everywhere - executing Python script in the system console, Eclipse PyDev, Spyder or Jupyter Notebook /Google colab.
Jupiter notebook / Google colab probably have some kind of embedded protection from such fools as me))

[Carreau]

Thank you for quickly seeing my banal typo, which Google’s super engineers could not see

Haha, that happen to everyone.

I actually formatted your Python script with black, git commit , copy and pasted from the notebook, formatted with black again and diffed. So yeah it happens to everyone.