Image stylization example only produces black images when run on CPU

With tensorflow on master (commit baa85cbf5e51a21f58bc28ef9eedc122e6118eb8). Command run:

CUDA_VISIBLE_DEVICES=-1 bazel-bin/magenta/models/image_stylization/image_stylization_transform       --num_styles=32       --checkpoint=$HOME/Downloads/multistyle-pastiche-generator-varied.ckpt      --input_image=${INPUT_IMG}       --which_styles="[0]"       --output_dir=${OUTPUT_FOLDER}       --output_basename="stylized"

Everything works as intended when running on GPU with CUDA_VISIBLE_DEVICES=0. Tensorflow wheel built with

bazel build -c opt --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-mfpmath=both --copt=-msse4.2 --config=cuda //tensorflow/tools/pip_package:build_pip_package

[psoulos]

I am running into this issue as well. To perform image stylization on CPU, it is necessary to install tensorflow 1.0.1 instead of tensorflow 1.1.0

[hav4ik]

Thanks @panmari for localizing the first occurrence of the issue. The problem is caused by NaN values in the weights of Conv2D layers. The reason why the network works on GPU is supposedly because GPU does not have a exception mechanism, so NaN values will be converted to zeros. On CPU, however, the NaN is preserved thorough computation.

My personal workaround of this issue is the following:

for var in tf.global_variables():
    w = var.eval()
    w = np.nan_to_num(w)
    var.assign(w).eval()

This snippet should be placed every time you load the model into the session. Worked for tensorflow==1.8.0. Will make a pull request soon onto both tensorflow/magenta and tensorflow/magenta-demos repositories.

A more correct solution should be replacing all the weights in the model according to the snippet above (so someone should upload the new weights onto download.magenta.tensorflow.org/models/ upstream).