Using CPU for inference with GPU-trained model

I trained a seq2seq model using CuDNN and plan to use this model on devices without GPUs. This model works fine with GPUs. However, when loading the model on devices without GPUs, I keep getting the following error. How do I use the model on devices without GPUs?

  device='GPU'; T in [DT_HALF]

         [[bidirectional_encoder_GRU/CudnnRNN]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ntran1/.local/lib/python3.6/site-packages/keras_wrapper/cnn_model.py", line 176, in loadModel
    model = load_model(model_name + '.h5', custom_objects=custom_objects, compile=compile_model)
  File "/DCNFS/users/student/ntran1/Desktop/keras/keras/engine/saving.py", line 453, in load_wrapper
    return load_function(*args, **kwargs)
  File "/DCNFS/users/student/ntran1/Desktop/keras/keras/engine/saving.py", line 545, in load_model
    model = _deserialize_model(h5dict, custom_objects, compile)
  File "/DCNFS/users/student/ntran1/Desktop/keras/keras/engine/saving.py", line 305, in _deserialize_model
    K.batch_set_value(weight_value_tuples)
  File "/DCNFS/users/student/ntran1/Desktop/keras/keras/backend/tensorflow_backend.py", line 2897, in batch_set_value
    get_session().run(assign_ops, feed_dict=feed_dict)
  File "/DCNFS/users/student/ntran1/Desktop/keras/keras/backend/tensorflow_backend.py", line 207, in get_session
    [tf.is_variable_initialized(v) for v in candidate_vars])
  File "/home/ntran1/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/home/ntran1/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ntran1/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/home/ntran1/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: No OpKernel was registered to support Op 'CudnnRNN' used by node bidirectional_encoder_GRU/CudnnRNN (defined at /home/ntran1/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) with these attrs: [direction="unidirectional", rnn_mode="gru", seed2=0, is_training=true, seed=87654321, dropout=0, input_mode="linear_input", T=DT_FLOAT]
Registered devices: [CPU, XLA_CPU, XLA_GPU]
Registered kernels:
  device='GPU'; T in [DT_DOUBLE]
  device='GPU'; T in [DT_FLOAT]
  device='GPU'; T in [DT_HALF]

         [[bidirectional_encoder_GRU/CudnnRNN]]

[philipcori]

Thanks. Just want to quickly note that to fix the issue Nam was facing, which I also faced, I needed to first save the CPU model with the saveModel() function provided by multimodal-keras-wrapper. Once it was saved, I could then load this model in a machine without a GPU.

[lvapeab]

Have you modified the models at model_zoo? I’ve been able to train a CuDNN-GRU, save it, load it as a regular GRU and continue its training without problems. Here’s a notebook.

Also, from the links I found in my previous post, it is suggested that the recurrent activation be set to “sigmoid.” How do I set the recurrent activation to “sigmoid”? Thanks a lot.

The default recurrent activation is “sigmoid” (implementation). Moreover, I would carefuly think about changing this activation, as it is intended to work as a gate, that decides the amount of information that passes through the unit (squashing the values to [0, 1]).