When I use TFBertEncoder in my laptop, I get an error.I can not build a model. Here is a simple examples.

🐛 Bug

Information

Model I am using TFBertEncoder:

Language I am using the model on English:

The problem arises when using:

• my own modified scripts: (give details below)

The tasks I am working on is:

• my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1. When I use, TFBertEncoder, I get an error.

Here is my code.

import tensorflow as tf
import numpy as np
from transformers.modeling_tf_bert import BertConfig, TFBertEncoder

print(tf.__name__, tf.__version__)
input_a = tf.keras.layers.Input(shape=(91, 128))

config = BertConfig()
config.hidden_size = 128
config.num_attention_heads = 4

# config.output_attentions = False
# config.output_hidden_states = False

head_mask = [None for _ in range(config.num_hidden_layers)]
encoder_output = TFBertEncoder(config=config)([input_a, None, head_mask])[0]
print(encoder_output.shape)

test_out = tf.keras.layers.Dense(128)(encoder_output)
print(test_out.shape)

Expected behavior

Here is the error:

(None, 91, 128)
2020-06-03 11:18:10.160647: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Failed precondition: Error while reading resource variable _AnonymousVar189 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar189/class tensorflow::Var does not exist.
	 [[{{node output_23/dense/BiasAdd/ReadVariableOp}}]]
Traceback (most recent call last):
  File "D:/python/tx/TEST.py", line 16, in <module>
    a = tf.keras.layers.Dense(128)(encoder_output)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 720, in __call__
    base_layer_utils.create_keras_history(inputs)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 187, in create_keras_history
    _, created_layers = _create_keras_history_helper(tensors, set(), [])
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 249, in _create_keras_history_helper
    layer_inputs, processed_ops, created_layers)
  [Previous line repeated 5 more times]
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer_utils.py", line 247, in _create_keras_history_helper
    constants[i] = backend.function([], op_input)([])
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\keras\backend.py", line 3727, in __call__
    outputs = self._graph_fn(*converted_inputs)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1551, in __call__
    return self._call_impl(args, kwargs)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1591, in _call_impl
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 1692, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py", line 545, in call
    ctx=ctx)
  File "D:\Anaconda3\lib\site-packages\tensorflow_core\python\eager\execute.py", line 67, in quick_execute
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.FailedPreconditionError:  Error while reading resource variable _AnonymousVar189 from Container: localhost. This could mean that the variable was uninitialized. Not found: Resource localhost/_AnonymousVar189/class tensorflow::Var does not exist.
	 [[node output_23/dense/BiasAdd/ReadVariableOp (defined at /python/tx/TEST.py:16) ]] [Op:__inference_keras_scratch_graph_5205]

Function call stack:
keras_scratch_graph

Environment info

• transformers version: 2.3.0 (in conda list)
• Platform:
• Python version:3.7
• PyTorch version (GPU?):
• Tensorflow version (GPU?):TF2.1.0(GPU)
• Using GPU in script?:
• Using distributed or parallel set-up in script?:No

[chuqidecha]

Both work for me , thanks!

[patrickvonplaten]

Hey guys,

I looked into the issue and I think the best solution is to use a keras layer wrapper as follows:

#!/usr/bin/env python3
import tensorflow as tf
import numpy as np
from transformers.modeling_tf_bert import BertConfig, TFBertEncoder

print(tf.__name__, tf.__version__)

config = BertConfig()
config.hidden_size = 128
config.num_attention_heads = 4


class NewTFBertEncoder(tf.keras.layers.Layer):

    def __init__(self, config):
        super(NewTFBertEncoder, self).__init__()
#        self.inputs = tf.keras.layers.Input(input_shape)  # not really needed here IMO.
        self.encoder = TFBertEncoder(config=config)
        self.dense = tf.keras.layers.Dense(config.hidden_size)

    def call(self, inputs):
        head_mask = [None for _ in range(config.num_hidden_layers)]
        output = self.encoder([inputs, None, head_mask])[0]
        dense_output = self.dense(output)

        return dense_output


new_bert_encoder = NewTFBertEncoder(config)
output = new_bert_encoder(np.ones((2, 91, 128)))  # batch size , sequence length, hidden size