Fine Tune Bert Ner using TFBertForTokenClassification.from_pretrained

Hey,

I am new to the transformers Bert Training world and am trying to fine tune Bert model for NER on a coNll like dataset. But its not training the model and giving the below error

ValueError: No gradients provided for any variable: [‘tf_bert_for_token_classification_8/classifier/kernel:0’, ‘tf_bert_for_token_classification_8/classifier/bias:0’].

Below is my code

tr_inputs = tf.convert_to_tensor(tr_inputs)
val_inputs = tf.convert_to_tensor(val_inputs)
tr_tags = tf.convert_to_tensor(tr_tags)
val_tags = tf.convert_to_tensor(val_tags)
tr_masks = tf.convert_to_tensor(tr_masks)
val_masks = tf.convert_to_tensor(val_masks)
tr_segs = tf.convert_to_tensor(tr_segs)
val_segs = tf.convert_to_tensor(val_segs)

input_features_dict = {"input_ids":tr_inputs, "attention_mask":tr_masks, "token_type_ids":tr_segs, 'labels':tr_tags}
val_features_dict = {"input_ids":val_inputs, "attention_mask":val_masks, "token_type_ids":val_segs, 'labels':tr_tags}
train_data = tf.data.Dataset.from_tensor_slices(input_features_dict)
batch_train_data = train_data.batch(batch_num)
valid_data = tf.data.Dataset.from_tensor_slices(val_features_dict)
batch_valid_data = valid_data.batch(batch_num)
modell = TFBertForTokenClassification.from_pretrained('bert-base-uncased',num_labels=len(tag2idx))
modell.layers[2].activation = tf.keras.activations.softmax
modell.layers[0].trainable = False
modell.compile(optimizer=optimizer, loss=loss, metrics=[metrics])
modell.fit(batch_train_data, epochs=epochs, validation_data=batch_val_data)

Not sure what needs to be done. Any advice/pointers on this would be highly helpful for me.

[jplu]

Hello @aks2193!

Sorry for this, but for now you cannot use .compile() + .fit() to train a Token Classification model. To make it short, this is because a layer is not used and then the gradients will be None, something that .fit() cannot handle.

If you want to train a NER I suggest you to use the example.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.