Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Dev build of TensorFlow causing issue with pre-trained BERT

See original GitHub issue

System Info

Python version: 3.7 TF branch: dev (this is part of our nightly CI checks for MLflow to test dev builds; sorry for not executing transformers-cli env for this report)

installed packages: absl-py-1.2.0 astunparse-1.6.3 cachetools-5.2.0 flatbuffers-22.9.24 gast-0.4.0 google-auth-2.12.0 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 grpcio-1.49.1 h5py-3.7.0 keras-nightly-2.11.0.dev2022101007 libclang-14.0.6 markdown-3.4.1 opt-einsum-3.3.0 protobuf-3.19.6 pyasn1-0.4.8 pyasn1-modules-0.2.8 rsa-4.9 tb-nightly-2.11.0a20221010 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-io-gcs-filesystem-0.27.0 termcolor-2.0.1 tf-estimator-nightly-2.11.0.dev2022101008 tf-nightly-2.11.0.dev20221010 wrapt-1.14.1

Who can help?

@LysandreJik

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, …)
  • My own task or dataset (give details below)

Reproduction

simply import transformers.models.bert

Issue line: https://www.google.com/url?q=https://github.com/huggingface/transformers/blob/10100979ed0594d4cfe1982cdfac9642a68e473e/src/transformers/modeling_tf_utils.py%23L39&sa=D&source=docs&ust=1665453601638298&usg=AOvVaw1r381k-VA_PhIdIhALrmxc

The stack trace:

self = <module 'transformers.models.bert' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/__init__.py'>
module_name = 'modeling_tf_bert'

    def _get_module(self, module_name: str):
        try:
>           return importlib.import_module("." + module_name, self.__name__)

module_name = 'modeling_tf_bert'
self       = <module 'transformers.models.bert' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/__init__.py'>

/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/utils/import_utils.py:1031: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = '.modeling_tf_bert', package = 'transformers.models.bert'

    def import_module(name, package=None):
        """Import a module.
    
        The 'package' argument is required when performing a relative import. It
        specifies the package to use as the anchor point from which to resolve the
        relative import to an absolute import.
    
        """
        level = 0
        if name.startswith('.'):
            if not package:
                msg = ("the 'package' argument is required to perform a relative "
                       "import for {!r}")
                raise TypeError(msg.format(name))
            for character in name:
                if character != '.':
                    break
                level += 1
>       return _bootstrap._gcd_import(name[level:], package, level)

character  = 'm'
level      = 1
name       = '.modeling_tf_bert'
package    = 'transformers.models.bert'

/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/importlib/__init__.py:127: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'transformers.models.bert.modeling_tf_bert'
package = 'transformers.models.bert', level = 1

>   ???

level      = 1
name       = 'transformers.models.bert.modeling_tf_bert'
package    = 'transformers.models.bert'

<frozen importlib._bootstrap>:1006: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'transformers.models.bert.modeling_tf_bert'
import_ = <function _gcd_import at 0x7f8299e66b00>

>   ???

import_    = <function _gcd_import at 0x7f8299e66b00>
module     = <object object at 0x7f8299e4e060>
name       = 'transformers.models.bert.modeling_tf_bert'

<frozen importlib._bootstrap>:983: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

name = 'transformers.models.bert.modeling_tf_bert'
import_ = <function _gcd_import at 0x7f8299e66b00>

>   ???

import_    = <function _gcd_import at 0x7f8299e66b00>
name       = 'transformers.models.bert.modeling_tf_bert'
parent     = 'transformers.models.bert'
parent_module = <module 'transformers.models.bert' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/__init__.py'>
path       = ['/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert']
spec       = ModuleSpec(name='transformers.models.bert.modeling_tf_bert', loader=<_frozen_importlib_external.SourceFileLoader objec...igin='/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py')

<frozen importlib._bootstrap>:967: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

spec = ModuleSpec(name='transformers.models.bert.modeling_tf_bert', loader=<_frozen_importlib_external.SourceFileLoader objec...igin='/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py')

>   ???

module     = <module 'transformers.models.bert.modeling_tf_bert' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py'>
spec       = ModuleSpec(name='transformers.models.bert.modeling_tf_bert', loader=<_frozen_importlib_external.SourceFileLoader objec...igin='/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py')

<frozen importlib._bootstrap>:677: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_frozen_importlib_external.SourceFileLoader object at 0x7f8[208](https://github.com/mlflow/mlflow/actions/runs/3219669785/jobs/5266077564#step:12:209)0ff6d0>
module = <module 'transformers.models.bert.modeling_tf_bert' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py'>

>   ???

code       = <code object <module> at 0x7f82080b9ae0, file "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py", line 16>
module     = <module 'transformers.models.bert.modeling_tf_bert' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py'>
self       = <_frozen_importlib_external.SourceFileLoader object at 0x7f82080ff6d0>

<frozen importlib._bootstrap_external>:728: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

f = <built-in function exec>
args = (<code object <module> at 0x7f82080b9ae0, file "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tra...ngAndCrossAttentions': <class 'transformers.modeling_tf_outputs.TFBaseModelOutputWithPoolingAndCrossAttentions'>, ...})
kwds = {}

>   ???

args       = (<code object <module> at 0x7f82080b9ae0, file "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tra...ngAndCrossAttentions': <class 'transformers.modeling_tf_outputs.TFBaseModelOutputWithPoolingAndCrossAttentions'>, ...})
f          = <built-in function exec>
kwds       = {}

<frozen importlib._bootstrap>:219: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """ TF 2.0 BERT model."""
    
    import math
    import warnings
    from dataclasses import dataclass
    from typing import Dict, Optional, Tuple, Union
    
    import numpy as np
    import tensorflow as tf
    
    from ...activations_tf import get_tf_activation
    from ...modeling_tf_outputs import (
        TFBaseModelOutputWithPastAndCrossAttentions,
        TFBaseModelOutputWithPoolingAndCrossAttentions,
        TFCausalLMOutputWithCrossAttentions,
        TFMaskedLMOutput,
        TFMultipleChoiceModelOutput,
        TFNextSentencePredictorOutput,
        TFQuestionAnsweringModelOutput,
        TFSequenceClassifierOutput,
        TFTokenClassifierOutput,
    )
>   from ...modeling_tf_utils import (
        TFCausalLanguageModelingLoss,
        TFMaskedLanguageModelingLoss,
        TFModelInputType,
        TFMultipleChoiceLoss,
        TFNextSentencePredictionLoss,
        TFPreTrainedModel,
        TFQuestionAnsweringLoss,
        TFSequenceClassificationLoss,
        TFTokenClassificationLoss,
        get_initializer,
        keras_serializable,
        unpack_inputs,
    )

Dict       = typing.Dict
Optional   = typing.Optional
TFBaseModelOutputWithPastAndCrossAttentions = <class 'transformers.modeling_tf_outputs.TFBaseModelOutputWithPastAndCrossAttentions'>
TFBaseModelOutputWithPoolingAndCrossAttentions = <class 'transformers.modeling_tf_outputs.TFBaseModelOutputWithPoolingAndCrossAttentions'>
TFCausalLMOutputWithCrossAttentions = <class 'transformers.modeling_tf_outputs.TFCausalLMOutputWithCrossAttentions'>
TFMaskedLMOutput = <class 'transformers.modeling_tf_outputs.TFMaskedLMOutput'>
TFMultipleChoiceModelOutput = <class 'transformers.modeling_tf_outputs.TFMultipleChoiceModelOutput'>
TFNextSentencePredictorOutput = <class 'transformers.modeling_tf_outputs.TFNextSentencePredictorOutput'>
TFQuestionAnsweringModelOutput = <class 'transformers.modeling_tf_outputs.TFQuestionAnsweringModelOutput'>
TFSequenceClassifierOutput = <class 'transformers.modeling_tf_outputs.TFSequenceClassifierOutput'>
TFTokenClassifierOutput = <class 'transformers.modeling_tf_outputs.TFTokenClassifierOutput'>
Tuple      = typing.Tuple
Union      = typing.Union
__builtins__ = <builtins>
__cached__ = '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/__pycache__/modeling_tf_bert.cpython-37.pyc'
__doc__    = ' TF 2.0 BERT model.'
__file__   = '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py'
__loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f82080ff6d0>
__name__   = 'transformers.models.bert.modeling_tf_bert'
__package__ = 'transformers.models.bert'
__spec__   = ModuleSpec(name='transformers.models.bert.modeling_tf_bert', loader=<_frozen_importlib_external.SourceFileLoader objec...igin='/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py')
dataclass  = <function dataclass at 0x7f8289b9cdd0>
get_tf_activation = <function get_tf_activation at 0x7f82080be680>
math       = <module 'math' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/lib-dynload/math.cpython-37m-x86_64-linux-gnu.so'>
np         = <module 'numpy' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/numpy/__init__.py'>
tf         = <module 'tensorflow' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/__init__.py'>
warnings   = <module 'warnings' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/warnings.py'>

/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/modeling_tf_bert.py:38: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """TF general model utils."""
    
    import functools
    import gc
    import inspect
    import json
    import os
    import pickle
    import re
    import warnings
    from collections.abc import Mapping
    from pathlib import Path
    from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Union
    
    import h5py
    import numpy as np
    import tensorflow as tf
    from tensorflow.python.keras import backend as K
    from tensorflow.python.keras.engine import data_adapter
    from tensorflow.python.keras.engine.keras_tensor import KerasTensor
    from tensorflow.python.keras.saving import hdf5_format
    
    from huggingface_hub import Repository, list_repo_files
>   from keras.saving.hdf5_format import save_attributes_to_hdf5_group
E   ModuleNotFoundError: No module named 'keras.saving.hdf5_format'

Any        = typing.Any
Callable   = typing.Callable
Dict       = typing.Dict
K          = <module 'tensorflow.python.keras.backend' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/python/keras/backend.py'>
KerasTensor = <class 'tensorflow.python.keras.engine.keras_tensor.KerasTensor'>
List       = typing.List
Mapping    = <class 'collections.abc.Mapping'>
Optional   = typing.Optional
Path       = <class 'pathlib.Path'>
Repository = <class 'huggingface_hub.repository.Repository'>
TYPE_CHECKING = False
Union      = typing.Union
__builtins__ = <builtins>
__cached__ = '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/__pycache__/modeling_tf_utils.cpython-37.pyc'
__doc__    = 'TF general model utils.'
__file__   = '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/modeling_tf_utils.py'
__loader__ = <_frozen_importlib_external.SourceFileLoader object at 0x7f82080b6890>
__name__   = 'transformers.modeling_tf_utils'
__package__ = 'transformers'
__spec__   = ModuleSpec(name='transformers.modeling_tf_utils', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f82080b6890>, origin='/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/modeling_tf_utils.py')
data_adapter = <module 'tensorflow.python.keras.engine.data_adapter' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py'>
functools  = <module 'functools' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/functools.py'>
gc         = <module 'gc' (built-in)>
h5py       = <module 'h5py' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/h5py/__init__.py'>
hdf5_format = <module 'tensorflow.python.keras.saving.hdf5_format' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/python/keras/saving/hdf5_format.py'>
inspect    = <module 'inspect' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/inspect.py'>
json       = <module 'json' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/json/__init__.py'>
list_repo_files = <bound method HfApi.list_repo_files of <huggingface_hub.hf_api.HfApi object at 0x7f8231bb3310>>
np         = <module 'numpy' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/numpy/__init__.py'>
os         = <module 'os' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/os.py'>
pickle     = <module 'pickle' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/pickle.py'>
re         = <module 're' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/re.py'>
tf         = <module 'tensorflow' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/tensorflow/__init__.py'>
warnings   = <module 'warnings' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/warnings.py'>

/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/modeling_tf_utils.py:39: ModuleNotFoundError

The above exception was the direct cause of the following exception:

    @pytest.mark.skipif(
        not (_is_importable("transformers") and keras_version >= Version("2.6.0")),
        reason="This test requires transformers, which is no longer compatible with Keras < 2.6.0",
    )
    def test_pyfunc_serve_and_score_transformers():
>       from transformers import BertConfig, TFBertModel  # pylint: disable=import-error


tests/keras/test_keras_model_export.py:662: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
<frozen importlib._bootstrap>:1032: in _handle_fromlist
    ???
        fromlist   = ('BertConfig', 'TFBertModel')
        import_    = <built-in function __import__>
        module     = <module 'transformers' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/__init__.py'>
        recursive  = False
        x          = 'TFBertModel'
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/utils/import_utils.py:1022: in __getattr__
    value = getattr(module, name)
        module     = <module 'transformers.models.bert' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/__init__.py'>
        name       = 'TFBertModel'
        self       = <module 'transformers' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/__init__.py'>
/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/utils/import_utils.py:1021: in __getattr__
    module = self._get_module(self._class_to_module[name])
        name       = 'TFBertModel'
        self       = <module 'transformers.models.bert' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/__init__.py'>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <module 'transformers.models.bert' from '/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/transformers/models/bert/__init__.py'>
module_name = 'modeling_tf_bert'

    def _get_module(self, module_name: str):
        try:
            return importlib.import_module("." + module_name, self.__name__)
        except Exception as e:
            raise RuntimeError(
                f"Failed to import {self.__name__}.{module_name} because of the following error (look up to see its"
                f" traceback):\n{e}"
>           ) from e
E           RuntimeError: Failed to import transformers.models.bert.modeling_tf_bert because of the following error (look up to see its traceback):
E           No module named 'keras.saving.hdf5_format'

Expected behavior

Changes made to Keras namespace (the addition of a legacy mode for serialization / deserialization) in this commit: https://github.com/keras-team/keras/commit/c06aa015e900a2029b5b379f374e5d4dc615fcbf will likely require an update for pre-trained huggingface models.

We wanted to make you aware of this if you hadn’t already known about it.

Issue Analytics

  • State:closed
  • Created a year ago
  • Comments:13 (7 by maintainers)

github_iconTop GitHub Comments

2reactions
Rocketknight1commented, Oct 13, 2022

cc @gante and @ydshieh - this might be nothing, but we should remember to do some testing once the RC arrives.

1reaction
Rocketknight1commented, Oct 13, 2022

Hmm, this looks like they changed something about saving in H5 format. The bug report is appreciated, but since the API might be unstable, I think we probably won’t change anything in transformers yet. However, if the bug still occurs in TF 2.11-rc0 then we definitely have a problem and will try to fix things before 2.11 final. Thank you!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Training a BERT-based model causes an OutOfMemory error ...
After looking at the issue rises from the final layer in your model. And I was able to get it to work with...
Read more >
Transfer learning and fine-tuning | TensorFlow Core
In this tutorial, you will learn how to classify images of cats and dogs by using transfer learning from a pre-trained network. A...
Read more >
BERT Explained: A Complete Guide with Theory and Tutorial
Training with BERT can cause out of memory errors. This is usually an indication that we need more powerful hardware — a GPU...
Read more >
pip install transformers==2.1.0 - PyPI
The main breaking change when migrating from pytorch-pretrained-bert to transformers is that every model's forward method always outputs a tuple with various ...
Read more >
Real-Time Natural Language Processing with BERT Using ...
A major problem faced by NLP researchers and developers is scarcity of ... Generating BERT TensorRT engine from pretrained checkpoints.
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Misalignment between documentation and implementation of mBART50 tokenisation for the decoder

Next page

ASR pipeline does not work with openai/whisper on current master