Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

ModuleAttributeError: 'GPT2LMHeadModel' object has no attribute 'backward'

See original GitHub issue

Environment info

transformers version: 4.2.1
Platform: Linux-4.19.0-12-cloud-amd64-x86_64-with-debian-10.6
Python version: 3.7.8
PyTorch version (GPU?): 1.6.0a0+9907a3e (True)
Tensorflow version (GPU?): not installed (NA)
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No(?)

Who can help

albert, bert, GPT2, XLM: @LysandreJik Trainer: @sgugger

To reproduce

Steps to reproduce the behavior:

Set up a TrainingArguments for a GPT2LMHeadModel with the following deepspeed config:

{
    "fp16": {
        "enabled": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "hysteresis": 2,
        "min_loss_scale": 1
    },

    "zero_optimization": {
        "stage": 2,
       "allgather_partitions": true,
       "allgather_bucket_size": 2e8,
       "reduce_scatter": true,
       "reduce_bucket_size": 2e8,
        "overlap_comm": true,
        "contiguous_gradients": true,
        "cpu_offload": false
    },

    "optimizer": {
        "type": "Adam",
        "params": {
            "adam_w_mode": true,
            "lr": 3e-5,
            "betas": [ 0.9, 0.999 ],
            "eps": 1e-8,
            "weight_decay": 3e-7
        }
    },

    "scheduler": {
        "type": "WarmupLR",
        "params": {
            "warmup_min_lr": 0,
            "warmup_max_lr": 3e-5,
            "warmup_num_steps": 500
        }
    }
}

Attempt to call trainer.train().

Expected behavior

Training should begin as expected.

Believed bug location

It would appear that line 1286 in trainer.py actually calls the backward method on the model, not the loss object. I will try rebuilding after fixing that line and seeing if it helps.

Issue Analytics

State:
Created 3 years ago
Comments:10 (6 by maintainers)

Top GitHub Comments

2reactions

stas00commented, Jan 20, 2021

The merged PR closed this report, but should you still have an issue please don’t hesitate to re-open it.

1reaction

stas00commented, Jan 26, 2021

You did nothing wrong, @victorstorchan.

I will propose an update to the installation page so that the distinction is loud and clear.

Top Results From Across the Web

'GPT2Model' object has no attribute 'gradient_checkpointing ...

It throws the following error as mentioned in the question: AttributeError: 'GPT2Model' object has no attribute 'gradient_checkpointing'.

'float' object has no attribute 'backward' - Intellipaat Community

Backward is the function of PyTorch Tensor. When you called the loss.tolist(), you broke each computation graph and you can't backward from ...

[Solved] AttributeError: 'module' object has no attribute

Click here to subscribe - https://www.youtube.com/channel/UCeVMnSShP_Iviwkknt83cww▻Instagram ...

Python AttributeError — What is it and how do you fix it?

AttributeError: '***' object has no attribute '***'What is an AttributeError in Python? What can you do to fix it? When does it happen?...

ecco from jalammar - Coder Social

Rankings across layers: After the model picks an output token, Look back at how each ... ModuleAttributeError: 'Embedding' object has no attribute 'shape'....