Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

[deepspeed checkpointing] AttributeError: 'NoneType' object has no attribute 'numel'

See original GitHub issue

So I took a public GPT-2 class implementation (not Megatron-LM) and I added deepspeed checkpointing to it for all 48 layers. In my train script for this class, I added the following line:

deepspeed.checkpointing.configure(mpu_=None, deepspeed_config=args.deepspeed_config)

My deepspeed config JSON is as follows:

{
  "train_batch_size": 128,
  "gradient_accumulation_steps": 8,
  "gradient_clipping": 1.0,
  "optimizer": {
    "type": "adam",
    "params": {
      "lr": 6.25e-5
    }
  },
  "fp16": {
    "enabled": true
  },
  "zero_optimization": {
    "stage": 2,
    "cpu_offload": true,
    "contiguous_gradients": true,
    "overlap_comm": false,
    "allgather_bucket_size": 500000000
  },

  "activation_checkpointing": {
    "partition_activations": true,
    "contiguous_memory_optimization": true,
    "number_checkpoints": 48,
    "cpu_checkpointing": true
  }

}

When I try running my script, I get the following error:

  File "/path/to/my/modeling_gpt2.py", line 221, in forward
    encoder_attention_mask)
  File "/usr/local/lib/python3.6/dist-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 582, in checkpoint
    return CheckpointFunction.apply(function, *args)
  File "/usr/local/lib/python3.6/dist-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 376, in forward
    partition_size = get_partition_size(item)
  File "/usr/local/lib/python3.6/dist-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 275, in get_partition_size
    size = item.numel()
AttributeError: 'NoneType' object has no attribute 'numel'

Any ideas what’s going on?

@tjruwase @ShadenSmith

Issue Analytics

State:
Created 3 years ago
Comments:23 (10 by maintainers)

Top GitHub Comments

1reaction

g-karthikcommented, Jan 12, 2021

@ShadenSmith thanks for the fix! Will test this out later this week and revert back to you!

1reaction

ShadenSmithcommented, Jan 6, 2021

Hey @g-karthik , happy new year and thanks for the ping! I hope you had a nice holiday. I spent some time away to focus on python scalability 😃.

Thanks for the deep dive on debugging this issue. I think you’re right. DeepSpeed’s activation checkpointing should work as a drop-in replacement in the simple case without partitioning, etc.

I don’t know if we’ve done any benchmarking beyond at large scale with model parallelism where it’s critical for huge activations. Maybe @samyam knows more.

Top Results From Across the Web

AttributeError: 'NoneType' object has no attribute 'n_feature'

I have trained a model with state dictionary and now i want to test a model on testing set. The problem is that...

Apache Airflow Attributeerror: 'Nonetype' Object Has No ...

Summary: NoneType attribute error occurs when the type of object being referenced is ... [deepspeed checkpointing] AttributeError: 'NoneType' object has no ...

Stoke

Module such that it has a forward method if not isinstance(model, torch.nn.Module): raise TypeError( f"Stoke -- Model is not of type torch.nn.

1.2.2 PDF

The above checkpoint contains all the arguments needed to init the ... The trainer object will also set an attribute interrupted to True...

AttributeError: 'NoneType' object has no attribute 'data'

AttributeError : 'NoneType' object has no attribute 'data'. I guess it cause by my custom layer and it may set up some weight...