[deepspeed checkpointing] AttributeError: 'NoneType' object has no attribute 'numel'
See original GitHub issueSo I took a public GPT-2 class implementation (not Megatron-LM) and I added deepspeed checkpointing to it for all 48 layers. In my train script for this class, I added the following line:
deepspeed.checkpointing.configure(mpu_=None, deepspeed_config=args.deepspeed_config)
My deepspeed config JSON is as follows:
{
"train_batch_size": 128,
"gradient_accumulation_steps": 8,
"gradient_clipping": 1.0,
"optimizer": {
"type": "adam",
"params": {
"lr": 6.25e-5
}
},
"fp16": {
"enabled": true
},
"zero_optimization": {
"stage": 2,
"cpu_offload": true,
"contiguous_gradients": true,
"overlap_comm": false,
"allgather_bucket_size": 500000000
},
"activation_checkpointing": {
"partition_activations": true,
"contiguous_memory_optimization": true,
"number_checkpoints": 48,
"cpu_checkpointing": true
}
}
When I try running my script, I get the following error:
File "/path/to/my/modeling_gpt2.py", line 221, in forward
encoder_attention_mask)
File "/usr/local/lib/python3.6/dist-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 582, in checkpoint
return CheckpointFunction.apply(function, *args)
File "/usr/local/lib/python3.6/dist-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 376, in forward
partition_size = get_partition_size(item)
File "/usr/local/lib/python3.6/dist-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 275, in get_partition_size
size = item.numel()
AttributeError: 'NoneType' object has no attribute 'numel'
Any ideas what’s going on?
Issue Analytics
- State:
- Created 3 years ago
- Comments:23 (10 by maintainers)
Top Results From Across the Web
AttributeError: 'NoneType' object has no attribute 'n_feature'
I have trained a model with state dictionary and now i want to test a model on testing set. The problem is that...
Read more >Apache Airflow Attributeerror: 'Nonetype' Object Has No ...
Summary: NoneType attribute error occurs when the type of object being referenced is ... [deepspeed checkpointing] AttributeError: 'NoneType' object has no ...
Read more >Stoke
Module such that it has a forward method if not isinstance(model, torch.nn.Module): raise TypeError( f"Stoke -- Model is not of type torch.nn.
Read more >1.2.2 PDF
The above checkpoint contains all the arguments needed to init the ... The trainer object will also set an attribute interrupted to True...
Read more >AttributeError: 'NoneType' object has no attribute 'data'
AttributeError : 'NoneType' object has no attribute 'data'. I guess it cause by my custom layer and it may set up some weight...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
@ShadenSmith thanks for the fix! Will test this out later this week and revert back to you!
Hey @g-karthik , happy new year and thanks for the ping! I hope you had a nice holiday. I spent some time away to focus on python scalability 😃.
Thanks for the deep dive on debugging this issue. I think you’re right. DeepSpeed’s activation checkpointing should work as a drop-in replacement in the simple case without partitioning, etc.
I don’t know if we’ve done any benchmarking beyond at large scale with model parallelism where it’s critical for huge activations. Maybe @samyam knows more.