RuntimeError: CUDA error: device-side assert triggered while eval
See original GitHub issueDescribe the bug
I trained a distilbert model for Classification. Now when I try to use the model to eval or predict I get a RuntimeError: CUDA error: device-side assert triggered
To Reproduce
Steps to reproduce the behavior:
Train a model using the params ClassificationModel('distilbert', 'distilbert-base-uncased', num_labels=len(df_bert_train['labels'].unique()), args={'reprocess_input_data': True, 'overwrite_output_dir': True, 'max_seq_length': 64, 'train_batch_size':16, 'fp16':False, 'num_train_epochs': 10})
Desktop (please complete the following information):
- Linux
Issue Analytics
- State:
- Created 4 years ago
- Comments:17 (7 by maintainers)
Top Results From Across the Web
CUDA Error: Device-Side Assert Triggered: Solved | Built In
A CUDA error: device-side assert triggered is an error that's often caused when you either have inconsistency between the number of labels and ......
Read more >RuntimeError: CUDA error: device-side assert triggered
When I freshly train the Token Classification model (DistilBertForTokenClassification) and run a prediction for a single sentence that I ...
Read more >Pytorch 1.6.0 - RuntimeError: CUDA error: device-side assert ...
When I run this function: def evaluate(model, dataloader, calc_loss=False): ''' Helper function to get classification accuracy and ...
Read more >CUDA error: device-side assert triggered... When train parallel ...
I am using this code: pytorch-cifar100/train.py at master · weiaicunzai/pytorch-cifar100 · GitHub I am getting this error, when I increase ...
Read more >RuntimeError: CUDA error: device-side assert triggered when ...
RuntimeError : CUDA error: device-side assert triggered. when trying to run either of learn.lr_find(suggestions=True). or learn.validate().
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
@ThilinaRajapakse Hi, Just letting you know that I faced the same error ‘RuntimeError: CUDA error: device-side assert triggered’ today with the latest version(0.22.0). After trying a lot of things to overcome it, I tried a lower version of simpletransformers(0.20.0) in the end and it worked. I was trying this example: https://towardsdatascience.com/simple-transformers-introducing-the-easiest-bert-roberta-xlnet-and-xlm-library-58bf8c59b2a3
Thanks a lot for the great wrapper 😃
Guys use CPU (it means use_cuda = False) if u faces this situation with GPU. If problem in data u can find better exeption output, and solve problem