Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Training step is too slow

See original GitHub issue

Hi, Thank you for your code. As I go deeply into this code, I found the training step is particular slow. The problem here (I guess) is the dataset construction processing, where too much functions (e.g., padding sequences, getting history) are implemented in the __get_item__. I wonder, have you tried to wrap these functions in the __init__ function? This might lead to more memory consuming but will absolutely accelerate the training process. Thanks.

Issue Analytics

State:
Created 4 years ago
Comments:6 (4 by maintainers)

Top GitHub Comments

1reaction

guoyang9commented, Oct 15, 2019

Hi @abhshkdz @kdexd ,thanks for your reply. I will try to divert the padding function from __getitem__ to __init__, and check the memory consumption. I will get back to you later. Thanks! 😃

0reactions

abhshkdzcommented, May 28, 2020

@shubhamagarwal92 Thanks for the suggestions! Both make sense to me. If you could send in a pull request, that’d be great, thanks!

Top Results From Across the Web

Training speed is too slow #959 - facebookresearch/fairscale

Hello, I am conducting multi-GPU training using fairseq's ShardedDDP and OSS. It is impressive that all gpus consume almost the same memory.

Slow model training is very frustrating - Fast.ai forums

The study of one epoch takes more than 50 minutes. This is very frustrating and does not encourage experimentation with model training. Do...

Optimizer.step() is very slow - autograd - PyTorch Forums

I am training a Densely Connected U-Net model on CT scan data of dimension 512x512 for segmentation task. My network training was very...

Why is it so slow? | Apple Developer Forums

I run the following code using tensorflow-macos and tensorflow_macos, respectively. import tensorflow as tf mnist = tf.keras.datasets.mnist (x_train, y_train), ...

Training Become very slow Yolov4 - TAO Toolkit

Hi,. I am trying to train yolov4 using custom dataset in which i have only one class. but training process running very slow...