Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Training Time

See original GitHub issue

Hi,

Thanks for sharing this great repo with the community. I have a question about the training time. I run your code on a single node with 4 GPUs (V100) with the ViT_S_16 backbone and according to the log, each batch takes around 1.02 sec (see below for an example log). When I scale this setting up to 16 GPUs this will take me around 36 hours to run for 100 epochs. However, according to Table 8 in the paper, this setting should take around 24 hours. I would like to know if I am missing something here and it would be great if you can please share the details of the time for each batch as well.

Epoch: [0/100]  [4400/5004]  eta: 0:10:15  loss: 5.507585 (7.369033)  lr: 0.000044 (0.000022)  wd: 0.040068 (0.040023)  time: 1.022257  data: 0.000111  max mem: 19499
Epoch: [0/100]  [4410/5004]  eta: 0:10:05  loss: 5.519022 (7.364826)  lr: 0.000044 (0.000022)  wd: 0.040069 (0.040023)  time: 1.024017  data: 0.000109  max mem: 19499

Issue Analytics

State:
Created 2 years ago
Reactions:1
Comments:6 (3 by maintainers)

Top GitHub Comments

2reactions

mathildecaron31commented, Jun 8, 2021

Sure, do not hesitate to reopen if you still have issue.

2reactions

fatemeh-slhcommented, Jun 8, 2021

Great! Thanks for the clarification. My run was with --use_fp16 false, so I think it should be the same as yours with mixed precision.

Top Results From Across the Web

training time Definition | Law Insider

Training hour means at least 50 minutes of actual learning, including, but not limited to, time devoted to lecture, learning activities, small group...

Training time - Oxford Reference

The rate at which exercise is to be accomplished during an exercise or work interval in an interval training programme. A training time...

29 CFR § 553.226 - Training time. - Law.Cornell.Edu

(b) While time spent in attending training required by an employer is normally considered compensable hours of work, following are situations where time...

Predicting Training Time Without Training

Could you predict the time it takes for a network to converge, before even starting to train it? We look to efficiently estimate...

Training Time: What Time Is Best to Work Out?

The perfect training time for you; Training in the morning; Training at lunchtime; Training in the evening. What does your biological clock say?...