Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

training on voxceleb1+2 is very slow?

See original GitHub issue

Dear all: I noticed that when training on voxceleb1+2, it will take me up to 25 hours for single epoch. and even with ddp on 4 gpu cards, the training speed does not reduce at all. I guess the cpu is the bottleneck? anyone has the same phenomena? thank you.

7%|████████▎                                        | 16569/241547 [1:45:07<25:09:56,  2.48it/s, train_loss=13

Issue Analytics

State:
Created 2 years ago
Comments:35 (4 by maintainers)

Top GitHub Comments

1reaction

TParcolletcommented, Oct 28, 2021

On LIA clusters you can increase this value as our nfs is pretty terrible. (4-8) to be tested. It’s the number of threads used to load the data. Having 100% usage is pretty normal as well, but the training should not be that long.

0reactions

Adel-Moumencommented, Oct 6, 2022

Hello,

It seems that the issue has been answered. Therefore, I am closing this issue. If you have any other questions, please feel free to reopen the issue! Thanks. 🙂

Top Results From Across the Web

Build a SRE Challenge System: Lessons from VoxSRC 2022 ...

Abstract. Different speaker recognition challenges have been held to assess the speaker verification system in the wild and probe the performance limit.

VoxCeleb

We host a VoxCeleb Speaker Recognition Challenge (VoxSRC) at Interspeech every year. This is a speaker recognition challenge held on the VoxCeleb datasets!...

VoxCeleb-PT – a dataset for a speech processing course

PDF | This paper introduces VoxCeleb-PT, a small dataset of voices of Portuguese celebrities that can be used as a language-specific ...

CNN WITH PHONETIC ATTENTION FOR TEXT ... - Microsoft

ABSTRACT. Text-independent speaker verification imposes no constraints on the spoken content and usually needs long observations.

arXiv:2009.14153v1 [eess.AS] 29 Sep 2020

Clova Baseline System for the VoxCeleb Speaker Recognition Challenge 2020 ... During training, we use a fixed length 2-second temporal seg-.