Cannot get 1.1771 eer (a large gap) with the pretrained model in README

In the README, it mentions that:

A larger model trained with online data augmentation, described in [2], can be downloaded from here.

The following script should return: EER 1.1771.

python ./trainSpeakerNet.py --eval --model ResNetSE34V2 --log_input True --encoder_type ASP --n_mels 64 --trainfunc softmaxproto --save_path exps/test --eval_frames 400  --initial_model baseline_v2_ap.model

However, with the pretrained baseline_v2_ap model and same script, I can only get EER 1.7073, MinDCF 0.12625. Is there a mistake in your model or code?

[msh9184]

As in the comment of @chmod740, it seems to be a BC (breaking changes) issue depending on the torch version.

From the torch version 1.10.0, the torch.nn.functional.pairwise_distance function computes the pairwise distance between vectors. (Unlike the batchwise pairwise distance as in the version 1.7.0 ~ 1.9.1)

It seems to be compatible if you use torch.cdist function, for instance,

223    #dist = F.pairwise_distance(ref_feat.unsqueeze(-1), com_feat.unsqueeze(-1).transpose(0,2)).detach().cpu().numpy();
224    dist = torch.cdist(ref_feat.reshape((num_eval, -1)), com_feat.reshape((num_eval, -1))).detach().cpu().numpy();

https://github.com/clovaai/voxceleb_trainer/blob/9481143015109554840dcdd4a9bbefce897114a2/SpeakerNet.py#L223

Result using torch.nn.functional.pairwise_distance() in pytorch version 1.10

Result using torch.cdist() in pytorch version 1.10

[chmod740]

I also found the same problem on the master branch, but I have successfully reproduced this result a year or two ago. I compared the two codes and found some differences. First of all, you should switch the version of pytorch to 1.7.1, because I have successfully reproduced in this version of pytorch, I compared pytorch 1.7.1 and 1.10.0, the torch.nn.functional.pairwise_distance used to calculate the score function has changed. In the second step, you can try to add the following line of code to the forward method in ResNetSE34V2.py, because the wavfile module is used to read the speech instead of the soundfile module in the previous code. When read with the wavfile module, the voice is not normalized.

    def forward(self, x):
        x *= 32768.0
        with torch.no_grad():
            with torch.cuda.amp.autocast(enabled=False):
                x = self.torchfb(x)+1e-6
                if self.log_input: x = x.log()
                x = self.instancenorm(x).unsqueeze(1)