Asian training dataset(from glint) discussion.

1. Download dataset from http://trillionpairs.deepglint.com/data (after signup). msra is a cleaned subset of MS1M from glint while celebrity is the asian dataset.
2. Generate lst file by calling src/data/glint2lst.py. For example:

python glint2lst.py /data/glint_data msra,celebrity > glint.lst

or generate the asian dataset only by:

python glint2lst.py /data/glint_data celebrity > glint_cn.lst

1. Call face2rec2.py to generate .rec file.
2. Merge the dataset with existing one by calling src/data/dataset_merge.py without setting param model which will combine all IDs from those two datasets.

Finally you will get a dataset contains about 180K IDs.

Use src/eval/gen_glint.py to prepare test feature file by using pretrained insightface model.

You can also post your private testing results here.

[meanmee]

有格林的人在吗，我是楼下比特大陆的，下载速度太慢了，可以直接去楼上直接拷贝吗？

[JianbangZ]

After test, this dataset is pretty clean, but still containing 0.3%~0.8% noise. Also, we found their ms1m and Asian parts still have about 15-30 overlaps, though I guess it doesn’t matter when the scale is already so large. Another findings is that this dataset suffers long tail a lot. Take the asian part for example, only 18K identites out of 10K have over 25 images per class, and only few thousand identities have over 60 images.