Could anyone please show me the whole process to make your own dataset?

I am trapped in the big confusion on how to make a dataset that can be used to train arcface model out of my own face data directory, in the structure as below:

.
├── PersonA
│   ├── 001.jpeg
│   └── 002.jpeg
├── PersonB
│   ├── 001.jpeg
│   └── 002.jpeg
└── PersonC
    ├── 001.jpeg
    ├── 002.jpeg
    └── 003.jpeg

3 directories, 7 files

It would be better to show me the steps in detail in the following form:

1. …
2. …
3. … …

Much appreciation and thanks!

[Talgin]

Hi @YokkaBear, I think you have to search other places or read papers of LFW, YTF dataset creators to get an understanding of how datasets are collected and refined, the issues here are mostly about insightface project and not about some dataset, because they’ve used merged or publicly available datasets (you can read their paper (https://arxiv.org/pdf/1801.07698.pdf)). But to clear some points you can follow the next steps (regarding this project):

1. You have to collect images of people: one folder for images of one person (the images can be of different quality, having different lightning conditions, from different cameras, sources, etc.);
2. You have to refine the folders you have: delete/merge duplicate folders (having the same person images), because it will affect the accuracy of your training;
3. After first two steps you should have one folder with subfolders having the structure like you provided above;
4. Align your dataset with the size of the images 112*112: for this you can use facenet’s alignment script (https://github.com/davidsandberg/facenet/blob/master/src/align/align_dataset_mtcnn.py) or if it doesn’t work for you, you can use my script (https://github.com/Talgin/preparing_data/blob/master/align_dataset_mtcnn_v1.py), which is the revised version of facenet script - some libraries were out of date, so I had to change them.
5. Then… you have to divide your dataset into Train, Validation, Test sets (we used 80%,10%,10% ratio);
6. Then you have to create .lst file of you dataset: you can use our script (https://github.com/Talgin/preparing_data/blob/master/insightface_pairs_gen_v1.py);
7. Generate .rec and .idx files using: face2rec2.py (insightface/src/data/face2rec2.py);
8. Generate pairs.txt;
9. Using pairs.txt generate .bin file using lfw2pack.py (https://github.com/deepinsight/insightface/blob/master/src/data/lfw2pack.py) - bin file is needed for validation;
10. Collect .rec, .idx, .bin, property files into one folder and start training.

You can also read some info on my page (I’ll rewrite/restructure it in 1-2 weeks): https://github.com/Talgin/preparing_data

P.S. Codes provided above are my publicly available codes and codes shared in github by other people.

Good luck! 😃

[AAI-Armughan-Shahid]

@YokkaBear See this link . Might help u out https://github.com/VictorZhang2014/facenet/blob/master/mydata/generate_pairs.py The following link is better and tested: https://github.com/armanrahman22/Facial-Recognition-and-Alignment/blob/master/facenet_sandberg/generate_pairs.py