Mosaic Augmentation for Object Detection

Mosaic augmentation for object detection is used in Yolo-V4 literature, FR. I’m not sure if it’s used for classification tasks there. I was wondering if it’s possible to do so for the image classification task. Here are two issues:

• References. Not sure if it’s ever used for classification only in any literature.
• Creation of class labels.

For creating class labels, here is one possible solution, described HERE; using Dirichlet distribution.

# for 2 images. Equivalent to λ and (1-λ)
>>> np.random.dirichlet((1, 1), 1)  
array([[0.92870347, 0.07129653]])  

>>> np.random.dirichlet((1, 1, 1), 1)  # for 3 images.
array([[0.38712673, 0.46132787, 0.1515454 ]])

>>> np.random.dirichlet((1, 1, 1, 1), 1)  # for 4 images.
array([[0.59482542, 0.0185333 , 0.33322484, 0.05341645]])

As mosaic takes 4 images.

---

update:

Mosaic augmentation for classification did use in literature. Please check: https://github.com/keras-team/keras-cv/issues/250#issuecomment-1130256698

[AlexeyAB]

We set the ground truth probability in proportion to the area occupied by each sample: https://github.com/AlexeyAB/darknet/blob/4ee3be7e68fb9c7eda5cc390e47e59f01e40dded/src/data.c#L1913-L1942 So if areas for: Car=10%, Table=20%, Cat=30%, Dog = 40%, then we set labels: Car=0.1, Table=0.2, Cat=0.3, Dog=0.4.

We have not tried using it in any other way.

Mosaic for Classifier reduces Top5-error from 6.0% to 5.5% (-10 relative %) on Imagenet-1k, and it works better than CutMix/MixUp, Tables 2 and 3: https://arxiv.org/abs/2004.10934

---

We used Mosaic data augmentation:

• for Classifier: YOLOv4
• for Detector: YOLOv4, Scaled-YOLOv4, YOLOv5, YOLOR: https://github.com/WongKinYiu/yolor

We didn’t use Mosaic for Classifier in (Scaled-YOLOv4, YOLOv5, YOLOR), just because they don’t need Classification or any pre-trained weights. We train these Detector from scratch to achieve the best accuracy/speed ratio.

It was originally introduced in the YOLOv4 paper: https://arxiv.org/abs/2004.10934

Some later work has used Mosaic:

• https://arxiv.org/abs/2004.12432
• https://arxiv.org/abs/2004.12178
• https://openaccess.thecvf.com/content/CVPR2021/html/Wang_Scaled-YOLOv4_Scaling_Cross_Stage_Partial_Network_CVPR_2021_paper.html
• https://arxiv.org/abs/2105.04206 …

[innat]

Not sure but some suspicious feelings about the creation of mosaic_x/y, s1-s4, center_x, center_y. Can you check this and this implementation, it might give you some pointers.