Detection and segmentation many small objects

Hello!

Thanks for a this great repo 😃

In the article and on the main page of the repository, there are examples of using YOLACT on images that contain a small number of objects

I tried to train YOLACT on full marked-up image that contain hundreds of objects, as in the example below

For training I use GPU RTX 2080 Ti When I start training, sometimes the following happens:

(TorchEnvLpr) PS C:\Users\Reutov\Repository\yolact> python train.py --config=yolact_resnet50_grans_config --batch_size=1 --num_workers=0
Scaling parameters by 0.12 to account for a batch size of 1.
Per-GPU batch size is less than the recommended limit for batch norm. Disabling batch norm.
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
C:\Users\Reutov\.conda\envs\TorchEnvLpr\lib\site-packages\torch\jit\_recursive.py:182: UserWarning: 'lat_layers' was found in ScriptModule constants,  but it is a non-constant submodule. Consider removing it.
  " but it is a non-constant {}. Consider removing it.".format(name, hint))
C:\Users\Reutov\.conda\envs\TorchEnvLpr\lib\site-packages\torch\jit\_recursive.py:182: UserWarning: 'pred_layers' was found in ScriptModule constants,  but it is a non-constant submodule. Consider removing it.
  " but it is a non-constant {}. Consider removing it.".format(name, hint))
C:\Users\Reutov\.conda\envs\TorchEnvLpr\lib\site-packages\torch\jit\_recursive.py:182: UserWarning: 'downsample_layers' was found in ScriptModule constants,  but it is a non-constant submodule. Consider removing it.
  " but it is a non-constant {}. Consider removing it.".format(name, hint))
Initializing weights...
Begin training!

(TorchEnvLpr) PS C:\Users\Reutov\Repository\yolact>

The training doesn’t begin at all

Sometimes I get OpenCV error:

(TorchEnvLpr) PS C:\Users\Reutov\Repository\yolact> python train.py --config=yolact_resnet50_grans_config --batch_size=1 --num_workers=0
Scaling parameters by 0.12 to account for a batch size of 1.
Per-GPU batch size is less than the recommended limit for batch norm. Disabling batch norm.
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
loading annotations into memory...
Done (t=0.02s)
creating index...
index created!
C:\Users\Reutov\.conda\envs\TorchEnvLpr\lib\site-packages\torch\jit\_recursive.py:182: UserWarning: 'lat_layers' was found in ScriptModule constants,  but it is a non-constant submodule. Consider removing it.
  " but it is a non-constant {}. Consider removing it.".format(name, hint))
C:\Users\Reutov\.conda\envs\TorchEnvLpr\lib\site-packages\torch\jit\_recursive.py:182: UserWarning: 'pred_layers' was found in ScriptModule constants,  but it is a non-constant submodule. Consider removing it.
  " but it is a non-constant {}. Consider removing it.".format(name, hint))
C:\Users\Reutov\.conda\envs\TorchEnvLpr\lib\site-packages\torch\jit\_recursive.py:182: UserWarning: 'downsample_layers' was found in ScriptModule constants,  but it is a non-constant submodule. Consider removing it.
  " but it is a non-constant {}. Consider removing it.".format(name, hint))
Initializing weights...
Begin training!

OpenCV Error: Assertion failed (dsize.area() > 0) in cv::hal::resize, file C:\ci\opencv_1512688052760\work\modules\imgproc\src\resize.cpp, line 2961
Traceback (most recent call last):
  File "train.py", line 504, in <module>
    train()
  File "train.py", line 270, in train
    for datum in data_loader:
  File "C:\Users\Reutov\.conda\envs\TorchEnvLpr\lib\site-packages\torch\utils\data\dataloader.py", line 435, in __next__
    data = self._next_data()
  File "C:\Users\Reutov\.conda\envs\TorchEnvLpr\lib\site-packages\torch\utils\data\dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\Reutov\.conda\envs\TorchEnvLpr\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\Reutov\.conda\envs\TorchEnvLpr\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\Reutov\Repository\yolact\data\coco.py", line 94, in __getitem__
    im, gt, masks, h, w, num_crowds = self.pull_item(index)
  File "C:\Users\Reutov\Repository\yolact\data\coco.py", line 159, in pull_item
    {'num_crowds': num_crowds, 'labels': target[:, 4]})
  File "C:\Users\Reutov\Repository\yolact\utils\augmentations.py", line 688, in __call__
    return self.augment(img, masks, boxes, labels)
  File "C:\Users\Reutov\Repository\yolact\utils\augmentations.py", line 55, in __call__
    img, masks, boxes, labels = t(img, masks, boxes, labels)
  File "C:\Users\Reutov\Repository\yolact\utils\augmentations.py", line 158, in __call__
    masks = cv2.resize(masks, (width, height))
cv2.error: C:\ci\opencv_1512688052760\work\modules\imgproc\src\resize.cpp:2961: error: (-215) dsize.area() > 0 in function cv::hal::resize

I assume that there is a limit on the number of placed objects in the frame that YOLACT can be trained on. Also I think there is a problem in the markup. But I want to ask, how can I solve this problem And how effective is it to use YOLACT to detect and segment many small objects in the images

Thanks in advance for answer 😃

[VABer-dv]

if number of objects at the image is more than 512 open cv resize method crushes. To solve this problem modify the line masks = cv2.resize(masks, (width, height)) at /yolact-master/utils/augmentations.py, (~ line 162):

cv_limit = 512
if masks.shape[2] <= cv_limit:
    masks = cv2.resize(masks, (width, height))
else:
    # split masks array on batches with max size 512 along channel axis, resize and merge them back
    masks = np.concatenate([cv2.resize(masks[:, :, i:min(i + cv_limit, masks.shape[2])], (width, height))
                            for i in range(0, masks.shape[2], cv_limit)], axis=2)

[tehkillerbee]

@susanin1970 Sorry for late reply, I detect around 1000 in my dataset. Each frame is scaled to 768x768.

The inference speed depends on the number of objects but it is usually fast enough for my requirements (1000mS or less)