Error occurred when finalizing GeneratorDataset iterator | Trying to run image segmentation on custom dataset

Hello, I am trying to train the segmentation model on my custom dataset, which has two classes 1: Copper and 2:Belmouth. I used generate_gt_segmentation.py codes to convert JSON to .npy file. While running the training process on google colab tf version 2.4.1 getting following error:

tensorflow.python.framework.errors_impl.ResourceExhaustedError:  OOM when allocating tensor with shape[4,16,720,1280] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[node fcn/ResNet56v2/conv2d_18/Conv2D (defined at fcn-12.3.1.py:154) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
 [Op:__inference_train_function_16535]

Function call stack:
train_function

2021-04-05 18:46:10.106657: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
	 [[{{node PyFunc}}]]

I also noticed that data_generator.py code give the # of class = 4, which is wrong also number of classes supposed to be 3. How may I solve the issue?

[roatienza]

Hi, The short answer is - unfortunately, it is not straightforward. You must change the image variable:

https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/5d97af8ef5d8b32bda38bdaeb9137fb4f49674a1/chapter12-segmentation/utils/generate_gt_segmentation.py#L51

to something like: image = np.ones((w, h, n_classes), dtype="uint8")

In your case, (w,h) is your image dim and n_classes = 3, or 2 classes + 1 bg class.

Assuming you have the polygon annotation using via tool, the mask will be set on the bg variable:

https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/5d97af8ef5d8b32bda38bdaeb9137fb4f49674a1/chapter12-segmentation/utils/generate_gt_segmentation.py#L67

Rowel

[roatienza]

Hi, For the OOM problem, pls adjust the batch size. The default is 4 for big memory GPUs like V100. For smaller GPUs like 1060Ti, pls use 1. For the number of classes, the data generator reads the npy file and automatically computes the number of unique classes based on the GT mask. https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras/blob/5d97af8ef5d8b32bda38bdaeb9137fb4f49674a1/chapter12-segmentation/data_generator.py#L51

Therefore, if you want to make n_classes=3 (inc bg), you have to prepare dataset with key=image_file_name and value=tensor_mask where the dim of tensor_mask is image_dim x 3. The tensor_mask defines the segmentation mask of each object in the image.