How to Create a Multi-inputs Convolutional Neural Network Model for Images Classification?

I am beginner in deep learning and I hope if you help me to solve my issue.

I want to create a CNN model that takes two inputs of images and produce one output which is the class of the two images. The model takes one image from dataset type1 and one image from dataset type2. I have two datasets: type1 and type2, and each dataset contains the same number of classes, but the number of images in each class in the dataset type1 is higher than the number of images in each class in the dataset type2. The following is the structure of the datasets.

The model should take one image from Type1 dataset and one image from Type2 dataset and then classify these images to one class (ClassA or ClassB or------).

Type1 dataset
|Train
              |ClassA
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassB
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassC
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassD
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
       ----------------
|Validate
            -----------
|Test
           --------------

Type2 dataset
|Train
              |ClassA
                             |image1
                             |image2
                            -----
              |ClassB
                             |image1
                             |image2
                            -----
              |ClassC
                             |image1
                             |image2
                            -----
              |ClassD
                             |image1
                             |image2
                            -----
       ----------------
|Validate
            -----------
|Test
           --------------

So, I want to create a model that inputs two images (from type 1 and 2), as long as they are from the same class. Also, I want each image from type1 be paired with every images from type2 from the same class. How can I do this???

The code:

in1 = Input(...)  
 x = Conv2D(...)(in1)
--------
--------
 out1 = Dense(...)(x)  

 in2 = Input(...)  
x = Conv2D(...)(in2)
--------
--------
out2 = Dense(...)(x)  

concatenated_layer = concatenate([out1, out2])  # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(concatenated_layer)
modal= Model(inputs=[in1, in2], outputs=[output_layer])



input_imgen = ImageDataGenerator(rescale = 1./255, 
                                   shear_range = 0.2, 
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator()



def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label


inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_iris_data,
                                           dir2=train_face_data,
                                           batch_size=32,
                                           img_height=224,
                                           img_width=224)       

testgenerator=generate_generator_multiple(generator=test_imgen,
                                          dir1=valid_iris_data,
                                          dir2=valid_face_data,
                                          batch_size=1,
                                          img_height=224,
                                          img_width=224)  

[nadaalay]

The following codes would be helpful, try them and do not hesitate if you have any questions.

This the code of image generator for three inputs

train_size = 5300

test_size = 1802

 

input_imgen_1 = ImageDataGenerator(

                                  rotation_range=10,

                                  shear_range=0.2,

                                  zoom_range=0.1,

                                  width_shift_range=0.1,

                                  height_shift_range=0.1

                                 )

 

input_imgen_2 = ImageDataGenerator(

                                  rotation_range=10,

                                  shear_range=0.2,

                                  zoom_range=0.2,

                                  width_shift_range=0.1,

                                  height_shift_range=0.1,

                                  horizontal_flip=True

                                 )

 

input_imgen_3 = ImageDataGenerator(

                                  rotation_range=10,

                                  shear_range=0.2,

                                  zoom_range=0.1,

                                  width_shift_range=0.1,

                                  height_shift_range=0.1,

                                  horizontal_flip = True

                                  )

                                                            

 

test_imgen = ImageDataGenerator()

 

bs = 64

 

# Create generator for multiple inputs

 

def generate_generator_multiple(1_generator, 2_generator,  3_generator, dir1, dir2, dir3, batch_size, img_height,img_width):

    

    

    gen_1 = 1_generator.flow_from_directory(dir1,

                                          target_size = (img_height,img_width),

                                          class_mode = 'categorical',

                                          batch_size = batch_size,

                                          shuffle=False, 

                                          seed=7)

    

    gen_1.reset()

   

    gen_2 = 2_generator.flow_from_directory(dir2,

                                          target_size = (img_height,img_width),

                                          class_mode = 'categorical',

                                          batch_size = batch_size,

                                          shuffle=False, 

                                          seed=7)

   

    gen_2.reset()

 

    gen_3 = 3_generator.flow_from_directory(dir3,

                                          target_size = (img_height,img_width),

                                          class_mode = 'categorical',

                                          batch_size = batch_size,

                                          shuffle=False, 

                                          seed=7)

   

    gen_3.reset()

    

    while True:

            1_ = gen_1.next()    

            2_ = gen_2.next()

            3_ = gen_3.next()

            yield [1_[0], 2_[0], 3_[0]], 2_[1]  #Yield images and their label

            

            

multi_train_generator=generate_generator_multiple(1_generator=input_imgen_1,

                                           2_generator=input_imgen_2,

                                           3_generator=input_imgen_3,

                                           dir1=train_1_data2,

                                           dir2=train_2_data,

                                           dir3=train_3_data2,

                                           batch_size=bs,

                                           img_height=224,

                                           img_width=224)       

     

multi_valid_generator=generate_generator_multiple(1_generator=test_imgen,

                                           2_generator=test_imgen,

                                           3_generator=test_imgen,

                                           dir1=valid_1_data2,

                                           dir2=valid_2_data,

                                           dir3=valid_3_data2,

                                           batch_size=bs,

                                           img_height=224,

                                           img_width=224) 

 

multi_test_generator=generate_generator_multiple(1_generator=test_imgen,

                                          2_generator=test_imgen,

                                          3_generator=test_imgen,

                                          dir1=test_1_data2,

                                          dir2=test_2_data,

                                          dir3=test_3_data2,

                                          batch_size=1,

                                          img_height=224,

                                          img_width=224)

This the code of model of three inputs :

# Merge the fully connected layers of the two models
concatenated_layer1 = concatenate([fc12, fc22, fc32])  
dropout = Dropout(0.4, name='dropout1')(concatenated_layer1)

output = Dense(no_classes, activation='softmax', name='prediction4')(dropout)
multimodal_model1 = Model(inputs=[input_layer1, input_layer2, input_layer3], outputs=[output])

[Aditijain114]

how to get labels/classes from these generator multi_train_generator.labels()