Simple Keras model fails to categorize after conversion to kmodel

I have a small model somewhat based on MNIST. The Keras definition is:

input_shape = (40, 40, 1)
num_classes = 13

inputs = Input(shape=input_shape)
x = Conv2D(32, kernel_size=(3, 3), name='conv1', activation="relu")(inputs)
x = Conv2D(32, kernel_size=(3, 3), name='conv2', activation="relu")(x)
x = MaxPooling2D(pool_size=(2, 2), name='maxPool1')(x)
x = Conv2D(64, kernel_size=(3, 3), name='conv3', activation="relu")(x)
x = Conv2D(64, kernel_size=(3, 3), name='conv4', activation="relu")(x)
x = MaxPooling2D(pool_size=(2, 2), name='maxPool2')(x)
x = Flatten(name='flatten')(x)
x = Dropout(0.5, name='dropout')(x)
output = Dense(num_classes, name='output', activation="softmax")(x)

The model summary shows:

Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 40, 40, 1)]       0         
_________________________________________________________________
conv1 (Conv2D)               (None, 38, 38, 32)        320       
_________________________________________________________________
conv2 (Conv2D)               (None, 36, 36, 32)        9248      
_________________________________________________________________
maxPool1 (MaxPooling2D)      (None, 18, 18, 32)        0         
_________________________________________________________________
conv3 (Conv2D)               (None, 16, 16, 64)        18496     
_________________________________________________________________
conv4 (Conv2D)               (None, 14, 14, 64)        36928     
_________________________________________________________________
maxPool2 (MaxPooling2D)      (None, 7, 7, 64)          0         
_________________________________________________________________
flatten (Flatten)            (None, 3136)              0         
_________________________________________________________________
dropout (Dropout)            (None, 3136)              0         
_________________________________________________________________
output (Dense)               (None, 13)                40781     
=================================================================
Total params: 105,773
Trainable params: 105,773
Non-trainable params: 0

My dataset uses about 4800 40x40 bmp files for training and 700 for validation. I typically converges between 25-35 epochs at around 98-99%

I have been able to convert this to tflite model and run the inference using the tflite interpreter with slightly lower accuracy/confidence levels, but very similar overall performance.

I’ve converted it to kmodel v4 using nncase, but running inference using ncc infer produces properly shaped .bin files (13 floats) but there is no apparent correlation to the input data.

I’m using tensorflow 1.15 (could not get 2.x tflite model working with ncc. I’ve tried ncc both from the May release and the most recent CI build. Running on Windows. My ncc command line and its output is:

ncc.exe compile ..\tflite\model1.tflite ..\tflite\model1.kmodel --input-format tflite --dataset path\to\test --dump-ir --dump-weights-range

1. Import graph...
2. Optimize Pass 1...
Dump weights range ...
Conv2D_0{-0.212548, 0.195113}
Conv2D_1{-0.712639, 0.458298}
Conv2D_2{-0.663264, 0.535185}
Conv2D_3{-0.732874, 0.418174}
3. Optimize Pass 2...
4. Quantize...
  4.1. Add quantization checkpoints...
  4.2. Get activation ranges...
  Plan buffers...
  Run calibration...
  [==================================================] 100% 27.505s
  4.5. Quantize graph...
5. Lowering...
6. Optimize Pass 3...
7. Generate code...
  Plan buffers...
  Emit code...
Working memory usage: 217728 B

SUMMARY
INPUTS
0       Input_0 1x1x40x40
OUTPUTS
0       output/Softmax  1x13

I then run the inference using the following command to run inferences on the 700 files in the test dataset (same set used as the representative sample for the quantization.

ncc infer ..\tflite\model1.kmodel ..\tflite\kinfer --dataset path\to\test

Clearly I’d like to get meaningful inferences from the ncc infer command before trying to run it on the Maixduino.

I’m pretty new to all of this, so hopefully I’m just making a simple mistake. I’m attaching keras_model.zip the .dot files that are generated by the ncc conversion.

I do have a curiosity about why each of the KPUConv2D stages shows two outputs with one being ignored.

[johnseghers]

I converted all of my images to from RGB bmp files to single-channel grayscale PNG files. I am now able to get very similar results to the tflite results using a quantized kmodel (and float kmodel).

Thank you for your input–it put me on the track of thinking about what the inputs are and revisiting my assumption that ncc would do the appropriate image conversion.

[johnseghers]

Yes. My inputs are RGB images. These are converted to grayscale and rescaled by 1./255.

The input is of the net is one channel. Does nncase properly convert from RGB to grayscale when reading images?