Define 8 classes but only 3 are assigned (independently n. iter)

Greetings! I’ve been guided mainly by @shivam-kotwalia 's (https://github.com/shivam-kotwalia/KittiSeg/) fork for multi-class segmentation, which has been reported as a sucessful implementation for multi-class seg. Using my datasets with 8 classes, I still stuck in this problem.

Part of my hype is defined as such:

"model": {
    "input_file": "PATH/inputs/kitti_seg_input.py",
    "architecture_file" : "../encoder/fcn8_vgg.py",
    "objective_file" : "../decoder/fcn.py",
    "optimizer_file" : "../optimizer/generic_optimizer.py",
    "evaluator_file" : "../evals/kitti_eval.py"
  },
  "colors": {
    "background_color" : [0, 0, 0],
    "sky_color" : [0, 255, 255],    
    "roof_color" : [0, 0, 255],
    "wall_color" : [255, 255, 0],
    "window_color" : [255, 0, 0],
    "door_color" : [255, 128, 0],
    "shop_color" : [0, 255, 0],
    "balcony_color" : [255, 0, 255]    
  },
 "arch": {
    "fcn_in": "pool5",
    "num_classes" : 8,
    "image_size" : 50,    
    "weight": [1, 2, 2, 2, 2, 2, 2, 2],
    "num_channels" : 3,
    "whitening": false
  },

Any considerable modification were done in other files. After to get a good segmentation, I intend to improve the weights training with another datasets (which is in different image dimensions). I was wondering which kind of disturbance it could led.

So, please, if anyone have a hint I would appreciate!

Cheers, Rodolfo.

[rodolfolotte]

Hi @ywangeq! Thank you for replying!

I will try to describe all my changes. First, I simply delete every file that were not useful to the multi-class purpose, rearranged the logging messages, delete unnecessary comments, so on. Still, my work-space is not done for a commit.

I also follow what @shivam-kotwalia has done for multi-class segmentation. In his version, the evaluate method in eval/kitti_eval.py was commented. Plus, the eval_image method was still to binary segmentation. Then, I introduced the @bendidi version for this evaluation. Keeping the image saving commented:

# if phase == 'val':
  # Saving RB Plot
  #     ov_image = seg.make_overlay(image, output_im)
  #     name = os.path.basename(image_file)
  #     image_list.append((name, ov_image))
                  
  #     name2 = name.split('.')[0] + '_green.png'
                    
  #     hard = output_im > 0.5
  #     green_image = utils.fast_overlay(image, hard)
  #     image_list.append((name2, green_image))

The hyper-parameters used to the respective user was simply adapted for the number of classes that I needed (8 instead of 21). Optimizer, architecture, and directories values were kept:

{
  "model": {
    "input_file": "PATH/inputs/kitti_seg_input.py",
    "architecture_file" : "../encoder/fcn8_vgg.py",
    "objective_file" : "../decoder/fcn.py",
    "optimizer_file" : "../optimizer/generic_optimizer.py",
    "evaluator_file" : "../evals/kitti_eval.py"
  },  
  "path": ["../submodules"],
  "dirs": {
    "cnn_dir": "PATH/cnn/",    
    "val_name": "eval",
    "test_name": "tests"
  },
  "data": {    
    "data_dir" : "PATH/images",
    "train_file" : "inputs/train.txt",
    "val_file" : "inputs/val.txt",
    "tests_file" : "inputs/tests.txt"    
  },
  "colors": {
    "background" : [0, 0, 0],
    "sky" : [0, 255, 255],    
    "roof" : [0, 0, 255],
    "wall" : [255, 255, 0],
    "window" : [255, 0, 0],
    "door" : [255, 128, 0],
    "shop" : [0, 255, 0],
    "balcony" : [255, 0, 255]    
  },
 "arch": {
    "fcn_in": "pool5",    
    "num_classes" : 8,
    "image_size" : 50,    
    "weight": [1, 2, 2, 2, 2, 2, 2, 2],
    "num_channels" : 3,
    "whitening": false
  },
"jitter": {
    "random_resize": false,
    "lower_size": 0.4,
    "upper_size": 1.7,
    "sig": 0.15,
    "res_chance": 0.4,
    "random_crop": true,
    "max_crop": 32,
    "crop_patch": false,
    "patch_height": 256,
    "patch_width": 256,
    "crop_chance": 0.8,
    "fix_shape": false,
    "reseize_image": false,
    "image_height" : 1067,
    "image_width" : 800,
    "augment_level": 0
  },
  "logging": {
    "display_iter": 10,
    "eval_iter": 200,
    "write_iter": 200,
    "save_iter": 200,
    "image_iter": 200
  },
  "solver": {
      "opt": "Adam",
      "batch_size": 1,
      "epsilon": 0.000000001,
      "adam_eps": 0.00001,
      "threads": 4,
      "learning_rate": 1e-5,
      "learning_rate_step": null,
      "max_steps": 1000
  },
  "use_fc_wd": true,
  "loss": "xentropy",
  "clip_norm" : 1.0,
  "wd": 5e-4
}

Another change, was in kitt_seg_input.py, in inputs/. Basically, only the _make_data_gen method:

def _make_data_gen(hypes, phase, data_dir):
    """Return a data generator that outputs image samples.

    @ Returns
    image: integer array of shape [height, width, 3].
    Representing RGB value of each pixel.
    gt_image: boolean array of shape [height, width, num_classes].
    Set `gt_image[i,j,k] == 1` if and only if pixel i,j
    is assigned class k. `gt_image[i,j,k] == 0` otherwise.

    [Alternativly make gt_image[i,j,*] a valid propability
    distribution.]
    """
    if phase == 'train':
        data_file = hypes['data']["train_file"]
    elif phase == 'val':
        data_file = hypes['data']["val_file"]
    else:
        assert False, "Unknown Phase %s" % phase

    data_file = os.path.join(data_dir, data_file)

    classes = hypes['colors']
    num_classes = len(classes)
    assert num_classes > 1, "Min amount of segmentation classes is 2 but only %d class(es) is defined" % num_classes

    data = _load_gt_file(hypes, data_file)

    for image, gt_image in data:
        gt_classes = []
        for color in classes.values():
            gt_classes.append(np.all(gt_image == color, axis=2))
        assert(gt_classes[0].shape == gt_classes[-1].shape)

        gt_reshaped = []
        shape = gt_classes[0].shape
        for gt_class in gt_classes:
            gt_reshaped.append(gt_class.reshape(shape[0], shape[1], 1))

        gt_image = np.concatenate(gt_reshaped, axis=2)

        if phase == 'val':
            yield image, gt_image
        elif phase == 'train':
            yield jitter_input(hypes, image, gt_image)
            yield jitter_input(hypes, np.fliplr(image), np.fliplr(gt_image))

Finally, the demo.py became segment.py. An adaptation to segment a bigger number of images:

import json
import logging
import os
import sys

import collections

# configure logging
logging.basicConfig(format='%(asctime)s %(levelname)s %(message)s',
                    level=logging.INFO,
                    stream=sys.stdout)

# https://github.com/tensorflow/tensorflow/issues/2034#issuecomment-220820070
import numpy as np
import scipy as scp
import scipy.misc
import tensorflow as tf

flags = tf.app.flags
FLAGS = flags.FLAGS

sys.path.insert(1, 'submodules')

from evaluation import seg_utils as seg

try:
    # Check whether setup was done correctly
    import tensorvision.utils as tv_utils
    import tensorvision.core as core
except ImportError:
    # You forgot to initialize submodules
    logging.error("Could not import the submodules.")
    logging.error("Please execute:"
                  "'git submodule update --init --recursive'")
    exit(1)

flags.DEFINE_string('logdir', None,
                    'Path to logdir.')
flags.DEFINE_string('input_folder', None,
                    'Folder to apply FacadeSeg.')
flags.DEFINE_string('output_folder', None,
                    'Folder to save FacadeSeg results.')

def resize_label_image(image, gt_image, image_height, image_width):
    image = scp.misc.imresize(image, size=(image_height, image_width),
                              interp='cubic')
    shape = gt_image.shape
    gt_image = scp.misc.imresize(gt_image, size=(image_height, image_width),
                                 interp='nearest')

    return image, gt_image


def main(_):
    tv_utils.set_gpus_to_use()

    if FLAGS.input_folder is None:
        logging.error("No input_image was given.")
        logging.info(
            "Usage: python segment.py [--input_folder /path/to/data/] "
            "[--output_folder /path/to/result/] [--logdir /path/to/weights] "
            "[--gpus GPUs_to_use] ")

        exit(1)

    if FLAGS.logdir is None:
        # Download and use weights from FacadeSeg
        if 'TV_DIR_RUNS' in os.environ:
            runs_dir = os.path.join(os.environ['TV_DIR_RUNS'], 'FacadeSeg')
        else:
            runs_dir = 'RUNS'
        
        logdir = os.path.join(runs_dir, "FacadeSegPreTrained")
    else:
        logging.info("Using weights found in {}".format(FLAGS.logdir))
        logdir = FLAGS.logdir

    # Loading hyperparameters from logdir
    hypes = tv_utils.load_hypes_from_logdir(logdir, base_path='hypes')
    logging.info("Hypes loaded successfully.")

    # Loading tv modules (encoder.py, decoder.py, eval.py) from logdir
    modules = tv_utils.load_modules_from_logdir(logdir)
    logging.info("Modules loaded successfully. Starting to build tf graph.")

    # Create tf graph and build module.
    with tf.Graph().as_default():
        # Create placeholder for input
        image_pl = tf.placeholder(tf.float32)
        image = tf.expand_dims(image_pl, 0)

        # build Tensorflow graph using the model from logdir
        prediction = core.build_inference_graph(hypes, modules, image=image)

        logging.info("Graph build successfully.")

        # Create a session for running Ops on the Graph.
        sess = tf.Session()
        saver = tf.train.Saver()

        # Load weights from logdir
        core.load_weights(logdir, sess, saver)

        logging.info("Weights loaded successfully.")

    # classes
    classes_colors =  [[0, 0, 0],
                       [0, 255, 255], 
                       [0, 0, 255], 
                       [255, 255, 0], 
                       [255, 0, 0], 
                       [255, 128, 0], 
                       [0, 255, 0], 
                       [255, 0, 255]]

    input_folder = FLAGS.input_folder
    output_folder = FLAGS.output_folder

    # If the folder is not empty
    if(not os.listdir(input_folder)==""):
        for root, dirs, files in os.walk(input_folder):
            for file in files:
                input_image = os.path.join(input_folder, file)

                logging.info("Starting inference using {} as input".format(input_image))

                # Load and resize input image
                image = scp.misc.imread(input_image)

                if hypes['jitter']['reseize_image']:
                    # Resize input only, if specified in hypes
                    image_height = hypes['jitter']['image_height']
                    image_width = hypes['jitter']['image_width']
                    image = scp.misc.imresize(image, size=(image_height, image_width), interp='cubic')
            

                # Run FacadeSeg model on image
                feed = {image_pl: image}
                softmax = prediction['softmax']
                logits = prediction['logits']
                output, lll = sess.run([softmax, logits], feed_dict=feed)

                # Reshape output from flat vector to 2D Image
                shape = image.shape
                output_image = output.reshape(shape[0], shape[1], -1)

                x = np.argmax(output_image, axis=2)
                im = np.zeros((shape[0], shape[1], 3), dtype=np.uint8)
                for i,_ in enumerate(x):
                    for j,_ in enumerate(x[i]):
                        value = x[i][j]
                        color_code = classes_colors[value]
                        im[i][j] = color_code

                # Save output images to disk.
                if(not os.path.isdir(output_folder)):
                    logging.info("Output directory does not exist. Creating in: " + output_folder)
                    os.makedirs(output_folder)
                
                raw_image_name = file.split('.')[0] + '.png'
                full_raw_image_name = os.path.join(output_folder, raw_image_name)

                scp.misc.imsave(full_raw_image_name, im)
                logging.info("Labelled image saved in " + full_raw_image_name + "\n")   
    else:
        logging.info("Input folder is empty. Check it again")

if __name__ == '__main__':
    tf.app.run()

However, after to test different number of iterations, I’ve notice that the network segment only 5 classes (with background), instead of the 8 needed. First guess, was that the number of iterations was not enough. Then, I settled 100000 iterations. Again, only 5 classes. As you can see, 4 classes are perfectly assigned, but the other still missing:

Then, I realize that (by mistake) I could be deleted something that I could not. I reverted everything by cloning @shivam-kotwalia fork again. I simply got the same segmentation.

[rodolfolotte]

The dataset I’m using is not completely open. In order to have it, you will need to fill out a form: https://varcity.ethz.ch/3dchallenge/ . Thank you!