Using additional channels for object detection

🚀 Feature

Is your feature request related to a problem? Please describe. Is is possible to use images with more then 3 RGB channels for training? I mean e.g. using the infrared light as an additional channel?

My first guess is not, since in Parser we return file_path not the image?, but maybe I’m wrong?

def filepath(self, o) -> Union[str, Path]:
        return self.source / f"{o.image_id}.jpg"

Describe the solution you’d like

Maybe instead of returning the image path, would it be possible to alternatively return an image itself (since sometimes infrared images are in seprate files so some form of manual merging of the images should be conducted)?

Additional context This would enable using the ice vision to the wider variety of tasks, especially the real-world, industrial applications

[rbavery]

And even beyond earth science, there’s plenty of other use cases in medical imaging and geology where more than 3 bands are useful. Geology in particular often makes use of 100s of bands (hyperspectral data) https://www.usgs.gov/centers/geology%2C-geophysics%2C-and-geochemistry-science-center/science/science-topics/hyperspectral

[tugot17]

I guess that it is not much different then training on RGB images.

One solution would be to add additional channel in the input layer, similarly to as it is done in this efficientnet implementation, so to replace the first layer with Conv of different size, and treat infrared as just an additional channel

if  in_channels != 3:
      Conv2d = get_same_padding_conv2d(image_size=self._global_params.image_size)
      out_channels = round_filters(32, self._global_params)
      self._conv_stem = Conv2d(in_channels, out_channels, kernel_size=3, stride=2, bias=False)