Arbitrary crop/rescale/reformat and ROI return

Start with the why:

Often in CV/AI pipelining there is some ROI (region of interest), or series of ROIs, which should be used in the pipeline - whether set dynamically based on results in the pipeline - or set statically based on some a-priori or tacitly-understood application information.

So being able to specify via the DepthAI API (USB, SPI (https://github.com/luxonis/depthai/issues/140), or microPython (https://github.com/luxonis/depthai/issues/207)) for the results from some other node (e.g. the color camera output) to be cropped, rescaled and/or reformatted and for this ROI to be returned (internally, to a next stage, or to the host) is very valuable.

And example is ignoring the hood of a car when making a smart dash-cam (as YOLO will identify the hood as ‘car’, and well, that is already known).

Move to the how:

Implement support for at arbitrary crop/rescale/reformat Gen2 pipeline builder node (https://github.com/luxonis/depthai/issues/136) which can be controlled by DepthAI API via USB, SPI, or on-board microPython.

Move to the what:

Support arbitrary crop/rescale/reformat nodes as part of the Gen2 pipeline builder.

[Luxonis-Brandon]

Thanks @neixlo !

if I understand this feature right, it will enable the depthai device to crop a defined ROI and send this ROI with high resolution (uncompressed) to the host.

Yes, that’s exactly right and exactly what we had envisioned this for.

For example: I configure a pipeline with Gen2 pipeline builder to 1. detect a ROI of the color stream, 2. crop this ROI based on the detection, 3. send this ROI with full resolution to the host at higher frame rates then the full resolution of the color stream.

Yes, that’s exactly right.

Is this possible with Gen2 pipeline builder and the december release? If so, can you allready estimate a release date?

I think we just finished this actually. But I need to check about when it will be integrated and usable in the Gen2 branch (it might be there already). @themarpe can you comment? And CC: @VanDavv

Here’s the latest internally:

Currently it can do:

• crops
• resizing
• thumbnailing.

For future features, we want to add:

• color space conversion
• grayscale
• fp16 normalization
• flip

Im highly interessed in this feature and of cause your tremendous efforts in depthai hard- and software. Thanks for making this possible!

Thank you for the kind words! There have been products in my past where I think “Man, I sure am happy that someone made this.” This is exactly what we are trying to build here. So hearing this is great, and we hope to make it even more valuable over time.

Thanks again, Brandon

[neixlo]

Awesome! I tried, it worked. Good example to see how the host can change the crop while runtime.