What is the best way to integrate Custom Detection + Tracking + Depth/Spatial Location Calculation

Hello I am currently trying to use YOLOv5 with multiple object tracking (specifically SORT), retrieve the spatial coordinates and then save the associated depth frame.

I have used some of the code from https://github.com/luxonis/depthai-experiments/tree/master/gen2-yolov5 for decoding the detections from YOLOv5 and then passed that to my MOT to get bounding boxes associated with the tracked objects. All of this so far has worked well. I have used the depthai_demo.py code and inserted my own snippets to make it work, and I can verify it via the previews.

Then, I’ve tried using SpatialLocationCalculator, similar to how it was done here https://docs.luxonis.com/projects/api/en/latest/samples/SpatialDetection/spatial_location_calculator/#spatial-location-calculator . However it seems like there is a frame or two of lag between enqueue’ing the ROIs and when I can actually read those same ROIs from the “spatialData” queue.

I have tried setting the SpatialLocationCalculator

slc.setWaitForConfigInput(True) and using

spatialDataQueue.get()

And that fixes the ordering of messages received but seems to cause issues with reading new depth frames, which I can see in the depth preview window and also in the end print statement, where TOTAL FPS will usually show color FPS much higher than depth or depthRaw. The color stream looks to be unchanged, though. It seems to me that the setWaitForConfigInput(True) also pauses the depth stream while it waits for a config input… is this expected behavior?

Also is there a better way for me to do this?

[szabi-luxonis]

@AlexanderJGomez

does this mean that for that example the bounding boxes we see are not actually properly aligned?

Yes.

The bounding boxes which are obtained from preview output of RGB camera are not aligned properly with depth frame because depth is aligned to rectified right perspective by default (right stereo camera). The only alignment that is done is scaling the bounding box, taking into consideration FOV difference between RGB and right stereo camera. To obtain proper alignment you need to use setDepthAlign and setIspScale option as mentioned before, which will align the depth frame to RGB camera perspective, so bounding boxes will have proper aligment.

[szabi-luxonis]

@AlexanderJGomez calc_spatials uses the mean value in ROI while on-device SLC uses the average. Here is the equivalent of SLC algorithm on host: https://github.com/luxonis/depthai-experiments/blob/master/gen2-spi/spatial-location-calculator/main.py#L150