[BUG] Multiple OAK-D devices not closing properly

Describe the bug When using multiple OAK-D devices when closing the devices with USB3 the final device will not close properly (meaning you can’t create a new Device from it). This is resolved by powercycling.

To Reproduce Steps to reproduce the behavior:

1. Run script: https://github.com/luxonis/depthai-experiments/tree/master/gen2-multiple-devices
2. Observe “Connected to XXXXXX” in the console twice
3. Quit the script (press q, or CTRL+C, either results in the same issue)
4. Re-run the script
5. Observe “Connected to XXXXXX” in the console once - potentially followed by an error in the format: RuntimeError: Failed to find device (14442C1041A809D100-ma2480), error message: X_LINK_DEVICE_NOT_FOUND

Expected behavior Running the script multiple times works as the same every time without a need to power cycle.

Screenshots

Attach system log

{
    "architecture": "64bit WindowsPE",
    "machine": "AMD64",
    "platform": "Windows-10-10.0.19041-SP0",
    "processor": "AMD64 Family 25 Model 33 Stepping 0, AuthenticAMD",
    "python_build": "tags/v3.9.4:1f2e308 Apr  6 2021 13:40:21",
    "python_compiler": "MSC v.1928 64 bit (AMD64)",
    "python_implementation": "CPython",
    "python_version": "3.9.4",
    "release": "10",
    "system": "Windows",
    "version": "10.0.19041",
    "win32_ver": "10 10.0.19041 SP0 Multiprocessor Free",
    "uname": "Windows Johns-New-PC 10 10.0.19041 AMD64 AMD64 Family 25 Model 33 Stepping 0, AuthenticAMD",
    "packages": [
        "depthai==2.2.1.0",
        "numpy==1.20.2",
        "opencv-contrib-python==4.5.1.48",
        "pip==21.0.1",
        "setuptools==49.2.1"
    ],
    "usb": [
        "No USB backend found"
    ]
}

Additional context The problem does not occur when usb2Mode=True

[John-Dean]

From a mixture of AMD updates and latest DepthAI updates I can’t replicate this anymore - so for now I think this is safe to say is fixed, thanks for looking into this.

[John-Dean]

So after some cleanup of my code I have the following three python functions (it’s very similar to the https://github.com/luxonis/depthai-experiments/blob/master/gen2-multiple-devices/main.py code)

(Using cv2, depthai and contextlib as imports)

The create_pipeline function creates and returns a depthai pipeline that has colour and depth outputs called rgb and depth

def create_pipeline():
	# Start defining a pipeline
	pipeline = depthai.Pipeline()

	# Define a source - color camera
	cam_rgb = pipeline.createColorCamera()
	cam_rgb.setPreviewSize(600, 600)
	cam_rgb.setBoardSocket(depthai.CameraBoardSocket.RGB)
	cam_rgb.setResolution(depthai.ColorCameraProperties.SensorResolution.THE_1080_P)
	cam_rgb.setInterleaved(False)


	# Define a source - two mono (grayscale) cameras
	left = pipeline.createMonoCamera()
	left.setResolution(depthai.MonoCameraProperties.SensorResolution.THE_800_P)
	left.setBoardSocket(depthai.CameraBoardSocket.LEFT)

	right = pipeline.createMonoCamera()
	right.setResolution(depthai.MonoCameraProperties.SensorResolution.THE_800_P)
	right.setBoardSocket(depthai.CameraBoardSocket.RIGHT)

	left.setFps(60.0)
	right.setFps(60.0)


	# Define a source - depth camera
	depth = pipeline.createStereoDepth()
	depth.setConfidenceThreshold(200)
	left.out.link(depth.left)
	right.out.link(depth.right)
	depth.setOutputDepth(True)


	# Create output
	xout_rgb = pipeline.createXLinkOut()
	xout_rgb.setStreamName("rgb")
	cam_rgb.preview.link(xout_rgb.input)


	xout_depth = pipeline.createXLinkOut()
	xout_depth.setStreamName("depth")
	depth.depth.link(xout_depth.input)
	
	return pipeline

The get_all_cameras function takes a pipeline and returns a list of devices, a list of rgb source queues and a list of depth source queues.

def get_all_cameras(pipeline):
	devices = []
	
	rgb_sources = []
	depth_sources = []
	
	for device_info in depthai.Device.getAllAvailableDevices():
		device = depthai.Device(pipeline=pipeline, deviceDesc=device_info, usb2Mode=False)
		
		devices.append(device)
		
		print("Conected to " + device_info.getMxId())
		device.startPipeline()
		queue_rgb = device.getOutputQueue(name="rgb", maxSize=1, blocking=False)
		rgb_sources.append(queue_rgb)	
		queue_depth = device.getOutputQueue(name="depth", maxSize=1, blocking=False)
		depth_sources.append(queue_depth)
	
	return devices, rgb_sources, depth_sources

And finally the close_cameras function takes a list of devices and closes them (I was originally using the contextlib.ExitStack() approach described in the https://github.com/luxonis/depthai-experiments/blob/master/gen2-multiple-devices/main.py file, but I found this harder to troubleshoot which is why I swapped to this approach).

def close_cameras(devices):
	for device in devices:
		device.close()

Finally the wrapper code to run it all:

pipeline = create_pipeline()

devices, rgbs, depths = get_all_cameras(pipeline)

running = True
while running:	
	for i, q_rgb in enumerate(rgbs):
		in_rgb = q_rgb.tryGet()
		if in_rgb is not None:
			cv2.imshow("rgb-" + str(i + 1), in_rgb.getCvFrame())

	if cv2.waitKey(1) == ord('q'):
		running = False

close_cameras(devices)

In total the file looks like this:

import cv2
import depthai
import contextlib



def create_pipeline():
	# Start defining a pipeline
	pipeline = depthai.Pipeline()

	# Define a source - color camera
	cam_rgb = pipeline.createColorCamera()
	cam_rgb.setPreviewSize(600, 600)
	cam_rgb.setBoardSocket(depthai.CameraBoardSocket.RGB)
	cam_rgb.setResolution(depthai.ColorCameraProperties.SensorResolution.THE_1080_P)
	cam_rgb.setInterleaved(False)


	# Define a source - two mono (grayscale) cameras
	left = pipeline.createMonoCamera()
	left.setResolution(depthai.MonoCameraProperties.SensorResolution.THE_800_P)
	left.setBoardSocket(depthai.CameraBoardSocket.LEFT)

	right = pipeline.createMonoCamera()
	right.setResolution(depthai.MonoCameraProperties.SensorResolution.THE_800_P)
	right.setBoardSocket(depthai.CameraBoardSocket.RIGHT)

	left.setFps(60.0)
	right.setFps(60.0)


	# Define a source - depth camera
	depth = pipeline.createStereoDepth()
	depth.setConfidenceThreshold(200)
	left.out.link(depth.left)
	right.out.link(depth.right)
	depth.setOutputDepth(True)


	# Create output
	xout_rgb = pipeline.createXLinkOut()
	xout_rgb.setStreamName("rgb")
	cam_rgb.preview.link(xout_rgb.input)


	xout_depth = pipeline.createXLinkOut()
	xout_depth.setStreamName("depth")
	depth.depth.link(xout_depth.input)
	
	return pipeline


def get_all_cameras(pipeline):
	devices = []
	
	rgb_sources = []
	depth_sources = []
	
	for device_info in depthai.Device.getAllAvailableDevices():
		device = depthai.Device(pipeline=pipeline, deviceDesc=device_info, usb2Mode=True)
		
		devices.append(device)
		
		print("Conected to " + device_info.getMxId())
		device.startPipeline()
		# Output queue will be used to get the rgb frames from the output defined above
		queue_rgb = device.getOutputQueue(name="rgb", maxSize=1, blocking=False)
		rgb_sources.append(queue_rgb)	
		queue_depth = device.getOutputQueue(name="depth", maxSize=1, blocking=False)
		depth_sources.append(queue_depth)
	
	return devices, rgb_sources, depth_sources

def close_cameras(devices):
	for device in devices:
		device.close()
		

pipeline = create_pipeline()

devices, rgbs, depths = get_all_cameras(pipeline)

running = True
while running:	
	for i, q_rgb in enumerate(rgbs):
		in_rgb = q_rgb.tryGet()
		if in_rgb is not None:
			cv2.imshow("rgb-" + str(i + 1), in_rgb.getCvFrame())

	if cv2.waitKey(1) == ord('q'):
		running = False

close_cameras(devices)

I can confirm that it is primarily occuring when using devices on different USB controllers.

In this screenshot the first 3 executions were taken on my front connector USB ports (ones connected via a USB 3 header). At the red line I swap one of the cameras to use a USB directly attached to the motherboard and I have issues (the 2nd camera isn’t connecting when it’s run again). If I move them both to the motherboard USB then there is no issue.

Ideally they would be connected via different streams to not run into bandwidth limitations on a USB controller (this is a common issue with SteamVR basestations and tracker dongles overwhelming USB controllers), but I can accept running them this way.