[BUG] DepthAI does not work with OAK-D in LXD containers

Description of the bug

Running the DepthAI demo with python3 depthai_demo.py using a USB-C OAK-D in an LXD container fails to find the device after booting. Instead it gives the following errors:

$ python3 depthai_demo.py
Using depthai module from:  /home/geoff/.local/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so
Depthai version installed:  2.8.0.0
Available devices:
[0] 14442C1061F95ED700 [X_LINK_UNBOOTED]
Traceback (most recent call last):
  File "depthai_demo.py", line 132, in <module>
    with dai.Device(pm.p.getOpenVINOVersion(), device_info, usb2Mode=conf.args.usb_speed == "usb2") as device:
RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND

This is the dmesg output from plugging in the USB cable until the demonstration application terminates with an error.

[2528003.223803] usb 3-2: new high-speed USB device number 38 using xhci_hcd
[2528003.434236] usb 3-2: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
[2528003.434239] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[2528003.434240] usb 3-2: Product: Movidius MyriadX
[2528003.434242] usb 3-2: Manufacturer: Movidius Ltd.
[2528003.434242] usb 3-2: SerialNumber: 03e72485
[2528007.911720] usb 3-2: USB disconnect, device number 38
[2528008.424018] usb 4-2: new SuperSpeed Gen 1 USB device number 19 using xhci_hcd
[2528008.448490] usb 4-2: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
[2528008.448494] usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[2528008.448496] usb 4-2: Product: Luxonis Device
[2528008.448497] usb 4-2: Manufacturer: Intel Corporation
[2528008.448498] usb 4-2: SerialNumber: 14442C1061F95ED700
[2528016.490494] usb 3-2: new high-speed USB device number 39 using xhci_hcd
[2528016.697634] usb 3-2: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
[2528016.697637] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[2528016.697638] usb 3-2: Product: Movidius MyriadX
[2528016.697639] usb 3-2: Manufacturer: Movidius Ltd.
[2528016.697640] usb 3-2: SerialNumber: 03e72485
[2528016.937216] usb 4-2: USB disconnect, device number 19

Running it again after this fails to even find the device in bootloader mode:

$ python3 depthai_demo.py
Using depthai module from:  /home/geoff/.local/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so
Depthai version installed:  2.8.0.0
Available devices:
[0] <error> [X_LINK_UNBOOTED]
Traceback (most recent call last):
  File "depthai_demo.py", line 132, in <module>
    with dai.Device(pm.p.getOpenVINOVersion(), device_info, usb2Mode=conf.args.usb_speed == "usb2") as device:
RuntimeError: Failed to find device (ma2480), error message: X_LINK_DEVICE_NOT_FOUND

There is no dmesg output while running the application this second time.

Reconnecting the USB cable and running the demonstration application again gives the first result above, where the device is at first found, then lost.

Launching the same application in the host OS works fine. The camera image and disparity map are displayed correctly until Ctrl-C is pressed.

$ python depthai_demo.py
Using depthai module from:  /home/geoff/.local/lib/python3.9/site-packages/depthai.cpython-39-x86_64-linux-gnu.so
Depthai version installed:  2.8.0.0
Available devices:
[0] 14442C1061F95ED700 [X_LINK_UNBOOTED]
^CTraceback (most recent call last):
  File "/home/geoff/src/depthai/depthai_demo.py", line 297, in <module>
    key = cv2.waitKey(1)
KeyboardInterrupt

This is the dmesg output for when the application runs correctly on the host OS, from connecting the USB cable through to pressing Ctrl-C:

[2527911.859755] usb 3-2: new high-speed USB device number 35 using xhci_hcd
[2527912.066882] usb 3-2: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
[2527912.066885] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[2527912.066887] usb 3-2: Product: Movidius MyriadX
[2527912.066888] usb 3-2: Manufacturer: Movidius Ltd.
[2527912.066888] usb 3-2: SerialNumber: 03e72485
[2527921.597178] usb 3-2: USB disconnect, device number 35
[2527922.106617] usb 4-2: new SuperSpeed Gen 1 USB device number 17 using xhci_hcd
[2527922.131113] usb 4-2: New USB device found, idVendor=03e7, idProduct=f63b, bcdDevice= 1.00
[2527922.131117] usb 4-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[2527922.131120] usb 4-2: Product: Luxonis Device
[2527922.131121] usb 4-2: Manufacturer: Intel Corporation
[2527922.131123] usb 4-2: SerialNumber: 14442C1061F95ED700
[2527926.566500] usb 3-2: new high-speed USB device number 36 using xhci_hcd
[2527926.773672] usb 3-2: New USB device found, idVendor=03e7, idProduct=2485, bcdDevice= 0.01
[2527926.773675] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[2527926.773677] usb 3-2: Product: Movidius MyriadX
[2527926.773678] usb 3-2: Manufacturer: Movidius Ltd.
[2527926.773679] usb 3-2: SerialNumber: 03e72485
[2527926.939837] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.940000] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.940157] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.940313] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.940469] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.940624] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.940780] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.940940] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.941101] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.941261] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.941418] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.941574] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.941731] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.941888] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.942043] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.942200] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.942357] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.942577] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.942738] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.942895] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.943051] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.943222] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.943380] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.943537] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.943696] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.943853] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.944011] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.944167] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.944324] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.944482] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.944638] usb 4-2: usbfs: usb_submit_urb returned -19
[2527926.944795] usb 4-2: usbfs: usb_submit_urb returned -19
[2527927.006460] usb 4-2: USB disconnect, device number 17

To Reproduce

Steps to reproduce the behavior:

1. Create an LXD container with Ubuntu 20.04

   lxc launch ubuntu:20.04 my_container
   

2. Enter the container

   lxc exec my_container --user 1000 --group 1000 -- bash --login
   

3. In the container, set up the udev rule for DepthAI

   echo 'SUBSYSTEM=="usb", ATTRS{idVendor}=="03e7", MODE="0666"' | sudo tee /etc/udev/rules.d/80-movidius.rules
   sudo udevadm control --reload-rules && sudo udevadm trigger
   

4. Plug in the OAK-D
5. On the host, add the USB device to the container:

   lxc config device add my_container oakd usb vendorid=03e7
   

6. In the container, execute the depthai_demo.py demonstration application
7. Observe that the application finds the OAK-D in its bootloader mode, but fails to find the device after it reboots.

Expected behavior

The demonstration application launches in an LXD container, opens the OAK-D device and displays the camera image and the disparity map.

Attach system log

Output of log_system_information.py:

{
    "architecture": "64bit ELF",
    "machine": "x86_64",
    "platform": "Linux-5.12.15-arch1-1-x86_64-with-glibc2.33",
    "processor": "",
    "python_build": "default Jun 30 2021 10:22:16",
    "python_compiler": "GCC 11.1.0",
    "python_implementation": "CPython",
    "python_version": "3.9.6",
    "release": "5.12.15-arch1-1",
    "system": "Linux",
    "version": "#1 SMP PREEMPT Wed, 07 Jul 2021 23:35:29 +0000",
    "win32_ver": "",
    "uname": "Linux atlas 5.12.15-arch1-1 #1 SMP PREEMPT Wed, 07 Jul 2021 23:35:29 +0000 x86_64",
    "packages": [
        "alabaster==0.7.12",
        "appdirs==1.4.4",
        "apsw==3.35.4.post1",
        "argcomplete==1.12.1",
        "asn1crypto==1.4.0",
        "astroid==2.4.2",
        "autopep8==1.5.5",
        "Babel==2.9.1",
        "backcall==0.2.0",
        "beautifulsoup4==4.9.3",
        "blobconverter==1.0.0",
        "boto3==1.18.18",
        "botocore==1.21.18",
        "breathe==4.30.0",
        "bs4==0.0.1",
        "btrfsutil==5.12.1",
        "CacheControl==0.12.6",
        "cchardet==2.1.7",
        "certifi==2020.12.5",
        "cffi==1.14.6",
        "chardet==3.0.4",
        "colorama==0.4.4",
        "contextlib2==0.6.0.post1",
        "cryptography==3.4.7",
        "css-parser==1.0.6",
        "cssselect==1.1.0",
        "decorator==5.0.9",
        "depthai==2.8.0.0",
        "distlib==0.3.2",
        "distro==1.5.0",
        "dnspython==1.16.0",
        "docopt==0.6.2",
        "docutils==0.17.1",
        "entrypoints==0.3",
        "exhale==0.2.3",
        "feedparser==5.2.1",
        "ffmpy3==0.2.4",
        "flake8==3.9.2",
        "greenlet==1.1.0",
        "html2text==2020.1.16",
        "html5-parser==0.4.9",
        "html5lib==1.1",
        "idna==2.10",
        "ifaddr==0.1.7",
        "imagesize==1.2.0",
        "importlib-metadata==4.6.0",
        "ipython==7.25.0",
        "ipython_genutils==0.2.0",
        "isc==2.0",
        "isort==5.7.0",
        "jedi==0.17.2",
        "jeepney==0.6.0",
        "Jinja2==3.0.1",
        "jmespath==0.10.0",
        "keyring==23.0.1",
        "keyutils==0.6",
        "lazy-object-proxy==1.4.3",
        "lensfun==0.3.95",
        "louis==3.18.0",
        "lxml==4.6.3",
        "Markdown==3.3.4",
        "MarkupSafe==2.0.1",
        "matplotlib-inline==0.1.2",
        "mccabe==0.6.1",
        "mechanize==0.4.5",
        "more-itertools==8.7.0",
        "msgpack==1.0.2",
        "mutagen==1.45.1",
        "mypy==0.800",
        "mypy-extensions==0.4.3",
        "netifaces==0.11.0",
        "netsnmp-python==1.0a1",
        "networkx==2.5.1",
        "nftables==0.1",
        "npyscreen==4.10.5",
        "numpy==1.20.3",
        "opencv-contrib-python==4.5.1.48",
        "opencv-python==4.5.1.48",
        "ordered-set==4.0.2",
        "packaging==20.9",
        "parso==0.7.1",
        "pcp==5.0",
        "pep517==0.10.0",
        "pexpect==4.8.0",
        "pickleshare==0.7.5",
        "Pillow==8.3.1",
        "pip==21.2.3",
        "pluggy==0.13.1",
        "ply==3.11",
        "progress==1.5",
        "prompt-toolkit==3.0.19",
        "psutil==5.8.0",
        "ptyprocess==0.7.0",
        "pwquality==1.4.4",
        "py7zr==0.11.3",
        "pychm==0.8.6",
        "pycodestyle==2.7.0",
        "pycparser==2.20",
        "pycryptodome==3.10.1",
        "pydocstyle==6.1.1",
        "pyflakes==2.3.1",
        "Pygments==2.9.0",
        "PyGObject==3.40.1",
        "pylama==7.7.1",
        "pylint==2.6.0",
        "pyls-isort==0.2.0",
        "pyls-mypy==0.1.8",
        "pynvim==0.4.3",
        "pyOpenSSL==20.0.1",
        "pyparsing==2.4.7",
        "PyQt5==5.15.4",
        "PyQt5-sip==12.9.0",
        "PyQtWebEngine==5.15.4",
        "pyserial==3.5",
        "python-dateutil==2.8.1",
        "python-jsonrpc-server==0.4.0",
        "python-language-server==0.36.2",
        "pytube==10.8.5",
        "pytz==2021.1",
        "pyusb==1.2.1",
        "PyYAML==5.4.1",
        "regex==2021.7.6",
        "requests==2.24.0",
        "requests-cache==0.5.2",
        "resolvelib==0.5.5",
        "retrying==1.3.3",
        "rope==0.18.0",
        "s3transfer==0.5.0",
        "scipy==1.7.0",
        "SecretStorage==3.3.1",
        "setuptools==57.1.0",
        "Shapely==1.7.1",
        "sip==4.19.25",
        "six==1.16.0",
        "slip==0.6.5",
        "slip.dbus==0.6.5",
        "snowballstemmer==2.1.0",
        "soupsieve==2.2.1",
        "Sphinx==4.1.0",
        "sphinx-multiversion==0.2.4",
        "sphinx-rtd-theme==0.5.2",
        "sphinx-tabs==3.0.0",
        "sphinxcontrib-applehelp==1.0.2",
        "sphinxcontrib-devhelp==1.0.2",
        "sphinxcontrib-htmlhelp==2.0.0",
        "sphinxcontrib-jsmath==1.0.1",
        "sphinxcontrib-qthelp==1.0.3",
        "sphinxcontrib-serializinghtml==1.1.5",
        "team==1.0",
        "texttable==1.6.3",
        "toml==0.10.2",
        "traitlets==5.0.5",
        "trimesh==3.9.20",
        "tvdb-api==3.1.0",
        "typed-ast==1.4.2",
        "typing-extensions==3.7.4.3",
        "udiskie==2.3.3",
        "ujson==4.0.2",
        "unrardll==0.1.4",
        "urllib3==1.25.11",
        "urwid==2.1.2",
        "vcstool==0.2.15",
        "wcwidth==0.2.5",
        "webencodings==0.5.1",
        "wrapt==1.12.1",
        "yapf==0.30.0",
        "youtube-dl==2021.6.6",
        "zeroconf==0.29.0",
        "zipp==3.5.0"
    ],
    "usb": [
        {
            "port": 0,
            "vendor_id": "0x1d6b",
            "product_id": "0x0003",
            "speed": "SuperPlus"
        },
        {
            "port": 2,
            "vendor_id": "0x03e7",
            "product_id": "0x2485",
            "speed": "High"
        },
        {
            "port": 0,
            "vendor_id": "0x1d6b",
            "product_id": "0x0002",
            "speed": "High"
        },
        {
            "port": 0,
            "vendor_id": "0x1d6b",
            "product_id": "0x0003",
            "speed": "Super"
        },
        {
            "port": 9,
            "vendor_id": "0x3297",
            "product_id": "0x1969",
            "speed": "Full"
        },
        {
            "port": 8,
            "vendor_id": "0x047d",
            "product_id": "0x2041",
            "speed": "Low"
        },
        {
            "port": 7,
            "vendor_id": "0x0764",
            "product_id": "0x0501",
            "speed": "Full"
        },
        {
            "port": 4,
            "vendor_id": "0x05e3",
            "product_id": "0x0715",
            "speed": "High"
        },
        {
            "port": 3,
            "vendor_id": "0x14cd",
            "product_id": "0x1212",
            "speed": "High"
        },
        {
            "port": 4,
            "vendor_id": "0x0bda",
            "product_id": "0x5411",
            "speed": "High"
        },
        {
            "port": 10,
            "vendor_id": "0x0451",
            "product_id": "0x8142",
            "speed": "High"
        },
        {
            "port": 0,
            "vendor_id": "0x1d6b",
            "product_id": "0x0002",
            "speed": "High"
        }
    ]
}

[gbiggs]

can you try forcing USB2 mode in the script you are using?

I edited rgb_video.cpp to force USB2 mode and ran it, but it gives the same behaviour, except that the running-mode device shows up on the USB2 bus now instead of the USB3 bus (confirmed in dmesg).

Its odd that only that message is missing. Maybe LXD has any options for that case?

I agree, it’s very odd that it consistently misses just that message. I haven’t found any further information for LXD hotplugging other than “it just works”; which it does seem to mostly. It’s just that one event…

Also I saw that you have options for passing USB devices with vendor ID set to 0x03e7. I presume this goes for all devices, but note that the ROM Bootloader has product ID 0x2485 and running FW 0xf63b, just in case this comes up somewhere

Adding a device with a vendor ID only makes the product ID a wildcard match, apparently. Just to be sure, though, I added devices for both product IDs specifically. It made no difference.

[themarpe]

@gbiggs One extra thing worth checking out: Use environment variable LIBUSB_DEBUG=4 inside the container when running a depthai application to get more debug information of libusb. Check if the following message is printed out:

libusb: debug [linux_netlink_read_message] ignoring netlink message with non-zero sender UID 65534

If so, there is a file libusb/os/linux_netlink.c with the following contents:

cred = (struct ucred *)CMSG_DATA(cmsg);
if (cred->uid != 0) 
{
		usbi_dbg("ignoring netlink message with non-zero sender UID %u", (unsigned int)cred->uid);
		return -1;
}

Remove the if statement check, recompile the libusb and depthai and retest.

If the above is not apparent, feel free to skim through libusb debug logs and try spotting anything unordinary. You may also diff compare it with a disabled udev libusb on host (otherwise messages won’t be the same most likely)