Use Pybind11 for rclpy Python Bindings

Feature request

@azeey @iantheengineer @cottsay FYI

Feature description

This is a ticket to use Pybind11 for the CPython extension instead of manually writing code to interact with the CPython API.

Implementation considerations

Plan: https://docs.google.com/document/d/1BsTlxgzAkpX-5Q7FW6_Hyw1fszDEQef_OEvYmdnrBrE/edit?usp=sharing

• Replace custom CMake with Pybind11’s pybind11_add_module (#667)
  • Investigate CMake issues in Windows Debug https://github.com/ros2/rclpy/pull/667#issuecomment-778687182
  • Fix windows warning reported in #687, (#688)
  • Remove code unsetting Py_DEBUG definition once all modules use Pybind11 (https://github.com/ros2/rclpy/pull/681#issuecomment-784579212) #814
• Convert each module to use Pybind11 with the existing C to Python API
  • _rclpy (#675 ✔️, #691 ✔️, #692 ✔️, #693 ✔️, #694 ✔️, #695 ✔️, #696 ✔️, #699 ✔️, #701 ✔️, #703 ✔️, #706 ✔️ , #709 ✔️ , #712 ✔️ , #715 ✔️ , #717 ✔️ , #718 ✔️ , #719 ✔️, #721 ✔️ , #723 ✔️, #730 ✔️ , #732 ✔️ , #733 ✔️, #735 ✔️, #736 ✔️, #738 ✔️)
  • _rclpy_action (#678)
  • _rclpy_handle (#668)
  • _rclpy_logging (#659)
  • _rclpy_pycapsule (#652)
  • _rclpy_signal_handler (#728)
    • skipping - uses C atomic types that can’t be used with C++ due to p0943r1 #814
    • Actually, couldn’t skip because it uses rclpy_common, so rewrote in Python.
  • _rclpy_common (Common library, not a module, started in #675 ✔️, #727 ✔️, #743) #814
• Replace checking rcl_ret_t with void throw_if_not_ok(rcl_ret_t ret, const std::string & extra_error_msg);
  • Or use RCLError(), like #685 does for _rclpy_action
• Refactor rclpy objects to use C++ classes one object at a time, leaf objects first
  • rcl_time_point_t -> bind struct #737
  • rmw_service_info_t -> bind struct (#748)
  • rcl_duration_t -> bind struct (claimed) https://github.com/ros2/rclpy/pull/744
  • rmw_qos_profile_t -> bind struct (#741)
  • rmw_request_id_t -> bind struct (#748)
  • rcl_wait_set_t -> not sure if rclpy::WaitSet or bind struct (maybe tricky) (#769)
  • rclpy_client_t -> rclpy::Client (#739)
  • rclpy_service_t -> rclpy::Service (#747)
  • rcl_guard_condition_t -> rclpy::GuardCondition #772 (for H-turtle branch)
  • rcl_event_t -> rclpy::Event (#750)
  • rclpy_publisher_t -> rclpy::Publisher (#756)
  • rclpy_subscription_t -> rclpy::Subscription (#756)
  • rcl_timer_t -> rclpy::Timer (#745)
  • rcl_clock_t -> rclpy::Clock (#749)
  • rcl_action_client_t -> rclpy::ActionClient (#759)
  • rcl_action_goal_handle_t -> rclpy::ActionGoalHandle (#751, #767)
  • rcl_action_server_t -> rclpy::ActionServer (#766)
  • rcl_node_t -> rclpy::Node (#771)
  • rcl_context_t -> rclpy::Context (#771)

Refactor order: Arrows point from a type that needs to be refactored before the type it points to. Types with no arrows pointing them can be done as soon as all methods using them are converted to pybind11.

dot graph of order

digraph order_of_conversions {
rcl_time_point_t
rcl_wait_set_t
rmw_qos_profile_t
rmw_request_id_t
rmw_service_info_t
rcl_duration_t
action_client -> rcl_node_t
action_server -> rcl_node_t
rcl_action_goal_handle_t -> action_server
rcl_node_t -> context
subscription -> rcl_node_t
publisher -> rcl_node_t
service -> rcl_node_t
client -> rcl_node_t
timer -> context
timer -> clock
clock
guard_condition -> context
rcl_event_t -> publisher
rcl_event_t -> subscription
}

[sloretz]

@sloretz Given that we’ve gone as far as we are going to go with this in Galactic, should we remove this ticket from the Galactic Project board and move it to the H-Turtle one?

I would keep it in Galactic. Our current plan is to backport the remainder of the work to Galactic for the first patch release.

[ahcorde]

Related work

• Remove rclpy_common by eliminating any use of the C api’s in rclpy_common for the C++ ones that use pybind11:
  • Once this PR is merge we should remove handle.c
  • Moved function from common.c to utils.cpp https://github.com/ros2/rclpy/pull/787
  • Removed common.c https://github.com/ros2/rclpy/pull/789
• Move exceptions.hpp/exceptions.cpp to the _rclpy_pybind11 module (https://github.com/ros2/rclpy/pull/788)
• Delele handle.hpp/handle.cpp as they should no longer be used (https://github.com/ros2/rclpy/pull/786)