crazyflie is unstable and often flips after taking off

I have encountered a serious problem, but I still don’t know the problem: Recently, I cloned the new master-dfd69b4. When I ran python niceHover.py, I found out: When flying two cfs, one cf will always turn over; at the same time, when I test a single cf, it will also Discovered the phenomenon of flight instability and overturning. I suspect it is the latest master-dfd69b4 problem. I cloned an older branch ros_time_abstraction today. This problem also exists. When I tested the single cf, I also found that the flight was unstable and flipped.

What I need to add is: when the cf flipped and fell, I heard the sound of cf restarting (not every time there is a restarting sound), but I can’t tell whether it was restarting during flight that caused the cf to fall or cf to fall. After falling behind, it restarts.

1. There does not seem to be a problem with the data and positioning estimates of the motion capture system. I followed @whoenig 's tips and used NatNetSDKCrossplatform to get the data of the motion capture system. The data seemed to have no obvious problems. At the same time, I turned on logging and recorded the estimated value of xyz coordinates, which seems to be correct.
2. I suspect it is the problem of cf firmware, but I don’t know why the same problem exists when I use different branches of crazyswarm, maybe this may also be a communication problem. I upload a log of my running nicehover.py, can anyone help me analyze it. Cf flips after hovering: Here is my flight log： https://drive.google.com/drive/folders/14TvsCL2WbxcivZRMQHemH5bl7j9a83_r?usp=sharing

roslaunch crazyswarm hover_swarm.launch
... logging to /home/ss/.ros/log/7d2ef0e2-ea0e-11ea-a663-200db01c05e8/roslaunch-407-19165.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://localhost:35511/

SUMMARY
========

PARAMETERS
 * /crazyflieTypes/default/batteryVoltageWarning: 3.8
 * /crazyflieTypes/default/batteryVoltateCritical: 3.7
 * /crazyflieTypes/default/bigQuad: False
 * /crazyflieTypes/default/dynamicsConfiguration: 0
 * /crazyflieTypes/default/firmwareParams/ctrlMel/i_range_m_z: 1500
 * /crazyflieTypes/default/firmwareParams/ctrlMel/i_range_xy: 2.0
 * /crazyflieTypes/default/firmwareParams/ctrlMel/i_range_z: 0.4
 * /crazyflieTypes/default/firmwareParams/ctrlMel/kR_xy: 70000
 * /crazyflieTypes/default/firmwareParams/ctrlMel/kR_z: 60000
 * /crazyflieTypes/default/firmwareParams/ctrlMel/kd_omega_rp: 200
 * /crazyflieTypes/default/firmwareParams/ctrlMel/kd_xy: 0.2
 * /crazyflieTypes/default/firmwareParams/ctrlMel/kd_z: 0.4
 * /crazyflieTypes/default/firmwareParams/ctrlMel/ki_m_z: 500
 * /crazyflieTypes/default/firmwareParams/ctrlMel/ki_xy: 0.05
 * /crazyflieTypes/default/firmwareParams/ctrlMel/ki_z: 0.05
 * /crazyflieTypes/default/firmwareParams/ctrlMel/kp_xy: 0.4
 * /crazyflieTypes/default/firmwareParams/ctrlMel/kp_z: 1.25
 * /crazyflieTypes/default/firmwareParams/ctrlMel/kw_xy: 20000
 * /crazyflieTypes/default/firmwareParams/ctrlMel/kw_z: 12000
 * /crazyflieTypes/default/firmwareParams/ctrlMel/mass: 0.032
 * /crazyflieTypes/default/firmwareParams/ctrlMel/massThrust: 132000
 * /crazyflieTypes/default/markerConfiguration: 0
 * /crazyflieTypes/large/batteryVoltageWarning: 11.4
 * /crazyflieTypes/large/batteryVoltateCritical: 11.1
 * /crazyflieTypes/large/bigQuad: True
 * /crazyflieTypes/large/dynamicsConfiguration: 0
 * /crazyflieTypes/large/firmwareParams/ctrlMel/i_range_m_z: 1500
 * /crazyflieTypes/large/firmwareParams/ctrlMel/i_range_xy: 2.0
 * /crazyflieTypes/large/firmwareParams/ctrlMel/i_range_z: 4.0
 * /crazyflieTypes/large/firmwareParams/ctrlMel/kR_xy: 30000
 * /crazyflieTypes/large/firmwareParams/ctrlMel/kR_z: 40000
 * /crazyflieTypes/large/firmwareParams/ctrlMel/kd_omega_rp: 100
 * /crazyflieTypes/large/firmwareParams/ctrlMel/kd_xy: 3.0
 * /crazyflieTypes/large/firmwareParams/ctrlMel/kd_z: 3.0
 * /crazyflieTypes/large/firmwareParams/ctrlMel/ki_m_z: 500
 * /crazyflieTypes/large/firmwareParams/ctrlMel/ki_xy: 1.0
 * /crazyflieTypes/large/firmwareParams/ctrlMel/ki_z: 2.0
 * /crazyflieTypes/large/firmwareParams/ctrlMel/kp_xy: 8.0
 * /crazyflieTypes/large/firmwareParams/ctrlMel/kp_z: 8.0
 * /crazyflieTypes/large/firmwareParams/ctrlMel/kw_xy: 9000
 * /crazyflieTypes/large/firmwareParams/ctrlMel/kw_z: 10000
 * /crazyflieTypes/large/firmwareParams/ctrlMel/mass: 0.38
 * /crazyflieTypes/large/firmwareParams/ctrlMel/massThrust: 5400
 * /crazyflieTypes/large/markerConfiguration: 2
 * /crazyflieTypes/medium/batteryVoltageWarning: 7.6
 * /crazyflieTypes/medium/batteryVoltateCritical: 7.4
 * /crazyflieTypes/medium/bigQuad: True
 * /crazyflieTypes/medium/dynamicsConfiguration: 0
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/i_range_m_z: 1500
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/i_range_xy: 2.0
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/i_range_z: 2.0
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/kR_xy: 50000
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/kR_z: 60000
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/kd_omega_rp: 200
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/kd_xy: 1.0
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/kd_z: 1.0
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/ki_m_z: 500
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/ki_xy: 0.1
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/ki_z: 0.5
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/kp_xy: 2.0
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/kp_z: 3
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/kw_xy: 16000
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/kw_z: 12000
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/mass: 0.11
 * /crazyflieTypes/medium/firmwareParams/ctrlMel/massThrust: 23000
 * /crazyflieTypes/medium/markerConfiguration: 1
 * /crazyflies: [{'initialPositio...
 * /crazyswarm_server/broadcast_address: FFE7E7E7E7
 * /crazyswarm_server/enable_logging: True
 * /crazyswarm_server/enable_parameters: True
 * /crazyswarm_server/firmware: crazyswarm
 * /crazyswarm_server/firmwareParams/commander/enHighLevel: 1
 * /crazyswarm_server/firmwareParams/ring/effect: 16
 * /crazyswarm_server/firmwareParams/ring/headlightEnable: 0
 * /crazyswarm_server/firmwareParams/ring/solidBlue: 255
 * /crazyswarm_server/firmwareParams/ring/solidGreen: 0
 * /crazyswarm_server/firmwareParams/ring/solidRed: 0
 * /crazyswarm_server/firmwareParams/stabilizer/controller: 2
 * /crazyswarm_server/firmwareParams/stabilizer/estimator: 3
 * /crazyswarm_server/force_no_cache: False
 * /crazyswarm_server/genericLogTopicFrequencies: [10]
 * /crazyswarm_server/genericLogTopic_log1_Variables: ['stateEstimate.y...
 * /crazyswarm_server/genericLogTopics: ['log1']
 * /crazyswarm_server/motion_capture_type: optitrack
 * /crazyswarm_server/object_tracking_type: motionCapture
 * /crazyswarm_server/optitrack_host_name: 192.168.1.10
 * /crazyswarm_server/print_latency: False
 * /crazyswarm_server/save_point_clouds: ~/pointCloud.ot
 * /crazyswarm_server/world_frame: /world
 * /crazyswarm_server/write_csvs: False
 * /crazyswarm_teleop/csv_file: /home/ss/crazyswa...
 * /crazyswarm_teleop/timescale: 0.8
 * /dynamicsConfigurations/0/maxFitnessScore: 0.001
 * /dynamicsConfigurations/0/maxPitch: 1.4
 * /dynamicsConfigurations/0/maxPitchRate: 20.0
 * /dynamicsConfigurations/0/maxRoll: 1.4
 * /dynamicsConfigurations/0/maxRollRate: 20.0
 * /dynamicsConfigurations/0/maxXVelocity: 2.0
 * /dynamicsConfigurations/0/maxYVelocity: 2.0
 * /dynamicsConfigurations/0/maxYawRate: 10.0
 * /dynamicsConfigurations/0/maxZVelocity: 3.0
 * /joy/dev: /dev/input/js0
 * /markerConfigurations/0/numPoints: 4
 * /markerConfigurations/0/offset: [0.0, -0.01, -0.04]
 * /markerConfigurations/0/points/0: [0.0177184, 0.013...
 * /markerConfigurations/0/points/1: [-0.0262914, 0.05...
 * /markerConfigurations/0/points/2: [-0.0328889, -0.0...
 * /markerConfigurations/0/points/3: [0.0431307, -0.03...
 * /markerConfigurations/1/numPoints: 4
 * /markerConfigurations/1/offset: [0.0, 0.0, -0.03]
 * /markerConfigurations/1/points/0: [-0.00896228, -0....
 * /markerConfigurations/1/points/1: [-0.0156318, 0.09...
 * /markerConfigurations/1/points/2: [0.0461693, -0.08...
 * /markerConfigurations/1/points/3: [-0.0789959, -0.0...
 * /markerConfigurations/2/numPoints: 4
 * /markerConfigurations/2/offset: [0.0, 0.0, -0.06]
 * /markerConfigurations/2/points/0: [0.0558163, -0.00...
 * /markerConfigurations/2/points/1: [-0.0113941, 0.00...
 * /markerConfigurations/2/points/2: [-0.0306277, 0.05...
 * /markerConfigurations/2/points/3: [0.0535816, -0.04...
 * /markerConfigurations/3/numPoints: 1
 * /markerConfigurations/3/offset: [0.0, -0.01, -0.04]
 * /markerConfigurations/3/points/0: [0.0177184, 0.013...
 * /numDynamicsConfigurations: 1
 * /numMarkerConfigurations: 4
 * /rosdistro: kinetic
 * /rosversion: 1.12.15

NODES
  /
    crazyswarm_server (crazyswarm/crazyswarm_server)
    crazyswarm_teleop (crazyswarm/crazyswarm_teleop)
    joy (joy/joy_node)
    rviz (rviz/rviz)

auto-starting new master
process[master]: started with pid [19184]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 7d2ef0e2-ea0e-11ea-a663-200db01c05e8
process[rosout-1]: started with pid [19197]
started core service [/rosout]
process[crazyswarm_server-2]: started with pid [19214]
ERROR: cannot launch node of type [joy/joy_node]: joy
ROS path [0]=/opt/ros/kinetic/share/ros
ROS path [1]=/home/ss/crazyswarm2019/crazyswarm/ros_ws/src
ROS path [2]=/opt/ros/kinetic/share
process[crazyswarm_teleop-4]: started with pid [19215]
process[rviz-5]: started with pid [19216]
[ INFO] [1598715854.711953101]: Wait for services...
[ INFO] [1598715854.712646666]: waitForService: Service [/emergency] has not been advertised, waiting...
[ INFO] [1598715854.734133956]: waitForService: Service [/emergency] is now available.
[ INFO] [1598715854.735849967]: Manager ready.
ch: 1
[ INFO] [1598715854.809451011]: Adding CF: cf1 (radio://0/80/2M/E7E7E7E701, cf1)...
[ INFO] [1598715854.825031971]: CF ctor: 0.015501 s
[ INFO] [1598715854.827767883]: [cf1] Requesting parameters...
[ WARN] [1598715854.829955190]: Don't know ack: Port: 13 Channel: 1 Len: 6
[ WARN] [1598715854.831446610]: Don't know ack: Port: 13 Channel: 1 Len: 6
[ INFO] [1598715854.832304646]: Found variables in cache.
[ INFO] [1598715855.088559364]: [cf1] reqParamTOC: 0.263457 s
[ INFO] [1598715855.088587966]: [cf1] Requesting logging variables...
[ WARN] [1598715855.091504408]: Don't know ack: Port: 5 Channel: 0 Len: 10
[ INFO] [1598715855.091542014]: Found variables in cache.
[ INFO] [1598715855.091877254]: [cf1] reqLogTOC: 0.003336 s
[ WARN] [1598715855.093201674]: Don't know ack: Port: 5 Channel: 0 Len: 10
[ INFO] [1598715855.097316764]: [cf1] logBlocks: 0.263457 s
[ INFO] [1598715855.097339939]: Requesting memories...
[ INFO] [1598715855.100340427]: Memories: 7
[ INFO] [1598715855.120649152]: [cf1] Ready. Elapsed: 0.295546 s
[ INFO] [1598715855.120681090]: CF run: 0.295672 s
update commander/enHighLevel to 1
update ring/effect to 16
update ring/headlightEnable to 0
update ring/solidBlue to 255
update ring/solidGreen to 0
update ring/solidRed to 0
update stabilizer/controller to 2
update stabilizer/estimator to 3
update ctrlMel/i_range_m_z to 1500
update ctrlMel/i_range_xy to 2
update ctrlMel/i_range_z to 0.4
update ctrlMel/kR_xy to 70000
update ctrlMel/kR_z to 60000
update ctrlMel/kd_omega_rp to 200
update ctrlMel/kd_xy to 0.2
update ctrlMel/kd_z to 0.4
update ctrlMel/ki_m_z to 500
update ctrlMel/ki_xy to 0.05
update ctrlMel/ki_z to 0.05
update ctrlMel/kp_xy to 0.4
update ctrlMel/kp_z to 1.25
update ctrlMel/kw_xy to 20000
update ctrlMel/kw_z to 12000
update ctrlMel/mass to 0.032
update ctrlMel/massThrust to 132000
[ INFO] [1598715855.138673917]: [cf1] Update parameters
[ INFO] [1598715855.208551016]: Update params: 0.087821 s
[ INFO] [1598715855.208706574]: Started 1 threads
[ WARN] [1598715855.226508278]: Latency too high! Is 0.017654 s.
[ INFO] [1598715861.133347997]: Takeoff!
[ INFO] [1598715863.662526160]: [cf1] GoTo
[ INFO] [1598715867.060084864]: NextPhase!
[ INFO] [1598715867.070247018]: Land!
[ WARN] [1598715868.886149911]: No updated pose for motion capture object cf1
[ INFO] [1598715869.084089777]: CF Console: SYS: ----------------------------
[ INFO] [1598715869.118583083]: CF Console: SYS: Crazyflie 2.1 is up and running!
[ INFO] [1598715869.141829917]: CF Console: SYS: Build 757:217cff5db4b2 (2016.09 +757) CLEAN
[ INFO] [1598715869.176499341]: CF Console: SYS: I am 0x203936564E4B501400200048 and I have 1024KB of flash!
[ INFO] [1598715869.188396121]: CF Console: CFGBLK: v1, verification [OK]
[ INFO] [1598715869.200080126]: CF Console: DECK_CORE: 1 deck(s) found
[ INFO] [1598715869.223598143]: CF Console: DECK_CORE: Calling INIT on driver bcLedRing for deck 0
[ INFO] [1598715870.126513628]: CF Console: IMU: BMI088 Gyro I2C connection [OK].
[ INFO] [1598715870.183363010]: CF Console: IMU: BMI088 Accel I2C connection [OK]
[ INFO] [1598715870.206584802]: CF Console: IMU: BMP388 I2C connection [OK]
[ INFO] [1598715870.230132433]: CF Console: ESTIMATOR: Using Complementary (1) estimator
[ INFO] [1598715870.253263578]: CF Console: CONTROLLER: Using PID (1) controller
[ INFO] [1598715870.722662752]: CF Console: EEPROM: I2C connection [OK].
[ INFO] [1598715871.537904803]: CF Console: SYS: The system resumed after watchdog timeout [WARNING]
[ INFO] [1598715871.561548518]: CF Console: SYS: Assert failed at src/drivers/src/nvic.c:142

[whoenig]

Sorry for the delay - I was on vacation with limited Internet access.

Latency: I think 20ms is OK - I have seen such numbers on other machines without any issues flying as well. For the measurement, the PC sends a message with a timestamp to the CF, the CF sends it back, and then the latency is computed as (arrival time - sent time) / 2. Higher latency can depend on many settings, including which USB port you use to attach your Crazyradio to and what decks you have attached to your CF.

Tracking: In the single-marker case the attitude is computed using the on-board EKF. This is very robust as long as you don’t hover for > 30s in one place (i.e., the EKF requires you to move in the xy-plane to recover the attitude correctly). In the four-marker case, the attitude is directly measured by your mocap. If your mocap would operate perfectly, four-markers are better. However, in many practical cases the coverage of the Mocap is not perfect, or the markers are too small. If you have temporary occlusions, the single-marker mode can be more robust in recovering a drone. I personally almost always use the single-marker mode these days. The “correct” marker size depends on your mocap (camera resolution + distance). You should switch to grayscale mode in Motive and count the number of pixel that a marker takes in the camera frame - at least 5 pixels diameter are recommended for Optitrack.

Firmware: I’ve seen the OW: Cmd 0x21 timeout error as well and it should not affect correct flight operation.

Flight Issues: I assume you can fly cf1 well independently and cf2 also well independently, but the issues do occur when flying both at the same time? I suggest you repeat your earlier tests: Do not fly, but move around both drones manually with logging enabled and make sure that the result is as expected. From your description, it sounds like cf2 either does not get any mocap update, or it receives the mocap update for cf1 instead. Also make sure that your NRF51 firmware is up-to-date.

[swarm5]

Thank you very much for your suggestion and wish you a happy holiday.