Traffic Manager times out when setting synchronous mode

Raising a new issue:

While running data_phase1 in parallel, I often get the following error, always when trying to turn traffic_manager’s sync mode on or off.

File “/home/aaronhua/WorldOnRails/leaderboard/leaderboard/leaderboard_evaluator.py”, line 162, in _cleanup self.traffic_manager.set_synchronous_mode(False) RuntimeError: rpc::timeout: Timeout of 2000ms while calling RPC function ‘set_synchronous_mode’

(pid=55313) File “/home/aaronhua/WorldOnRails/leaderboard/leaderboard/leaderboard_evaluator.py”, line 238, in _load_and_wait_for_world (pid=55313) self.traffic_manager.set_synchronous_mode(True) (pid=55313) RuntimeError: rpc::timeout: Timeout of 2000ms while calling RPC function ‘set_synchronous_mode’

My guess is that when running in parallel, if one worker happens to be writing data to disk at the end of a run while another worker is trying to reset the simulator, the entire system slows down and triggers the 2 second traffic_manager timeout. I modified the scripts/launch_carla to default to a server timeout of like 20 minutes but still get the above 2s timeout message. I’ve tried looking around to see if we can specify the traffic_manager specific timeout, but there’s no comprehensive list of CarlaUE4.sh command line options.

The closest I can find is here, implying that the C++ implementation of the TrafficManager has a method that sets the set_synchronous_mode timeout. The Python bindings don’t appear to have this method though. Do you have an idea of how I’d resolve this issue?

EDIT: as for your comment in the old thread, I’ve tried looking for errors that precede this one but haven’t been able to find any. The only thing is ERROR: failed to destroy actor 11571 : unable to destroy actor: not found and the like, which appears when cleaning up a completed run and doesn’t appear to actually break anything. I thought it was a timeout issue because I’m usually able to progress through a few routes and repetitions and gather valid data before workers will start to fail because of the error.

[aaronh65]

Sure, I’ll ask in the CARLA repo. Thanks for the responses!

[dotchen]

Hmm I see… I guess the best is probably to raise this issue in the CARLA repo then (maybe also with your system spec, since I personally have not experienced this error without preceding errors so it could be system specific), as this is more of a CARLA issue.

I also recommend decreasing num_workers and see if this problem persists.