Intermittent BrokenPipe

Hi,

I’m getting this error:

Traceback (most recent call last):
  File "/home/snpowers/Git/continual-reinforcement-learning/continual_rl/envs/thor/thor_general_env.py", line 135, in try_step
    event = controller.step(**thor_action_data)
  File "/home/snpowers/miniconda3/envs/venv_continual_jan2021_manual/lib/python3.8/site-packages/ai2thor/controller.py", line 916, in
step
    self.server.send(action)
  File "/home/snpowers/miniconda3/envs/venv_continual_jan2021_manual/lib/python3.8/site-packages/ai2thor/fifo_server.py", line 196, in
 send
    self._send_message(
  File "/home/snpowers/miniconda3/envs/venv_continual_jan2021_manual/lib/python3.8/site-packages/ai2thor/fifo_server.py", line 175, in
 _send_message
    self.client_pipe.flush()
BrokenPipeError: [Errno 32] Broken pipe
ERROR:/glusterfs/snpowers/thor_logs/thor_main_logs/thor_2021-07-23 03:20:14.973923.log:Stepping environment with action {'action': 'Pass'} failed with exception: [Errno 32] Broken pipe.

randomly (generally after about 3 hours of running or so - about 300k steps). The Player.log contains no error info to speak of (happy to post if desired). Any idea what might be happening to the Thor process to cause it to fail in this way? Or where a more helpful error message might be located? I would have assumed that generally, any failed actions would have ‘lastActionSuccess’ as False, with something populating the “errorMessage”, instead of this rather catastrophic crash. The action that was taken immediately prior to the start of the BrokenPipe messages was a simple “MoveAhead” action that seemed to succeed.

I’m on a build that was sent to me to test out the CloudRendering pipeline. It has generally been working, but this is causing some problems.

Thanks!

[laurimi]

@qazwini I just re-initialize the controller object by calling the constructor/init method, since I’m not sure if it is possible to do it otherwise. I have to set again all the controller options, but this is not an issues usually since the crashes happen for me at reset when I am setting everything up anyway.

[qazwini]

I need to run a bunch of Thor processes, but it hangs. Is there a way around this?