IndexError: list index out of range ERROR:root:ERROR: 500: {'exc_info': ...

I have a bunch of experiments (18 to be exact) running on various machines connected to one visdom server. I’m viewing a bunch of these experiments together, and then suddenly the plots stop loading and the server webpage wont load. About this time, the server machine stops accepting ssh connections.

When I restart the server machine and the visdom server, I get this error on the server when I navigate to the visdom server IP:

INFO:tornado.access:101 GET /vis_socket (<MY IP>) 0.42ms
INFO:root:Opened visdom socket from ip: <MY IP>
INFO:tornado.access:101 GET /socket (<MY IP>) 0.56ms
INFO:root:Opened new socket from ip: <MY IP>
INFO:root:comparing envs
ERROR:tornado.application:Uncaught exception POST /compare/DIM_rkhs_1024_dimz_192_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_192_pen_0.1_do_pt041+DIM_rkhs_1024_dimz_512_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_512_pen_0.1_do_pt041+DIM_rkhs_1024_dimz_64_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_64_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_192_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_192_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_512_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_512_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_64_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_64_pen_0.1_do_pt041+DIM_rkhs_512_dimz_192_pen_0.0_do_pt041+DIM_rkhs_512_dimz_192_pen_0.1_do_pt041+DIM_rkhs_512_dimz_512_pen_0.0_do_pt041+DIM_rkhs_512_dimz_512_pen_0.1_do_pt041+DIM_rkhs_512_dimz_64_pen_0.0_do_pt041+DIM_rkhs_512_dimz_64_pen_0.1_do_pt041 (<MY IP>)
HTTPServerRequest(protocol='http', host='<VIZ_IP>', method='POST', uri='/compare/DIM_rkhs_1024_dimz_192_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_192_pen_0.1_do_pt041+DIM_rkhs_1024_dimz_512_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_512_pen_0.1_do_pt041+DIM_rkhs_1024_dimz_64_pen_0.0_do_pt041+DIM_rkhs_1024_dimz_64_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_192_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_192_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_512_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_512_pen_0.1_do_pt041+DIM_rkhs_2048_dimz_64_pen_0.0_do_pt041+DIM_rkhs_2048_dimz_64_pen_0.1_do_pt041+DIM_rkhs_512_dimz_192_pen_0.0_do_pt041+DIM_rkhs_512_dimz_192_pen_0.1_do_pt041+DIM_rkhs_512_dimz_512_pen_0.0_do_pt041+DIM_rkhs_512_dimz_512_pen_0.1_do_pt041+DIM_rkhs_512_dimz_64_pen_0.0_do_pt041+DIM_rkhs_512_dimz_64_pen_0.1_do_pt041', version='HTTP/1.1', remote_ip='<MY IP>')
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1590, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/visdom/server.py", line 51, in _check_auth
    f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/visdom/server.py", line 1001, in post
    self.env_path)
  File "/usr/local/lib/python2.7/dist-packages/visdom/server.py", line 835, in compare_envs
    res = copy.deepcopy(envs[list(envs.keys())[0]])
IndexError: list index out of range
ERROR:root:ERROR: 500: {'exc_info': (<type 'exceptions.IndexError'>, IndexError('list index out of range',), <traceback object at 0x7f2d51c9e950>)}

What is going on here?

[JackUrb]

The index error hits on the server side from landing on the home page? That would imply a different callsite runs into the same problem, which I wasn’t expecting for the home page.

Definitely agreed that it might be an issue with comparing too many envs, seems like the right area for me to investigate when I get a chance to go into this one.

[rdevon]

So i can visit the page, but only the “main” env appears. On the server side, that error I posted appears.

Note also that after restarting the jobs and server, things work fine after a few steps, so whatever was broken get overwritten. However, if I try to view too many envs at once, things crash with the same issues.

Temp solution right now is just not to view too many envs at once.