Master branch can't pass make test

Current master branch can’t pass make test. However the failed tests will pass when running unittest separately.

======================================================================
ERROR: test_dm_control_tf_policy (tests.garage.envs.dm_control.test_dm_control_tf_policy.TestDmControlTfPolicy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/code/garage/tests/garage/envs/dm_control/test_dm_control_tf_policy.py", line 38, in test_dm_control_tf_policy
    runner.train(n_epochs=1, batch_size=10)
  File "/root/code/garage/garage/experiment/local_tf_runner.py", line 321, in train
    start_epoch=0)
  File "/root/code/garage/garage/experiment/local_tf_runner.py", line 407, in _train
    self.save(epoch, paths if store_paths else None)
  File "/root/code/garage/garage/experiment/local_tf_runner.py", line 210, in save
    snapshotter.save_itr_params(epoch, params)
  File "/root/code/garage/garage/logger/snapshotter.py", line 85, in save_itr_params
    with open(file_name, 'wb') as file:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpgplyc983/params.pkl'

======================================================================
ERROR: test_cem_cartpole (tests.garage.np.algos.test_cem.TestCEM)
Test CEM with Cartpole-v1 environment.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/code/garage/tests/garage/np/algos/test_cem.py", line 35, in test_cem_cartpole
    n_epochs=5, batch_size=2000, n_epoch_cycles=n_samples)
  File "/root/code/garage/garage/experiment/local_tf_runner.py", line 321, in train
    start_epoch=0)
  File "/root/code/garage/garage/experiment/local_tf_runner.py", line 407, in _train
    self.save(epoch, paths if store_paths else None)
  File "/root/code/garage/garage/experiment/local_tf_runner.py", line 210, in save
    snapshotter.save_itr_params(epoch, params)
  File "/root/code/garage/garage/logger/snapshotter.py", line 85, in save_itr_params
    with open(file_name, 'wb') as file:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpgplyc983/params.pkl'

======================================================================
ERROR: test_cma_es_cartpole (tests.garage.np.algos.test_cma_es.TestCMAES)
Test CMAES with Cartpole-v1 environment.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/code/garage/tests/garage/np/algos/test_cma_es.py", line 33, in test_cma_es_cartpole
    runner.train(n_epochs=1, batch_size=1000, n_epoch_cycles=n_samples)
  File "/root/code/garage/garage/experiment/local_tf_runner.py", line 321, in train
    start_epoch=0)
  File "/root/code/garage/garage/experiment/local_tf_runner.py", line 407, in _train
    self.save(epoch, paths if store_paths else None)
  File "/root/code/garage/garage/experiment/local_tf_runner.py", line 210, in save
    snapshotter.save_itr_params(epoch, params)
  File "/root/code/garage/garage/logger/snapshotter.py", line 85, in save_itr_params
    with open(file_name, 'wb') as file:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpgplyc983/params.pkl'

======================================================================
FAIL: test_trpo_recurrent_cartpole (tests.garage.tf.algos.test_trpo_with_model.TestTRPO)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/code/garage/tests/garage/tf/algos/test_trpo_with_model.py", line 39, in test_trpo_recurrent_cartpole
    assert last_avg_ret > 90
AssertionError

----------------------------------------------------------------------
Ran 623 tests in 789.240s

FAILED (failures=1, errors=3)
Makefile:60: recipe for target 'run-headless' failed
make: *** [run-headless] Error 1

[naeioi]

I agree. This global process snapshotter is flaky. @gautams3 also experienced a similar bug, that there were two instances of snapshotter, of which one is initialized and the other is not.

Because we take snapshot only at LocalRunner, I think it will not be too much effort removing this global snapshotter.

[ryanjulian]

yes please!