tmp_log.txt error

Need better handling of this error. It occurs

Exception in thread Thread-43:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/elpis/elpis/wrappers/objects/model.py", line 239, in background_train_task
    train()
  File "/elpis/elpis/wrappers/objects/model.py", line 232, in train
    p = run(f"cd {local_kaldi_path}; ./run.sh > {tmp_log_path}")
  File "/elpis/elpis/wrappers/objects/command.py", line 16, in run
    stderr=subprocess.STDOUT
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['bash', '-c', 'cd /elpis/state/models/ab6a2222936e2ad98bcc8232e8620674/kaldi; ./run.sh > /elpis/state/tmp_log.txt']' returned non-zero exit status 1.

Hard to reproduce, but the few times I have seen it the tmp_log.txt file ends after the aligning data line:

steps/train_deltas.sh: training pass 20
steps/train_deltas.sh: aligning data

[shuttle1987]

Just by the way you can set the current working directory on the python side and that will make the ./run.sh command cleaner

[benfoley]

after logging refactor this should no longer be an issue