🐛[bug] Tensoboard event file upload failed.

Describe the bug

After upgrade to 0.19.4, tensorboard event files are failed to upload to s3 checkpoint bucket after serveral epoch, error code is attached in screenshot section.

Reproduction Steps

1. Upgrade determined from 0.18.4 to 0.19.4.
2. Save images and scalars with TorchWriter

from determined.tensorboard.metric_writers.pytorch import TorchWriter
logger = TorchWriter() # 

# Save to torchwritter every 200 iterations.
logger.add_image(name, image, global_step)
logger.add_scalar(name, value, global_step)

Expected Behavior

No error.

Screenshot

[2022-09-29 16:42:27] [c193c38a] [rank=0] Traceback (most recent call last): <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/opt/conda/lib/python3.8/runpy.py”, line 194, in _run_module_as_main <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] return _run_code(code, main_globals, None, <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/opt/conda/lib/python3.8/runpy.py”, line 87, in _run_code <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] exec(code, run_globals) <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py”, line 132, in <module> <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] sys.exit(main(args.train_entrypoint)) <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/exec/harness.py”, line 123, in main <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] controller.run() <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py”, line 274, in run <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] self._run() <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/pytorch/_pytorch_trial.py”, line 342, in _run <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] self.upload_tb_files() <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/_trial_controller.py”, line 117, in upload_tb_files <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] self.context._core.train.upload_tensorboard_files( <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/core/_train.py”, line 127, in upload_tensorboard_files <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] self._tensorboard_manager.sync(selector, mangler, self._distributed.rank) <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/common/util.py”, line 80, in wrapped <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] return fn(*arg, **kwarg) <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/run/determined/pythonuserbase/lib/python3.8/site-packages/determined/tensorboard/s3.py”, line 64, in sync <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] self.client.upload_file(str(path), self.bucket, key_name) <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/opt/conda/lib/python3.8/site-packages/boto3/s3/inject.py”, line 131, in upload_file <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] return transfer.upload_file( <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] File “/opt/conda/lib/python3.8/site-packages/boto3/s3/transfer.py”, line 293, in upload_file <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] raise S3UploadFailedError( <none> [2022-09-29 16:42:27] [c193c38a] [rank=0] boto3.exceptions.S3UploadFailedError: Failed to upload /tmp/tensorboard-729.d7a76451-81d9-49e4-b2b2-61d46293cf29.6-0/events.out.tfevents.1664468248.exp-729-trial-725-0-729.d7a76451-81d9-49e4-b2b2-61d46293cf29.6.390.0 to ml-checkpoint/51085318-5dd5-45c2-81fd-d3ad495f541c/tensorboard/experiment/729/trial/725/events.out.tfevents.1664468248.exp-729-trial-725-0-729.d7a76451-81d9-49e4-b2b2-61d46293cf29.6.390.0: An error occurred (NoSuchUpload) when calling the UploadPart operation: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. <none> [2022-09-29 16:42:29] [c193c38a] Process 0 exit with status code 1. <none> [2022-09-29 16:42:29] [c193c38a] Terminating remaining workers after failure of Process 0.

Environment

• Device or hardware: Nvidia A100 * 10
• Environment: Kubernetes

Additional Context

No response

[mpkouznetsov]

@sijin-dm , thank you for reporting this. Yes, the partial fix in on master, and that has been released this week (closing the instance of TorchWriter used by our Trial API to report metrics). However, we cannot close the instances of TorchWriter that you create yourself so the workaround I suggested is the only solution for now.

We will revisit our TensorBoard support in the near future but have not scheduled this work yet.

[sijin-dm]

Did you get a chance to try the workaround?

We are trying about 3 days ago, and so far so good. By the way, the same error has happened in every experiment (different tasks and different repo) accidentally. We are using this workaound for every experiment now, even though they may not use TorchWriter. We want to test it longer, and I will update the test result here after maybe one or two weeks if everything is alright.