503 Error when pushing repeatedly

Describe the bug

Hello 👋 ,

We are currently uploading models programmatically to Hub for testing purposes of enabling widgets for the models of our new library. See code below:

def push_repo(repo_name, local_repo):
    # this token should be allowed to push to the skops-tests org.
    token = os.environ["SKOPS_TESTS_TOKEN"]
    repo_id = f"skops-tests/{repo_name}"

    print(f"Pushing {repo_id}")

    client = HfApi()
    try:
        client.delete_repo(repo_id, token=token)
    except RepositoryNotFoundError:
        # repo does not exist yet
        pass
    client.create_repo(repo_id=repo_id, token=token, repo_type="model")

    client.upload_folder(
        repo_id=repo_id,
        path_in_repo=".",
        folder_path=local_repo,
        commit_message="pushing files to the repo from test generator!",
        commit_description=None,
        token=token,
        repo_type=None,
        revision=None,
        create_pr=False,
    )
    # prevent AWS "503 Server Error: Slow Down for url" error
    time.sleep(SLEEP_BETWEEN_PUSHES)

If I don’t put it to sleep between pushes I get 503 error. The sleep time increases as we upload more models.

I wonder if this is the right repository to open the issue in and if this is an edge case (or vice versa, something you might want to prevent).

Reproduction

No response

Logs

Traceback (most recent call last):
  File "/opt/anaconda3/envs/api_inf/lib/python3.9/site-packages/huggingface_hub/_commit_api.py", line 257, in upload_lfs_files
    future.result()
  File "/opt/anaconda3/envs/api_inf/lib/python3.9/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/opt/anaconda3/envs/api_inf/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/opt/anaconda3/envs/api_inf/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/anaconda3/envs/api_inf/lib/python3.9/site-packages/huggingface_hub/_commit_api.py", line 301, in _upload_lfs_object
    lfs_upload(
  File "/opt/anaconda3/envs/api_inf/lib/python3.9/site-packages/huggingface_hub/lfs.py", line 283, in lfs_upload
    _upload_single_part(
  File "/opt/anaconda3/envs/api_inf/lib/python3.9/site-packages/huggingface_hub/lfs.py", line 311, in _upload_single_part
    upload_res.raise_for_status()
  File "/opt/anaconda3/envs/api_inf/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 503 Server Error: Slow Down for url: https://s3.us-east-1.amazonaws.com/lfs.huggingface.co/repos/..

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/mervenoyan/Desktop/inference-api-comm/api-inference-community/docker_images/sklearn/tests/generators/generate.py", line 251, in <module>
    main(sklearn_version)
  File "/Users/mervenoyan/Desktop/inference-api-comm/api-inference-community/docker_images/sklearn/tests/generators/generate.py", line 235, in main
    create_repos(
  File "/Users/mervenoyan/Desktop/inference-api-comm/api-inference-community/docker_images/sklearn/tests/generators/generate.py", line 147, in create_repos
    push_repo(
  File "/Users/mervenoyan/Desktop/inference-api-comm/api-inference-community/docker_images/sklearn/tests/generators/generate.py", line 59, in push_repo
    client.upload_folder(
  File "/opt/anaconda3/envs/api_inf/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 2334, in upload_folder
    pr_url = self.create_commit(
  File "/opt/anaconda3/envs/api_inf/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 2024, in create_commit
    upload_lfs_files(
  File "/opt/anaconda3/envs/api_inf/lib/python3.9/site-packages/huggingface_hub/_commit_api.py", line 260, in upload_lfs_files
    raise RuntimeError(
RuntimeError: Error while uploading skops-jntrddz2.pkl to the Hub

System Info

0.9.1

[Pierrci]

Did we already encounter other error codes than 503? If not, I think that we can focus on 503.

Upload of non-LFS files are not a problem, right ?

Correct, they’re not uploaded to S3 so they’re not a problem.

[Wauplin]

@SBrandeis Yes that’s definitely something we can do. It would require to tweak _request_wrapper to handle the exponential backoff on specific error codes.

I can see a need for the backoff in _upload_single_part and _upload_multi_part. Upload of non-LFS files are not a problem, right ?

And finally, for the error codes themselves I would be more in favor to filter only “503 service unavailable” since it’s really in this case that retrying is useful (maybe also “504 gateway timeout” ?). But I’m fine if you tell me that other error codes happen as well.