LFS: Authorization error when uploading many/large files.

The following error occurs when uploading ~1000 files (300GB total):

LFS: Authorization error: https://s3.us-east-1.amazonaws.com/lfs.huggingface.co/datasets/lvwerra/github-code/XXXXXX
Check that you have proper access to the repository

I am using the following line in Python to add/commit/push the commits:

from huggingface_hub import Repository

repo = Repository(
        local_dir="./data/",
        clone_from="lvwerra/github-code",
        repo_type="dataset",
        private=True,
        use_auth_token=True,
        git_user="lvwerra",
        git_email="[e-mail]",
        )

repo.git_add()
repo.git_commit("Test commit")
repo.git_push()

The upload seems to mostly succeed (I can see progress bars for the uploads) and at the end there are a few hundred of these messages. I assume that authorization worked at least initially since I was able to create the repo in the first place and I tested that I can push data to another test repo with the same settings after the failure.

Could this be related to this issue and be a problem on the server side?

Version: huggingface-hub 0.4.0

cc @LysandreJik @julien-c

[lvwerra]

No it did not - it happened overnight and it started around 11pm and I noticed it failed at 9am.

So using git lfs worked and was much faster. In Python pushing uploaded files individually and I could see that each file was uploading at around 1MB/s (sometimes a bit faster or slower) and maybe 10-20 files were uploaded concurrently (did not count) amounting to 10-20MB/s upload total. With git push in the command line it uploaded with 300MB/s!

[julien-c]

any way you could try with git lfs and not Repository?