Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Reduce github repository size

See original GitHub issue

Our .git starts to grow too fast, now. We discussed recently about that with @y0ast and repository size was ~500MB which is rather large. Now, it is 700MB:

/tmp/ignite# du -h -a -d 1
705M    ./.git
4.0K    ./codecov.yml
4.0K    ./.gitignore
28K     ./.circleci
4.0K    ./CODE_OF_CONDUCT.md
4.0K    ./pyproject.toml
916K    ./ignite
4.0K    ./requirements-dev.txt
4.0K    ./CITATION
92K     ./docker
1.2M    ./assets
28K     ./README.md
4.0K    ./LICENSE
12K     ./CONTRIBUTING.md
120K    ./.github
4.0K    ./setup.py
528K    ./docs
1.2M    ./tests
4.0K    ./.pre-commit-config.yaml
15M     ./examples
12K     ./conda.recipe
4.0K    ./setup.cfg
4.0K    ./mypy.ini
723M    .

That time I looked into ressources about reducing the size and it works locally, but it would be helpful to do that on GitHub as well.

Local compressing works:

/tmp/ignite# git gc --aggressive
/tmp/ignite# du -h -a -d 1
112M    ./.git
4.0K    ./codecov.yml
4.0K    ./.gitignore
28K     ./.circleci
4.0K    ./CODE_OF_CONDUCT.md
4.0K    ./pyproject.toml
916K    ./ignite
4.0K    ./requirements-dev.txt
4.0K    ./CITATION
92K     ./docker
1.2M    ./assets
28K     ./README.md
4.0K    ./LICENSE
12K     ./CONTRIBUTING.md
120K    ./.github
4.0K    ./setup.py
528K    ./docs
1.2M    ./tests
4.0K    ./.pre-commit-config.yaml
15M     ./examples
12K     ./conda.recipe
4.0K    ./setup.cfg
4.0K    ./mypy.ini
131M    .

but I have no idea how to apply that to remote…

cc @trsvchn @sdesrozis @fco-dv

Issue Analytics

State:
Created 3 years ago
Reactions:1
Comments:10 (6 by maintainers)

Top GitHub Comments

4reactions

vfdev-5commented, Mar 10, 2021

🎉 I think #1764 has solved the problem 🎉

/tmp# git clone https://github.com/pytorch/ignite.git
Cloning into 'ignite'...
remote: Enumerating objects: 1310, done.
remote: Counting objects: 100% (1310/1310), done.
remote: Compressing objects: 100% (233/233), done.
remote: Total 12057 (delta 1194), reused 1145 (delta 1077), pack-reused 10747
Receiving objects: 100% (12057/12057), 41.17 MiB | 19.61 MiB/s, done.
Resolving deltas: 100% (8434/8434), done.

Thanks again @y0ast for finding the issue ! Very grateful to you !

2reactions

y0astcommented, Mar 9, 2021

I think I found a smoking gun:

The gh-pages branches: https://github.com/pytorch/ignite/tree/gh-pages Has many many commits: https://github.com/pytorch/ignite/commits/gh-pages

Containing many small files: https://github.com/pytorch/ignite/commit/8b7dda3a88ae7d4c4cbd1ea489ec16cc7529208f And 14 individual 3MB files (all `.pickle), like this: https://github.com/pytorch/ignite/blob/8b7dda3a88ae7d4c4cbd1ea489ec16cc7529208f/master/.doctrees/environment.pickle

That’ ~40MB of additions to the .git folder every time the website is deployed (which is for every commit on master!)

So I think a new approach for publishing the website should be considered.