housekeeping: Remove large files from git history

• image files
• pdf files

set -xe

REPO_URL=https://github.com/intel/dffml
BRANCH=$BRANCH
TEMPDIR=$(mktemp -d)
GIT_FILTER_REPO_PATHS=$(mktemp)

rm -rf $TEMPDIR
mkdir $TEMPDIR
cd $TEMPDIR
git init
git remote add origin $REPO_URL
git fetch origin $BRANCH
git reset --hard origin/$BRANCH
git log -n 3 --oneline
git log --stat $BRANCH | grep -E '\.png|\.jpeg|\.jpg|\.gif'
cat > $GIT_FILTER_REPO_PATHS <<'EOF'
glob:*.gif
glob:*.png
glob:*.jpg
glob:*.jpeg
EOF
git filter-repo --force --invert-paths --paths-from-file $GIT_FILTER_REPO_PATHS
git diff --stat origin/$BRANCH
git log --stat $BRANCH | grep -E '\.png|\.jpeg|\.jpg|\.gif'
git branch

[pdxjohnny]

The repo is still way to big for some reason.

$ git clone https://github.com/intel/dffml test-dffml-2022-03-13-10-06-no-gifs-supposed-to-be
Cloning into 'test-dffml-2022-03-13-10-06-no-gifs-supposed-to-be'...
remote: Enumerating objects: 108172, done.
remote: Counting objects: 100% (8778/8778), done.
remote: Compressing objects: 100% (399/399), done.
remote: Total 108172 (delta 8463), reused 8394 (delta 8377), pack-reused 99394 eceiving objects: 100% (108172/1081
Receiving objects: 100% (108172/108172), 577.15 MiB | 13.44 MiB/s, done.
Resolving deltas: 100% (86371/86371), done.
$ cd test-dffml-2022-03-13-10-06-no-gifs-supposed-to-be
$ du -h --threshold=400k -a .
2.9M    ./.git/objects/pack/pack-3bddcee5979f5e78334ceae49f719fa4643e3c7f.idx
578M    ./.git/objects/pack/pack-3bddcee5979f5e78334ceae49f719fa4643e3c7f.pack
581M    ./.git/objects/pack
581M    ./.git/objects
581M    ./.git
920K    ./dffml
1.5M    ./docs/images/BSidesPDX_2019_Down_The_Dependency_Rabbit_Hole.pdf
3.3M    ./docs/images/GSoC_2019_Models.pdf
536K    ./docs/images/Theory_Operation_and_Application_of_Neural_Networks.pdf
5.7M    ./docs/images
6.3M    ./docs
936K    ./examples/ffmpeg/input.mp4
960K    ./examples/ffmpeg
720K    ./examples/notebooks/transferlearning.ipynb
868K    ./examples/notebooks
2.3M    ./examples
652K    ./model
592M    .

[pdxjohnny]

While diffing manifest-main and manifest

$ git diff origin/manifest | grep Binary
Binary files a/docs/examples/webhook/images/github_settings.png and /dev/null differ
Binary files a/docs/examples/webhook/images/localhost_run.png and /dev/null differ
Binary files a/docs/examples/webhook/images/ngrok_out.png and /dev/null differ
Binary files a/docs/images/how-to-read-ci-tests.png and /dev/null differ
Binary files a/docs/images/maintainance-arch.png and /dev/null differ
Binary files a/docs/images/oarch.jpg and /dev/null differ
Binary files a/docs/images/rubric-table.png and /dev/null differ
Binary files a/docs/images/website-before.png and /dev/null differ
Binary files a/examples/MNIST/image1.png and /dev/null differ
Binary files a/examples/MNIST/image2.png and /dev/null differ
Binary files a/examples/MNIST/image3.png and /dev/null differ
Binary files a/examples/MNIST/image4.png and /dev/null differ
Binary files a/examples/flower17/buttercup.jpg and /dev/null differ
Binary files a/examples/flower17/daisy.jpg and /dev/null differ
Binary files a/examples/flower17/pansy.jpg and /dev/null differ
Binary files a/examples/flower17/tigerlily.jpg and /dev/null differ
Binary files a/examples/shouldi/shouldi.jpg and /dev/null differ
$ git diff origin/manifest | grep Binary | awk '{print $3}' | sed -e 's/a\///'
docs/examples/webhook/images/github_settings.png
docs/examples/webhook/images/localhost_run.png
docs/examples/webhook/images/ngrok_out.png
docs/images/how-to-read-ci-tests.png
docs/images/maintainance-arch.png
docs/images/oarch.jpg
docs/images/rubric-table.png
docs/images/website-before.png
examples/MNIST/image1.png
examples/MNIST/image2.png
examples/MNIST/image3.png
examples/MNIST/image4.png
examples/flower17/buttercup.jpg
examples/flower17/daisy.jpg
examples/flower17/pansy.jpg
examples/flower17/tigerlily.jpg
examples/shouldi/shouldi.jpg