RFE: store resource files compressed

I’ve noticed that data directory is 39Mb. I think it can be compressed with e.g. zstd and decompressed on the fly, without large performance penalties, but with 7x size reduction (up to 5.3Mb):

$ cat lib/python3.7/site-packages/botocore/data/ec2/2016-11-15/service-2.json | pv >/dev/null
1.52MiB 0:00:00 [1.19GiB/s] [  <=>                                                                                                             ]

$ cat lib/python3.7/site-packages/botocore/data/ec2/2016-11-15/service-2.json | zstd -9 - > /tmp/compressed-services.json.zstd

$ ls -lh /tmp/compressed-services.json.zstd 
-rw-r--r-- 1 danbst users 148K гру 28 03:01 /tmp/compressed-services.json.zstd

$ cat /tmp/compressed-services.json.zstd | zstd -d - | pv > /dev/null
1.52MiB 0:00:00 [ 474MiB/s] [  <=>                                                                                                             ]

Perhaps this should only be done for python package, but source should be left in JSON.

[dacut]

In case you’re curious: pickling (instead of JSON) reduces the file load time for zstd from 21 msec to 13 msec on my laptop.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don’t have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.