Texture cache, new tmod implementation

Just here copying things there were said. Improve later…

@Chicken Bones hey
does your texture loading system load all textures in at the same time?
i SUDDENLY noticed that 400mb of memory that overhaul uses is just textures
99% of them aren't items and aren't needed immediately
if it's 400mb with overhaul, wouldn't it be like 1gb with content mods?
:GWfroggyMonkaThink:
[13:28][hamstar]
.tmod files would need to turn into installers of sorts(edited)
arent they compressed?
reading from disk from a compressed folder would be slow i'd imagine
[13:33][Chicken Bones]
@A Wild Mirsario™ ours does, and afaik Terraria does the same, we should be deallocating the local copies though, it should all be uploaded to gpu
[13:33][A Wild Mirsario™]
terraria only instantly loads item textures iirc(edited)
npcs for example are loaded on need
[13:33][Chicken Bones]
ooh, do some digging for us or ask jopo
[13:34][A Wild Mirsario™]
what digging is there to do?

all of this is booleans for whether or not textures of entities were loaded
tiles, gore, npcs, projectiles
basically everything big but items
ewredcodebtw
[13:35][Chicken Bones]
I see
[13:36][A Wild Mirsario™]
so i have 9 megabytes of textures

[13:36][Chicken Bones]
well I imagine that's quite a reasonable idea
[13:36][A Wild Mirsario™]
and only 40kb are items which need to be loaded on runtime

[13:36][Chicken Bones]
unfortunately zip files aren't random access files
so a solution is difficult to implement without some sort of cache similar to how music is handled
[13:37][A Wild Mirsario™]
cache should be fine tbh with how small textures are usually
especially when there's no texture unloading during gameplay
:thinkingwithblobs:
[13:38][Chicken Bones]
yeah, but updating and maintaining it is a pain
even so, you have 9MB not 400
[13:38][Shockah]
uhhh pretty sure ZIPs are fine with random access
7zs aren't
[13:38][A Wild Mirsario™]
they're 400MB in memory
that's all .png files
not raw bitmaps
[13:39][Chicken Bones]
why are we storing RAW bitmaps anywhere but gpu memory?
thoguh still, that's a regrettable amount of texture memory to consume
[13:39][A Wild Mirsario™]
i'm assuming that you'll need cache for when textures are only stored in gpu
[13:39][Chicken Bones]
@Shockah actually, you're correct
unfortuantely we don't use zip files directly
because I didn't look into integrity validation
the entire content payload is currently a gzipstream
[13:40][hamstar]
cache, installer, same difference?
[13:41][Chicken Bones]
we should be able to store them in compressed form in RAM
(png)
as a suitable compromise
[13:42][A Wild Mirsario™]
what's faster, uncompressing png and parsing it, or loading .rawimg from disk?
:GWseremePeepoThink:
[13:43][hamstar]
depends on how often the mods are loaded?
hmm, no maybe not
[13:47][Chicken Bones]
oh, I remember, exactly why I made them rawimg in the first place
because it was so slow to load png
I don't really want to abuse user's disk space with rawimg
we could recompress them individually (like xnb files)
but still, yuck
[13:48][A Wild Mirsario™]
maybe if the cache was a .zip file it'd still be faster?
idk if .zip reading speed is dependent on the amount of files in it(edited)
[13:49][Chicken Bones]
if tmod files were zip files it'd be better
but I didn't think that far ahead
[13:50][Shockah]
time for a new format version
[13:50][A Wild Mirsario™]
they can be made that
[13:50][Chicken Bones]
unfortunately no archive format has support for built in signing
[13:50][A Wild Mirsario™]
just leave some legacy loading code in
[13:51][Chicken Bones]
well a key feature of tml mod format is signing to prevent accidental or intentional tampering or masquerading as a different author (or from the Mod Browser)
zip files wouldn't support that
because there's no way to distribute the signature in the file
currently, all mod files are stored in a compressed stream which is read once on mod load
rawimg files are directly compressed within this stream (allowing their compression to take advantage of similarities to other image files as well)
the problem is that loading all textures into gpu is too memory hungry
which means we need to store them somewhere, compressed
the reasonable solution is to compress them either into a dictionary in RAM, or onto disk
[13:54][Shockah]
ZIP files have a Comment field, don't they?
yeah they do
i mean, ZIP entries
[13:55][Chicken Bones]
oh, no you need to sign the whole zip
not just each file within
[13:55][hamstar]
can't just use a CRC check?
[13:55][Chicken Bones]
CRC is not secure
[13:56][hamstar]
my impression was it's not easy to exploit
but i'm not an expert
[13:56][Chicken Bones]
sorry hamstar
storing in RAM is fairly reasonable, especially if say, Calamity, only had < 50MB of compressed textures
[13:57][A Wild Mirsario™]
can't you just insert binary data before zip's data begins?
:rolling_eyes:
[13:57][Chicken Bones]
you'd need to somehow remove that data before you used any zip parsing library
[13:58][A Wild Mirsario™]
isn't that done by just giving the lib a stream with an offset?
[13:58][Shockah]
^
[14:00][Chicken Bones]
I'm not aware you can feed streams and offsets, though you could possible make a wrapper stream that incorporates the offset, and is seekable
[14:00][Shockah]
well that was the idea
[14:00][Chicken Bones]
in that case though, we don't even need to use zip, we could just make our own file table and compress them individually(edited)
[14:00][Shockah]
skip the signature in the beginning of the file
and then feed the rest to the parser
[14:01][Chicken Bones]
our current format just compresses the entire data blob
which has slight advantages
really, the best option is to stop doing that
move all our file headers to a table with offsets
rather than being interspersed
I don't think the compression ratio will suffer significantly```

[Chicken-Bones]

I’ll probably target this for 1.3.6 as I believe there’s improvements to the content loading system

[Chicken-Bones]

Follow https://github.com/blushiemagic/tModLoader/issues/486 for updates