Can't create indexes from un-seekable fileobj's

In theory, it should be possible to create indexes from un-seekable archive fileobj’s (by using peek instead of read to check file headers, for example). I was able to get this to work by modifying an older version of ratarmount core here (https://github.com/codalab/codalab-worksheets/pull/4212/files#diff-ad5ad76eb55b6437b2e1aa24b324c6d11d176be1926ebea1950a006bfce4efbe). It would be nice if we could do the same for the current version of ratarmountcore, though the current version seems to rely a lot more on seek.

[mxmlnkn]

Looks like we might need to resolve pauldmccarthy/indexed_gzip#102 first though…

I was going to write something like that before you did 😉.

And it’s not only indexed_gzip. I wanna swap indexed_gzip with pragzip sometime in the future and it also might have that issue. And the same goes for bz2 and zstd. I know that you probably only need gzip but it feels like a bug if only very specific file formats are supported.

In the end, I agree that it might be useful but it seems difficult to implement with all of ratarmount features. It might be implemented as a separate “function” on top of ratarmountcore because you wouldn’t even need FUSE for that. I imagine something like wget -O- remote.tar.gz | tee downloaded.tar.gz | ratarmount --index-file downloaded.tar.gz.index. Theoretically, it could be enough to detect stdin being written to enter that alternate mode but we might also trigger it explicitly with something akin to a --create-index-from-stream option. The ratarmount CLI gets kinda clunky, I’m wanting to redesign it to something akin to git with subcommands when it makes sense. I didn’t think deeper about it yet, I only categorized the options in the help output.

Currently, I won’t be able to work on this in a timely manner. I will review PRs though if you find the time to take a deeper look into it.

Edit: One such mentioned problematic ratarmount option for streaming support could be --recursive because it might try to seek back. But similar to unsupported file formats, it could be checked if the alternate mode has been activated and print out an error message.

[epicfaace]

Currently, I won’t be able to work on this in a timely manner. I will review PRs though if you find the time to take a deeper look into it.

That works!! Thanks. By the way https://github.com/pauldmccarthy/indexed_gzip/issues/102 is now resolved.