`RarArchiveLoader` fails for nested rar archives

🐛 Describe the bug

For this example, I’m using this archive.

---

For a single archive, RarArchiveLoader works fine:

import pathlib

from torchdata.datapipes.iter import RarArchiveLoader, IterableWrapper, FileOpener

dp = IterableWrapper(["hmdb51_org.rar"])
dp = FileOpener(dp, mode="rb")
dp = RarArchiveLoader(dp)
print([pathlib.Path(data[0]).name for data in dp])

['shoot_gun.rar', 'sit.rar', 'situp.rar', 'smile.rar', 'smoke.rar', 'somersault.rar', 'stand.rar', 'swing_baseball.rar', 'sword.rar', 'sword_exercise.rar', 'talk.rar', 'throw.rar', 'turn.rar', 'walk.rar', 'wave.rar', 'brush_hair.rar', 'cartwheel.rar', 'catch.rar', 'chew.rar', 'clap.rar', 'climb.rar', 'climb_stairs.rar', 'dive.rar', 'draw_sword.rar', 'dribble.rar', 'drink.rar', 'eat.rar', 'fall_floor.rar', 'fencing.rar', 'flic_flac.rar', 'golf.rar', 'handstand.rar', 'hit.rar', 'hug.rar', 'jump.rar', 'kick.rar', 'kick_ball.rar', 'kiss.rar', 'laugh.rar', 'pick.rar', 'pour.rar', 'pullup.rar', 'punch.rar', 'push.rar', 'pushup.rar', 'ride_bike.rar', 'ride_horse.rar', 'run.rar', 'shake_hands.rar', 'shoot_ball.rar', 'shoot_bow.rar']

As you can see from the output, the archive is actually a rar of rars. Thus, we need to use RarArchiveLoader again. Trying to iterate now leads to

dp = RarArchiveLoader(dp)
next(iter(dp))

ValueError: whence value 7 unsupported

pointing to

https://github.com/pytorch/data/blob/fb29c810e6e8881e5a3558743d9573cb82134cd6/torchdata/datapipes/iter/util/rar_archive_loader.py#L22

@ejguan could you have a look?

Versions

Current main.

[pmeier]

Even better, thanks!

[ejguan]

@pmeier I have tried to resolve it by adding another wrapper to parse. But, it turns out rarfile seems not feasible to read nested rar file.

I tried the following case to read files in sequence, which should not need our patch, but it simply doesn’t work at all

rar = rarfile.RarFile("/data/home/erjia/data/hmdb51_org.rar")
for info in rar.infolist():
    f = rar.open(inf)
    inn_rar = rarfile.RarFile(f)

This would raise Error.

This issue may not be feasible to fix. How about extracting the outer rar and read from inner one?