Paths joined as strings

Looking through the code surrounding dataset downloads, I noticed that in get_local_storage_path two path-representing strings are joined via an f string:

https://github.com/activeloopai/deeplake/blob/392f8e92235b29ffa8bd59ee7d0969c7d4bbc2ee/deeplake/util/storage.py#L170

Are there reasons why these are not Paths? I would have expected the following:

from pathlib import Path

def get_local_storage_path(path: str, prefix: Path) -> Path:
    local_cache_name = path.replace("://", "_")
    local_cache_name = local_cache_name.replace("/", "_")
    local_cache_name = local_cache_name.replace("\\", "_")
    return prefix / local_cache_name

Or at least:

import os

def get_local_storage_path(path: str, prefix: str) -> str:
    local_cache_name = path.replace("://", "_")
    local_cache_name = local_cache_name.replace("/", "_")
    local_cache_name = local_cache_name.replace("\\", "_")
    return os.path.join(prefix, local_cache_name)

By the way, I noticed this because I got a string/path with two slashes (//) from get_local_dataset:

deeplake.util.exceptions.DatasetHandlerError: A dataset does not exist at the download location /tmp/deeplake//hub_activeloop_fashionpedia-train. Cannot use access method 'local'. Use access method 'download' to first download the dataset and then use access method 'local' in subsequent runs.

[FayazRahman]

@jangop Hey there, thanks for raising the issue. There’s no reason for joining the paths as strings and we should do it the way you suggested.

[FayazRahman]

One would not need to switch anything for the typical usecase “download if not downloaded; use local if available”.

Yep, there is no need to switch in this case with the new implementation. Just use access_method="local". Only ever need to use access_method="download" for re-downloading the dataset.

Just looking at the link you shared, it appears that the second point for local is now wrong:

Oops, that’s a mistake in the docs, thanks for pointing it out.