[FEATURE REQUEST]: Ability to specify path to create index outside system path.

Feature requested Currently, Hyperspace creates indexes under the system path specified by spark.hyperspace.system.path. The user should be able to specify different path to create/search the indexes. Note that #242 removes spark.hyperspace.index.creation.path and spark.hyperspace.index.search.paths since they are not used, but can be brought back when this feature is implemented.

Acceptance criteria

• The user can specify a different path to create an index outside spark.hyperspace.system.path. FYI, this is more or less possible by temporarily setting the spark.hyperspace.system.path to a different location, but this needs to be discusses if that’s a good solution.
• The user can specify multiple paths to “search” indexes to apply.

Success criteria N/A

Additional context N/A

[sezruby]

@Neoix, it depends only for the default value. You can change the path under any other storage.

EDIT: Note that the config should be set before creating Hyperspace object.

Example:

spark.conf.set("spark.hyperspace.system.path", "abfss://<containerName>@<accountName>.dfs.core.windows.net/path/to/indexes")
val hs = new Hyperspace(spark)
hs.createIndex(..)

[paryoja]

Hi @Neoix. I think you can specify spark.hyperspace.system.path with the storage account that will be used.