Factor out storage key generation logic, to eliminate code duplication
See original GitHub issueThis is a small-ish refactoring chore. See https://github.com/gojek/feast/pull/360#discussion_r362778747
Currently there is some code duplication between ingestion and serving for the generation and parsing of Redis keys. We multiplied this in the course of implementing Cassandra storage (#360), as it is already a large change set we didn’t want to introduce additional Redis refactoring to it and thus decided to log it as a follow-up.
This could be done for Redis independent of and before integrating the Cassandra implementation though, and Cass implementation can be updated to follow the pattern if so.
Update: The below storage modularization was realized through #529 and subsequent PRs that implemented its interfaces. This issue remains for tech debt of code duplication that still remains, enumerated in https://github.com/gojek/feast/issues/402#issuecomment-623264345
In the interest of storage modularization to minimize dependency headaches in the future (something I believe we’ll hash out in further RFC issues), I propose something like this:
1. A
storage-api
module, which defines an interface something like KeyUtil
1. A storage-redis
implementation module which implements KeyUtil
for Redis
I’m open to better naming suggestions, structure is more my focus here.
The KeyUtil
s could ideally be static
so that unit testing them is trivially simple and the unit tests can serve as a good spec/documentation of the key format.
Issue Analytics
- State:
- Created 4 years ago
- Comments:10 (8 by maintainers)
@ches @woop I’m thinking of giving this a go sometime this week, hopefully with your blessing, haha. After looking around, I’m partial to flink’s implementation for multi-connector support, and propose something like this:
I’m thinking of keeping the KeyUtil separate from the API since it’s specific to KV stores and not really a general need, so I don’t think Feast should be opinionated about their implementation. Connectors are free to share a single KeyUtil across their Storage and Retriever implementations.
The actual methods defined in the interfaces require further thought, but just throwing this out for comments first.
LGTM!
True, though I think it’s fine to have interfaces in
api
that not every implementation needs, but on the other hand should be conservative about promoting things there until their abstractions have some exercise.Hypothetical KeyUtil might not end up being a good case though anyway, among other things I had thought they’d ideally be static and there’s really no way to have static interfaces in Java, so maybe that’s a clue it’s against the grain… (not to mention “util” usually makes me feel a little ashamed of myself 😅).
Definitely intended this to be true though, as the original spirit of this issue.