Latest Only option for Historical Retrieval

Is your feature request related to a problem? Please describe.

In many batch workflows, it is worthwhile to retrieve the latest features by entity only. This is useful from the purposes of both production and backtesting purposes.

E.g. if I have an hourly/daily batch which goes through our whole customer base to find fraudulent customers, we wouldn’t really use the online store for this.

Describe the solution you’d like

Allow users to specify an entity set extracted from a feature view should have an option to be deduplicated by latest. Depends on #1611

my_daily_batch_scoring_df = store.get_latest_features(
    entity_df = "my_df", 
    feature_refs = [...],
)

Additional context Linked issue #1611

[MattDelac]

I still believe that this is an important feature for batch prediction pipelines. In that case you need the latest values from the offline store.

You also need to keep this idea of an “entity_df” that we don’t have with the pull_latest_from_table_or_query() method

[woop]

@MattDelac is this API moving closer to what you are using internally?

Not really

But we have the same need for batch predictions where we want to predict the latest values of the features in batch. Therefore we could bypass the historical retrieval logic and have a SQL template that is much more efficient.

In terms of API i would rather have another API eg: store.get_latest_features() rather than a boolean parameter. And as I said, store.get_latest_features() could be a very efficient SQL query

Hope that makes sense

store.get_latest_features() could be a shared method that is also used for materialization into the online store. Seems like a good idea to me.