Schema and statistics in Transform TFX Pipeline Component

The main entry point in Transform component (preprocessing_fn) should also provide computed Stats and Schema next to the inputs. In some scenarios users might want to benefit from the statistics e.g. to eliminate unnecessary features.

[JoshLipschultz]

I’d like to re-up this request—having easier access to the metadata and schema from within the Transform callback would be tremendously useful. Some feature transformations could depend on the inferred schema. Is there some other workaround?

[wsuchy]

@zoyahav Indeed. Currently in the TFX Transform Pipeline Component the processing_fn has a signature of

processing_fn(inputs:Dict[tf.Tensor]) -> Dict[tf.Tensor]

and I am proposing to change it as follows:

processing_fn(inputs:Dict[tf.Tensor], metadata:DatasetMetadata, stats:DatasetFeatureStatisticsList ) -> Dict[tf.Tensor]

or even better, as a higher order function / factory:

processing_fn_factory(metadata:DatasetMetadata, stats:DatasetFeatureStatisticsList ) 
-> Callable[Dict[tf.Tensor], Dict[tf.Tensor]]

So now my processing function could depend on the stats and schema. I know I could reach for it Stats and Schema manually, yet it requires me to talk to the SQLite database and keeping track of paths, etc.