ColumnTransformer should be able to use a function to select columns

The new ColumnTransformer allows the user to specify column names or indices. I think it should be possible to specify a set of columns as a function. Indeed, towards #10603, we should probably have an inbuilt function that distinguishes between categorical and numeric pd.Series dtypes.

In order to support the remainder functionality, I believe the function should have signature: (X,) -> column indices where column indices is any of the supported column specification formats.

Sound good @amueller, @jorisvandenbossche?

[jnothman]

I think you should be very careful to avoid any operator overloading. Scikit-learn tends to be quite conservative in this area, and tries to avoid “magic” in its APIs. Don’t worry about defining categorical_features, etc. Worry about ensuring that there’s an interface for users to write custom selectors that don’t require them to know the columns by name or index beforehand, which the current interface does. Later we can worry about making a library of such selectors and how that would look.​

[jnothman]

Thanks @partmor, I look forward to your pr. Please start with supporting functoons, testing it, and demonstrating with a realistic example. Then we can consider simplifying the API common use cases. @jorisvandenbossche’s proposal does not, for instance, account for disparity in the definition of “categorical feature”