API for preprocessing only ?

Is your feature request related to a problem? Please describe. Problem: getting access to preprocessed data without training

Describe the use case I would like to use ludwig but just to preprocess my data using the yaml file and skipping the training part.

Describe the solution you’d like

• CLI approach: ludwig preprocess --data_csv path/to/file.csv --dataset_definition_definition path/to/def.yml
• API approach:

from ludwig import LudwigPreprocessing

dataset_definition = {...}
preprocessor = LudwigPreprocessing(dataset_definition)
preprocessed_dataframe = preprocessor.preprocess(dataset_dataframe)

Describe alternatives you’ve considered An alternative right now would be to use it as suggested, killing the process during the training step then using the generated preprocessed data.

[w4nderlust]

I see, good point. I will consider adding it in the next releases, as the functions are all already there, it’s a matter of finding a nice way to expose them.

[w4nderlust]

Starting from v0.2, the preprocessing is available for user to play around with, as now one can import whatever they like from data.preprocessing., although the only fully documented part of the code remains the API, so you can play with preprocessing, but it is not fully documented yet, in particular because we are working towards many changes in the preprocessing pipeline for v0.3.