[KED-2336] kedro clean

Description

A command that deletes all intermediate data sets created during a pipeline run, not unlike make clean.

Context

In order to debug pipelines it is often helpful to add intermediate outputs to the data catalog and write them to disk. However, “leftover” intermediate data sets can be a source of bugs when developing pipelines. Suppose you rewired the pipelines and made an error, but it does not show up as pipeline failure because you forgot to delete an old intermediate data set.

Possible Implementation

One could perhaps mark data sets as “intermediate” in the sense of fully managed by pipelines - never modified or replaced except by kedro run. Then kedro clean could automatically delete those.

Possible Alternatives

Manually deleting these datasets from the data folder.

[DmitriiDeriabinQB]

example code to show how this can be done

Sure, first run kedro ipython from your project root, then

from pathlib import Path
from shutil import rmtree

pipeline = context.pipelines["<pipeline_name>"]
intermediate = pipeline.all_outputs() & pipeline.all_inputs() & catalog._data_sets.keys()

for dataset_name in intermediate:
    filepath = Path(catalog._data_sets[dataset_name]._filepath)
    if filepath.is_file():
        filepath.unlink()
    elif filepath.is_dir():
        rmtree(filepath)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.