Implement pipeline autodiscovery

Following https://github.com/kedro-org/kedro/issues/1284 and a discussion with Ivan, here’s what we’re going to do:

1. Define a function find_pipelines

In kedro.framework.project.__init__.py, something like this:

def find_pipelines():
    registered_pipelines = {}
    pipeline in Path(f"{PACKAGE_NAME}.pipelines").iterdir():
        importlib.import_module(pipeline)
        if hasattr(pipeline, "create_pipeline"):
            registered_pipelines[pipeline] = pipeline.create_pipeline()
    registered_pipelines["__default__"] = pipeline([])
    return registered_pipelines

Name is TBD (note I changed from previously-suggested autoregister because it doesn’t actually register any more, just finds them). It’s also like setup.py’s find_packages here.

2. Call this function from project template and all starters

In pipeline_registry.py:

from kedro.framework.project import find_pipelines

def register_pipelines() -> Dict[str, Pipeline]:
    """Register the project's pipelines.

    MAKE THIS DOCSTRING MORE INFORMATIVE

    Returns:
        A mapping from a pipeline name to a ``Pipeline`` object.

    """
    pipelines = find_pipelines()
    pipelines["__default__"] = sum(pipelines.values())
    return pipelines

3. Add documentation, tests, etc.

[deepyaman]

Personally I would do a whole new file docs/source/nodes_and_pipelines/pipeline_registry.md, but up to you really. I think the modular pipeline documentation needs some modifying as it stands, so I wouldn’t be surprised if these things move around a bit anyway. So long as it’s linked from and consistent with other relevant sections of the docs (e.g. linked from the instructions on “How to run a pipeline”; update spaceflights tutorial so there’s no need to edit pipeline_registry.py any more) I don’t mind much where it goes.

Great! I pinged @MerelTheisenQB on Friday, since I thought you were on holiday, and she was aligned in terms of creating a new page for the pipeline registry.

Also I know this is a fairly big documentation update, so do feel free to do bit by bit or in a separate PR if easier for you 🙂 🙏

I think the functionality is good to go on my current PR; just need to add the documentation. I’ll plan to:

1. Write basic pipeline registry documentation, and merge that PR.
2. Update it with a section on autodiscovery, and integrate into my open PR.

Alternatively, I’d also be happy to merge the current code (after review) and add the 1-2 documentation PRs on top of that.

[deepyaman]

@deepyaman Either approach works for me - up to you. Just mark the PR(s) as ready for review whenever you want me to have a look through and I’ll do so.

Great! In that case, I’ve marked my current PR as ready for review. 😃