Add example YAML definitions to common dataset Python docstrings

Description

Users often ask for an easy way to look up the relevant YAML dataset configuration for use in the DataCatalog would look like. We include a series of examples in this section of the documentation, but it’s not especially easy to (1) link to as these aren’t under headings (2) find via search engine for the same reason.

This would be useful for all datasets - but the highest priority are those which drive most traffic to our documentation website:

• kedro.extras.datasets.pandas.CSVDataSet
• kedro.extras.datasets.spark.SparkDataSet
• kedro.io.PartitionedDataSet
• kedro.extras.datasets.pandas.ParquetDataSet
• kedro.extras.datasets.pickle.PickleDataSet
• kedro.extras.datasets.pandas.ExcelDataSet
• kedro.extras.datasets.pandas.SQLQueryDataSet
• kedro.extras.datasets.pandas.GBQTableDataSet
• kedro.extras.datasets.spark.SparkHiveDataSet

Possible Implementation

def __init__(
        self,
        filepath: str,
        backend: str = "pickle",
        load_args: Dict[str, Any] = None,
        save_args: Dict[str, Any] = None,
        version: Version = None,
        credentials: Dict[str, Any] = None,
        fs_args: Dict[str, Any] = None,
    ) -> None:
        """Creates a new instance of ``PickleDataSet`` pointing to a concrete Pickle
        file on a specific filesystem. ``PickleDataSet`` supports four backends to
        serialize/deserialize objects: `pickle`, `joblib`, `dill`, and `compress_pickle`.

        Example YAML data catalog entry:
        >>> airplanes:
        >>>     type: pickle.PickleDataSet
        >>>     filepath: data/06_models/airplanes.pkl
        >>>     backend: pickle

        Args:
            filepath: Filepath in POSIX format to a Pickle file prefixed with a protocol like
                `s3://`. If prefix is not provided, `file` protocol (local filesystem) will be used.
                The prefix should be any protocol supported by ``fsspec``.
                Note: `http(s)` doesn't support versioning.
            backend: Backend to use, must be one of ['pickle', 'joblib', 'dill', 'compress_pickle'].
                Defaults to 'pickle'.
        """

To accomplish this, do the following:

• Follow the process in our Contribution Guide

[datajoely]

@avan-sh I’ve also hacked something together that might help you draft these, hopefully useful!:

import inspect
from kedro.extras.datasets import pandas
import anyconfig

def sample_yaml(ds):
    kind = ds.__name__
    kwargs = dict(inspect.signature(ds).parameters)
    hint = {k:' # '+inspect.formatannotation(v.annotation) for k,v in kwargs.items()}
    samples = {**{'type': kind} , **{k:f'...' for k,v in kwargs.items()}}
    structure =  {'example_'+kind.lower().replace('dataset', '') :samples}
    yaml_string = anyconfig.dumps(structure, 'yaml').split('\n')
    hint_lookup = {i:hint.get(x.strip().split(':')[0] ,'') for i,x in enumerate(yaml_string)}
    return '\n'.join([x+hint_lookup[i]  for i,x in enumerate(yaml_string)]).replace("'",'')

print(sample_yaml(pandas.CSVDataSet))

Which produces:

example_csv:
  type: CSVDataSet
  filepath: ... # str
  load_args: ... # Dict[str, Any]
  save_args: ... # Dict[str, Any]
  version: ... # kedro.io.core.Version
  credentials: ... # Dict[str, Any]
  fs_args: ... # Dict[str, Any]

[avan-sh]

Took a stab at this for CSV dataset, taking some inspiration from the old discussion. The new docs would look as below, hyperlinked to.

Would love to hear any suggestions for changes before I add it to other datasets.