Should `kedro-datasets` be a namespace package?

@noklam @MerelTheisenQB Bit late to the party, but… did you all consider/discuss keeping the import path as is? kedro-datasets would just expose the kedro.extras.datasets namespace package, and kedro would ideally exclude that namespace (even though it will probably still work, which is pretty neat from a compatibility perspective). This would mean that the change would largely be transparent to the user.

_Originally posted by @deepyaman in https://github.com/kedro-org/kedro-plugins/pull/38#discussion_r907477334_

[AntonyMilneQB]

@noklam as I understand it, the import path as it stands is kedro_datasets.datasets.pandas.CSVDataSet. Is there any reason for that rather than kedro_datasets.pandas.CSVDataSet?

[merelcht]

Notes from Technical Design session:

Discussion

Pros:

• Namespacing the package would make the experience of using kedro and datasets better for users, because they would only need to do import ... from kedro and not import .. from kedro and import ... from kedro_datasets
• We can make the move to the new repo non-breaking this way by keeping the import to kedro.extras.datasets

Cons:

• This could make our CI/CD setup more complicated
• If it should be possible to use kedro-datasets separately from Kedro, it might be a bad idea to package it under the “kedro” namespace. However, it’s only a super small use case and AbstractDataSet is and will remain a part of core Kedro so it’s a very unlikely use case.

Other concerns and questions raised:

• Can users still use the standalone datacatalog if we namespace kedro-datasets? Yes, because in order to use the datacatalog they need Kedro anyway, so this won’t change anything to that workflow.
• Should we rename kedro.extras.datasets to kedro.datasets and DataSet to Dataset? Yes, but not straight away. Both of these are breaking changes and should only be done when Kedro 0.19.0 will be released. Alternatively, we could alias the datasets which could be a fun exercise. It’s still a question of whether it’s worth the effort.
• It might be confusing for users that they can import kedro-datasets through import ... from kedro, but need to install a dataset with pip install kedro-datasets[xxx] and not pip install kedro[xxx]. We should add a redirect to make this possible
• How do we map the kedro version to the kedro-datasets version, e.g. when a user does pip install kedro[someDataSet] which version do we install? kedro-datasets should be a dependency inside kedro with a strict bound. So when a user is using kedro 0.18.x that will be mapped to kedro-datasets version x.x.x If a user then wants a newer dataset version they can install it manually (and accept the risks that this might break something that they need to fix themselves).

Conclusion

The team reached a consensus to namespace the kedro-datasets package under the kedro namespace.

Implementation

• To enable namespacing edit setup.py: https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#native-namespace-packages
• For now implement all changes in a non-breaking way
• Renaming from kedro.extras.datasets to kedro.datasets and from DataSet to Dataset will be done when Kedro 0.19.0 is going to be released.