Poetry Support for Kedro Projects

Description

The way kedro initiate a new project and create the folder structure does not goes well with Poetry . Usually I would create a Poetry environment before doing anything and then install all my required pacakges one by one. After I create a Poetry environment and added the kedro package the pyproject toml looks as followes:

poetry new --src KedroPoetry

[tool.poetry]
name = "KedroPoetry"
version = "0.1.0"
description = ""
authors = ["Your Name <[e-mail]>"]

[tool.poetry.dependencies]
python = "^3.8"
kedro = {version = "~0.18.2", python = ">=3.8,<3.11"}

[tool.poetry.dev-dependencies]
pytest = "^7.1"

... (more lines)

Lets run the demo pytest to see if everything works.

poetry run pytest .

This goes well:

KedroPoetry|⇒ poetry run pytest .
platform darwin -- Python 3.8.12, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/ALAMSHC/PythonProjects/KedroPoetry
collected 1 item                                                                                                       

tests/test_kedropoetry.py .                                                                                      [100%]

Now its time to add a Kedro Project: kedro new

The command completely ignored the current pyproject.toml file. and as there is a src file it did not add the project in the src folder instead create a directory on the root outside of src. Now there is no kedro setup section in pyproject.toml so kedro cli will complain for broken setup.

Context

As Poetry provide one of the modern approach for packaging Python projects it will be good to have direct support for Poetry like project structure for Kedro or at-least a hackable way out will also work.

Possible Implementation

There could be a new flag in cli to initiate project with Kedro when there is already a pyproject.toml file and a project setup for Poetry.

[AntonyMilneQB]

I’m reopening this because I think it’s a very good topic and I’d be interested in hearing from other users about it 🙂 It’s been mentioned several times before by differently people but we’ve never had thoughts collected together in one place, so let’s start doing that here! In the past we’ve also wondered whether we should switch to using poetry. Currently we support a pip-compile workflow but we’re planning to remove that in favour of just a plain requirements.txt file. Given https://github.com/kedro-org/kedro/issues/1724, it might be time to re-assess what system we use exactly.

Some previous related issues (there’s probably others too): https://github.com/kedro-org/kedro/issues/398 https://github.com/kedro-org/kedro/issues/391

From these and other conversations I know the following users have independently shown interest in kedro + poetry. There’s also been interest within QB, though I’m not sure exactly who. So I definitely think there’s some significant interest in this. @datajoely do you know anyone else here? @fkromer @danhje @Kastakin @Larkinnjm1 @shaunc

[arnaldog12]

TBH, I like the idea of adding support for Poetry in Kedro projects. I think the main advantages of Poetry are:

• it’s a widely used package manager (more than pip-compile at least)
• eliminate the need for setup.py
• it provides a better way to manage project/dev requirements.