Evaluating CLI command usage

As per discussions https://github.com/kedro-org/kedro/issues/1077 and https://github.com/kedro-org/kedro/discussions/844 we want to separate dev requirements into their own file outside requirements.txt. This means any requirements that are not required for kedro run or running a packaged kedro project (which only exposes kedro run). Remember in v0.18 there is no kedro install command.

• Change project template to move requirements from requirements.txt into the new dev requirements file. Also move docs requirements from setup.py extras_require to there
• Modify kedro CLI commands that currently install requirements (like kedro build-docs) to do the right thing
• Modify kedro CLI commands that need dev requirements but don’t currently install them (like kedro lint) do the right thing
• Change starters templates in same way
• Update documentation

Things to check and think about:

• Would all kedro lint, kedro build-docs etc. commands need to run a pip install to install dev requirements? Is this annoying and slow? Is Waylon’s idea of pre-commit a better alternative?
• Assuming we want a dev requirements file, what is the right name for it? Use whatever is convention these days
• Do commands like kedro pipeline, kedro catalog have any dependencies outside kedro, and if so where should those dependencies go (requirements or dev requirements)?
• What should kedro build-reqs do?

Also worth checking whether any kedro requirements should be moved to project requirements. e.g. pip-tools is there but after deprecation of kedro install can presumably be moved to a project requirement? Do we need jupyter_client in kedro requirements? etc.

[yetudada]

This stellar research @MerelTheisenQB 🎉 Thank you so much for this. I’ve read through your thoughts; thanks for the additions @MerelTheisenQB, @AntonyMilneQB and @NeroOkwa.

Here are my summary thoughts:

• We’re deleting all of these commands from our CLI, with particular emphasis on kedro activate-nbstripout and kedro jupyter convert
• We should create a discovery ticket to explore how we enable our users to document their workflows for sharing; there seemed to be something interesting around how they imagined kedro build-docs could help them but didn’t. However, this ticket is a low priority.
• I would prioritise making additions to our documentation. So it would look like new sections on how to use pytest in a Kedro workflow and maybe a ticket to document some of the linting tools.
• I like the plugin’s workflow; it’s slick. However, I would create a GitHub issue to track demand. Additionally, I would follow the consumption of the testing and linting documentation. My baseline assumption is that users wouldn’t use it because these features have had these convenience wrappers for a while, and they still didn’t use them.

So with these changes, we still maintain some of our value proposition. But it’s clear our users like other parts of Kedro and not necessarily our support of tooling that helps them create better software.

[merelcht]

@AntonyMilneQB Thanks for sharing and elaborating on your thoughts!

I like the idea of moving the commands we want to keep to a separate plugin if we’re keeping a reasonable amount. If we decide we’re removing most of them anyway, I’m not sure it’s even worth creating this plugin. From your earlier comment it sounded like you’d only really want to keep kedro lint, so then a new plugin might be overkill. But 100% agree to create a plugin if we decide to keep several of the commands.