Simpler UX for non-package (data science) projects

• I have searched the issues of this repo and believe that this is not a duplicate. Kind of a duplicate of #1132 and #1148, but would like to more discuss the UI than the technical limitations.
• I have searched the documentation and believe that my question is not covered.

Feature Request

Since pipenv is dead (pypa/pipenv#4058), I’ve been using Poetry. It’s a great tool, and better than pipenv—but the UX is oriented more specifically towards creating new Python packages, not managing multiple Python projects in general. When working on data science projects, which are not and never will become Python packages, I’d prefer a less-opinionated interface.

In Pipenv, I like that creating a new environment is minimalist and simple:

$ mkdir new-project
$ pipenv install jupyterlab ipython numpy

With poetry, I have to read the docs more carefully to figure out poetry init vs poetry new, then run poetry init and hit Enter a bunch of times to bypass the pyproject.toml metadata that I don’t particularly care about (or is irrelevant, such as the version), type out my packages one-by-one, forget that jupyterlab isn’t jupyter-lab, screw it up, ctrl-C, break my shell (#2067), open a new Terminal because I don’t know about reset, and do it all again.

poetry init -n is an okay workaround right now, but I usually forget the name of the flag and have to look it up, which is annoying.

If poetry is becoming the primary tool for Python environment and dependency management, it’s worth thinking about the UX for data science, since that’s such a common reason to use Python (and need to manage a bunch of per-project environments) these days.

A few ideas:

As discussed in #1132, pyproject.toml not requiring a version or name is probably a prerequisite to all of these.

• poetry add creates a pyproject.toml in the current directory, if not already within a poetry project (similar to pipenv semantics)
• poetry init is non-interactive by default
• poetry new only creates a directory and pyproject.toml by default, poetry new --package would add the boilerplate scaffolding of README, tests, package directory, etc.

This doesn’t address making the simple case simple and easy to document to beginners, though.

Perhaps a larger change could be moving the current semantics under a poetry package command group (poetry package new, poetry package init), and having poetry init/new be for creating non-package “data science” projects.

My (unsubstantiated) guess is that people create new Python projects way more frequently than they create new Python packages. How many times a week do you make a new package? (For me, very rarely.) How many times a week to you make an isolated data science project to try things out? (For me, nearly every day.) To me, this justifies making non-package projects the primary interface, and package-based projects more of a special case. But I recognize that that changes the current semantics and goals of Poetry a fair amount.

Also, happy to move this to a comment on #1132 if you do think it’s a duplicate issue.

[epogrebnyak]

@danmur - you are right, frsej init-install sequence works well. I had Cannot install package error, but can’t replicate it now, must be some editing or corruption of pyproject.toml. Can confirm fresh init and install without package work fine, just as you say, so no need for --no-package flag.

[0az]

As a Data Science student who’s been using Poetry for projects (without packaging) and packaged projects [0], here’s my 2¢.

Given that it’s late, I’ll skip most of what I’d like to say and focus on one point: documentation.

Poetry has a lot of space to improve, documentation-wise. This isn’t just online – I stumbled upon this issue while looking for how to use “package data” – but also in the executable itself. Specifically, the errors are confusing.

As an example, when trying to find information on poetry run, I opened Kitty, typed in poetry run --help, and ended up with this:

~> poetry run --help

[RuntimeError]
Poetry could not find a pyproject.toml file in /Users/az or its parents

Now, this is a case of “whoops, wrong command”, but it also illustrates another important point: Poetry currently doesn’t fail educationally.

Contrast cargo, which is often cited as a gold standard:

The same command, but using Cargo

~> cargo run --help
cargo-run 
Run the main binary of the local package (src/main.rs)

USAGE:
    cargo run [OPTIONS] [--] [args]...
[...]
~> cargo help run
cargo-run 
Run the main binary of the local package (src/main.rs)
[...]

This is but one unrepresentative example, but it does show that there is space for Python’s ecosystem to improve.

---

* `poetry add` creates a `pyproject.toml` in the current directory, if not already within a poetry project (similar to pipenv semantics)

I’m actually against this, because I’ve accidentally run yarn add $package / pipenv install $package in the wrong place enough times, and cleaned up the aftermath enough times, that I’d prefer a helpful error message instead.

* `poetry init` is non-interactive by default

I don’t hold any particular opinions here, other than bouncing off the interactive deps tool [1], but cargo init is non-interactive.

* `poetry new` only creates a directory and `pyproject.toml` by default, `poetry new --package` would add the boilerplate scaffolding of README, tests, package directory, etc.

I could get behind this. Though, at this point, why not have a poetry create analogous to create-react-app?

---

Could you elaborate on why you think that packaging a project is more common? I think it is way off assuming people equate a project == package. I guess that an abundance of more users would know what pip the package installer is and use it, than packaging python projects with e.g. setuptools, wheels.

Another point is this: regardless of if packaging is more common or not, should Poetry (and the broader ecosystem) take an opinionated stance of “Yes, all nontrivial Python code should be packaged”?

Rust, as a compiled language, takes that stance. Python probably shouldn’t make that declaration for all code – one-off scripts are an easy counterexample. But should all nontrivial projects see packaging? If so, then packaging needs to be as frictionless as possible.

---

[0] This is for a variety of reasons, including

• It’s just easier to use poetry install && poetry run python -m $package.
• poetry shell is easier to remember than fish -c ...
• venv has activate/deactivate, which provide a much worse UX because of the lack of separation.

[1] The interface didn’t give a good way to cancel, I think.

Testing the interface:

Search for package to add (or leave blank to continue): requests
Found 20 packages matching requests

Enter package # to add, or the complete package name if it is not listed: 
 [0] requests
 [1] requests3
 [2] requests2
 [3] lambda_requests
 [4] requests-transition
 [5] micropython-requests
 [6] tornado-requests
 [7] requests_handler
 [8] requests-onshape
 [9] requests-lb
 > ^M
'NoneType' object has no attribute 'replace'

There’s also an inconsistency between poetry new and poetry init here: only the latter uses this prompt interface. I’d argue that the expected behavior is the other way around, if not both.

---

I’ll probably have some new input later, once this quarter finishes and I have time to actually organize my thoughts.