test: use pytest fixtures to initialize and populate the backends

I propose to port the click commands from datamgr.py and impalamgr to pytest fixtures (with scope="session" to execute once per pytest invocation).

The python testing suite is dependent on the files under the ci/ directory. Examples:

PYTEST_BACKEND=pandas pytest ibis/backends or pytest --enable-pandas ibis/backends should:

1. Check that the testing data exists under the target directory
2. If not, download it otherwise skip
3. Execute the tests with pandas marker enabled

PYTEST_BACKEND=clickhouse pytest ibis/backends or pytest --enable-clickhouse ibis/backends should:

1. Check that the testing data exists under the target directory
2. If not, download it otherwise skip
3. Check that the required tables exist in clickhouse
4. If not, populate the tables otherwise skip
5. Execute the tests with clickhouse marker enabled

We can add pytest flags to --force-download/ --skip-download and --force-populate/--skip-populate the requested backends.

Even though the first run would take more time since it would implicitly download the data (the second run would be instant), new contributors could use the a single pytest command to execute the testing suite. We can add additional logging / progress bar to inform the developer about the data[base] initialization.

[anjakefala]

/take

[cpcloud]

Heck yeah!

Welcome @anjakefala.

Right, download is very simple and does a git clone of ibis-project/ibis-testing-data. Feel free to take a look at it, but it’s pretty bare bones.

The bulk of the work is going to be in figuring out how to move all the functionality in the load subcommands like

python ci/datamgpr.py load postgres
python ci/datamgpr.py load duckdb

into a pytest.fixture (that was my initial thought, not set in stone in any way).