Local Docker ETL with local inputs/outputs

Given a Docker container with our CI environment (#1605):

• Add local volumes to the container to point at PUDL_IN and PUDL_OUT. Maybe with docker compose?
• Get the container to run the equivalent of tox -e ci while reading & writing data on the local volume.
• Capture the logs and other outputs from the ETL for later review.
• Run equivalent of tox -e nuke: all CI, full ETL, and data validation, reading & writing data on the local volume.

[zaneselvans]

A good overview of docker logging. They outline several available logging strategies. It seems like using a docker logging driver is probably the right option for us, and there is a dedicated Google Cloud Logging driver

In theory the stdout and stderr from the container are sent to the logs, but maybe this only works while the container is running? I don’t seem to be able to get anything out of the logs locally by doing e.g.

docker logs pudl_etl

either when the tests are running or after they’ve completed.

I wonder if this might be affected by the fact that tox/pytest are sitting between the process and the logging? Maybe I should try running pudl_etl directly (and also have it generate some real outputs).

[zaneselvans]

Okay, the problem with not being able to write to PUDL_OUT was that there was no pre-populated sqlite or parquet output folders in there. Because the pudl_setup script is written during the build of the container, inside the container. Duh.