Python Instructions currently do not work for testing

Apache Iceberg version

main (development)

Query engine

Other

Please describe the bug 🐞

The instructions listed in the README.md under testing

poetry install -E pyarrow
poetry run pytest

Result in an error

ImportError while loading conftest '/Users/russellspitzer/Temp/pypypy/iceberg/python/tests/conftest.py'.
tests/conftest.py:40: in <module>
    import aiobotocore.awsrequest
E   ModuleNotFoundError: No module named 'aiobotocore'

This is because pytest triggers all test suites and requires all optional dependencies

 poetry install --all-extras

Fixes this issue but still fails some tests (tests/io/test_fsspec.py) because the S3 Mock doesn’t seem to be setup properly

FAILED tests/io/test_fsspec.py::test_fsspec_write_and_read_file - botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:9000/warehouse/18112126-9c9a-40e3-a08a-0fb199dd4f77"
FAILED tests/io/test_fsspec.py::test_fsspec_getting_length_of_file - botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:9000/warehouse/566507b0-e364-464f-9779-17a63329953a"
FAILED tests/io/test_fsspec.py::test_fsspec_file_tell - botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:9000/warehouse/fdc0fd10-132b-4d00-8d04-332dfe20f516"
FAILED tests/io/test_fsspec.py::test_fsspec_read_specified_bytes_for_file - botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:9000/warehouse/f3fc2540-1f7e-43ac-b27a-7a27a3a12216"
FAILED tests/io/test_fsspec.py::test_fsspec_raise_on_opening_file_not_found - botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:9000/warehouse/2ce8b101-bbbc-4242-8cde-6e4e3e83516a"
FAILED tests/io/test_fsspec.py::test_checking_if_a_file_exists - botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:9000/warehouse/does-not-exist.txt"
FAILED tests/io/test_fsspec.py::test_closing_a_file - botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:9000/warehouse/91077182-d448-4f1f-b310-2a60d6799617"
FAILED tests/io/test_fsspec.py::test_writing_avro_file - botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:9000/warehouse/c7aeded1-9bc6-4429-923a-d164be4f35e8"

[Fokko]

It seems that it only works on Python >= 3.9.8. There’s some bugfix on Protocols in the release notes that might ring a bell with you?

Great catch! I wasn’t aware of this!

The failed tests on Docker are because there is a single user in Docker. In the test, we create a file and then change the permissions in such a way that we don’t have read permissions. However, if you run it in Docker, you’re root, so the test doesn’t work. We can either introduce another user in Docker, or mock the call to the filesystem.

[rubenvdg]

Both Docker containers fail btw with:

#11 21.80 FAILED tests/io/test_pyarrow.py::test_raise_on_opening_a_local_file_no_permission
#11 21.80 FAILED tests/io/test_pyarrow.py::test_raise_on_checking_if_local_file_exists_no_permission
#11 21.80 FAILED tests/io/test_pyarrow.py::test_raise_on_creating_a_local_file_no_permission
#11 21.80 FAILED tests/io/test_pyarrow.py::test_raise_on_delete_file_with_no_permission

with a.o.

FileNotFoundError: [Errno 2] Failed to open local file '/tmp/tmpfj40zq4h/foo.txt'. Detail: [errno 2] No such file or directory

Is this expected?