Quick start docker-compose demo doesn't work

Apache Iceberg version

0.14.1 (latest release)

Query engine

Spark

Please describe the bug 🐞

Here is the docker-compose file that I tried to use:

services:
  postgres:
    image: postgres:13.4-bullseye
    container_name: iceberg_postgres
    environment:
      - POSTGRES_USER=admin
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=demo_catalog
    volumes:
      - ./postgres/data:/var/lib/postgresql/data

  spark-iceberg:
    image: tabulario/spark-iceberg
    depends_on:
      - postgres
    container_name: spark-iceberg
    environment:
      - SPARK_HOME=/opt/spark
      - PYSPARK_PYTON=/usr/bin/python3.9
      - PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/spark/bin
    volumes:
      - ./warehouse:/home/iceberg/warehouse
      - ./notebooks:/home/iceberg/notebooks/notebooks
    ports:
      - 8888:8888
      - 8080:8080
      - 18080:18080

There are 2 issues I came across:

1. localhost:8080 and localhost:18080 are not accessible from my web browser (Firefox and Safari)
2. Trying to run the example notebook Iceberg - Getting started, I can run the “spark” cell (the first cell), but the next one throws an error:

Py4JJavaError                             Traceback (most recent call last)
Cell In [4], line 1
----> 1 get_ipython().run_cell_magic('sql', '', '\nDROP TABLE IF EXISTS nyc.taxis\n')

File /usr/local/lib/python3.9/site-packages/IPython/core/interactiveshell.py:2362, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
   2360 with self.builtin_trap:
   2361     args = (magic_arg_s, cell)
-> 2362     result = fn(*args, **kwargs)
   2363 return result

File ~/.ipython/profile_default/startup/00-prettytables.py:61, in sql(line, cell)
     59         return _to_table(df, num_rows=args.limit)
     60 else:
---> 61     return _to_table(spark.sql(cell))

File /opt/spark/python/pyspark/sql/session.py:1034, in SparkSession.sql(self, sqlQuery, **kwargs)
   1032     sqlQuery = formatter.format(sqlQuery, **kwargs)
   1033 try:
-> 1034     return DataFrame(self._jsparkSession.sql(sqlQuery), self)
   1035 finally:
   1036     if len(kwargs) > 0:

File /opt/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py:1321, in JavaMember.__call__(self, *args)
   1315 command = proto.CALL_COMMAND_NAME +\
   1316     self.command_header +\
   1317     args_command +\
   1318     proto.END_COMMAND_PART
   1320 answer = self.gateway_client.send_command(command)
-> 1321 return_value = get_return_value(
   1322     answer, self.gateway_client, self.target_id, self.name)
   1324 for temp_arg in temp_args:
   1325     temp_arg._detach()

File /opt/spark/python/pyspark/sql/utils.py:190, in capture_sql_exception.<locals>.deco(*a, **kw)
    188 def deco(*a: Any, **kw: Any) -> Any:
    189     try:
--> 190         return f(*a, **kw)
    191     except Py4JJavaError as e:
    192         converted = convert_exception(e.java_exception)

File /opt/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
    324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325 if answer[1] == REFERENCE_TYPE:
--> 326     raise Py4JJavaError(
    327         "An error occurred while calling {0}{1}{2}.\n".
    328         format(target_id, ".", name), value)
    329 else:
    330     raise Py4JError(
    331         "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
    332         format(target_id, ".", name, value))

Py4JJavaError: An error occurred while calling o36.sql.
: org.apache.iceberg.exceptions.RESTException: Error occurred while processing GET request

The errors are probably related.

Sorry if wrongly reported as bug.

[samredai]

The latest image has been recently updated to use the REST catalog. This requires some slight changes to the docker compose which we’ve updated here in the repo.

If you want to get back to the old setup that uses a JDBCCatalog, you can simply add a tag to the image name in your current docker compose setup: tabulario/spark-iceberg:3.3.0_0.14.1.

[nastra]

Closing this as it’s fixed by https://github.com/apache/iceberg-docs/pull/178