Athena query on a specific "Catalog id"

It is not possible to specify the catalog id in the read_sql_query function. I would like to be able to request a specific catalog id on Athena.

boto3 takes into account the catalog id parameter in the “QueryExecutionContext” argument since at least version 1.14.58 (here).

A solution could be to add this in the athena._utils._start_query_execution here, something like:

    # database
    if database is not None:
        args["QueryExecutionContext"] = {"Database": database, "Catalog": catalog_name}

and have the catalog id parameter available from read_sql_query.

[igorborgest]

Actually this was a pretty useful comment. We MUST update our documentation and also improve our exception message to clarify this new ctas_approach=True limitation.

Thank you @Xiangyu-C !

[igorborgest]

Wrangler has two ways to run queries on Athena and fetch the result as a DataFrame:

ctas_approach=True (Default)

Wraps the query with a CTAS and then reads the table data as parquet directly from s3.

PROS:

• Faster for mid and big result sizes.
• Can handle some levels of nested types.

CONS:

• Requires create/delete table permissions on Glue.
• Does not support timestamp with time zone
• Does not support columns with repeated names.
• Does not support columns with undefined data types.
• A temporary table will be created and then deleted immediately.

ctas_approach=False

Does a regular query on Athena and parse the regular CSV result on s3.

PROS:

• Faster for small result sizes (less latency).
• Does not require create/delete table permissions on Glue
• Supports timestamp with time zone.

CONS:

• Slower (But stills faster than other libraries that uses the regular Athena API)
• Does not handle nested types at all.

Reference