feat: create a table of literal values

I’d like to be able to create a table from literal values. This can be handy for small utility tables that I might like to join to. It’s also handy for test scripts to avoid having to create physical tables when not needed.

One possible way to implement this would be with the UNNEST operator (https://github.com/ibis-project/ibis/issues/1146), though there may be other ways of accomplishing this.

Expected Ibis usage

With UNNEST support, I’d expect it to be used in combination with an array literal or struct literals.

table = ibis.unnest(
  ibis.array(
    [
      ibis.struct(dict(col1='hello', col2=1), type='struct<col1: string, col2: integer>'),
      ibis.struct(dict(col1='world', col2=2), type='struct<col1: string, col2: integer>'),
      ibis.struct(dict(col1='!', col2=3), type='struct<col1: string, col2: integer>'),
    ]
  )
)

Equivalent BigQuery SQL

Using the UNNEST operator (reference) on an array literal of struct literals results in an anonymous table-valued expression.

SELECT col1, col2
FROM UNNEST(
  [
    STRUCT<col1 STRING, col2 INT64>("hello", 1),
    STRUCT<col1 STRING, col2 INT64>("world", 2),
    STRUCT<col1 STRING, col2 INT64>("!", 3)
  ]
)
-- Optionally, include a table name via the alias feature
AS my_temp_table

Row	col1	col2
1	hello	1
2	world	2
3	!	3

Alternative Ibis expression

Alternatively, maybe ibis.table could take an optional values parameter? In the BigQuery dialect, I’d still expect this to compile to an UNNEST operator.

table = ibis.table(
  values=[
      ibis.struct(dict(col1='hello', col2=1), type='struct<col1: string, col2: integer>'),
      ibis.struct(dict(col1='world', col2=2), type='struct<col1: string, col2: integer>'),
      ibis.struct(dict(col1='!', col2=3), type='struct<col1: string, col2: integer>'),
  ],
)

See also

• feat: support for UNNEST (array flatten operation) #1146
• Creating a table from literal values in Ibis (StackOverflow)

[cpcloud]

However, I would be in favor of a top-level ibis.rows() API that produces a Rows node that would be compiled to VALUES in SQL. The pandas backend can call pd.DataFrame on the input during execution.

[cpcloud]

I suspect that table-valued-functions will require some breaking changes to the SQL compilers to support custom TableNode instances. Right now, our handling of tables is essentially hard-coded.