bug: RecursionError raised when attempting to create a duckdb table from a pandas-backed exp

Reading the documentation on Creating and Inserting Data, I was under the impression that it would be possible to create a new table from an arbitrary ibis expression, even if it did not stem from the same connection/backend.

Hence I wanted to use pandas to create an in-memory table and then store that into a duckdb database:

import ibis
import pandas as pd

df = pd.DataFrame(
    {

        "g": ["a", "a", "a", "a", "a"],
        "x": [0, 1, 2, 3, 4],
        "y": [3, 2, 0, 1, 1],
    }
)
t_pandas = ibis.pandas.connect({"t": df}).table("t")
duckdb_conn = ibis.duckdb.connect()
duckdb_conn.create_table("t", t_pandas)

However this results in RecursionError: maximum recursion depth exceeded:

Traceback (most recent call last):
  File "/Users/ogrisel/tmp/ibis_pandas_window.py", line 16, in <module>
    duckdb_conn.create_table("t", t_pandas)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/__init__.py", line 215, in create_table
    method = self._get_insert_method(expr)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/__init__.py", line 219, in _get_insert_method
    compiled = self.compile(expr)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/__init__.py", line 344, in compile
    return self.compiler.to_ast_ensure_limit(expr, limit, params=params).compile()
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/base.py", line 39, in compile
    compiled_queries = [q.compile() for q in self.queries]
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/base.py", line 39, in <listcomp>
    compiled_queries = [q.compile() for q in self.queries]
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/query_builder.py", line 176, in compile
    frag = self._compile_table_set()
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/query_builder.py", line 203, in _compile_table_set
    result = helper.get_result()
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/query_builder.py", line 40, in get_result
    self.join_tables.append(self._format_table(op))
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/query_builder.py", line 134, in _format_table
    result = ctx.get_compiled_expr(op)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/translator.py", line 73, in get_compiled_expr
    result = self._compile_subquery(node)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/translator.py", line 34, in _compile_subquery
    return self._to_sql(op, sub_ctx)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/translator.py", line 37, in _to_sql
    return self.compiler.to_sql(expr, ctx)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/query_builder.py", line 408, in to_sql
    return query.compile()
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/query_builder.py", line 176, in compile
    frag = self._compile_table_set()
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/query_builder.py", line 203, in _compile_table_set
    result = helper.get_result()
[...]
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/query_builder.py", line 203, in _compile_table_set
    result = helper.get_result()
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/query_builder.py", line 40, in get_result
    self.join_tables.append(self._format_table(op))
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/query_builder.py", line 134, in _format_table
    result = ctx.get_compiled_expr(op)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/translator.py", line 73, in get_compiled_expr
    result = self._compile_subquery(node)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/translator.py", line 34, in _compile_subquery
    return self._to_sql(op, sub_ctx)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/translator.py", line 37, in _to_sql
    return self.compiler.to_sql(expr, ctx)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/alchemy/query_builder.py", line 405, in to_sql
    query = cls.to_ast(expr, context).queries[0]
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/query_builder.py", line 527, in to_ast
    query = cls.select_builder_class().to_select(
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/select_builder.py", line 144, in to_select
    select_query = self._build_result_query()
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/select_builder.py", line 195, in _build_result_query
    self._populate_context()
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/select_builder.py", line 218, in _populate_context
    self._make_table_aliases(self.table_set)
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/select_builder.py", line 244, in _make_table_aliases
    elif not ctx.is_extracted(node):
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/translator.py", line 130, in is_extracted
    return node in self.top_context.extracted_subexprs
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/translator.py", line 59, in top_context
    return self.parent.top_context
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/translator.py", line 59, in top_context
    return self.parent.top_context
  File "/Users/ogrisel/code/ibis/ibis/backends/base/sql/compiler/translator.py", line 59, in top_context
    return self.parent.top_context
  [Previous line repeated 105 more times]
RecursionError: maximum recursion depth exceeded

Ideally I would have liked ibis to import the in-memory data to the duckdb table or alternative explicitly state that creating a table from an expr backed by another backend is not possible instead of raising a low-level RecursionError.

UPDATE: I also tried with a sqlite connection and I get the same error when passing the pandas df directly while it works when wrapping it into an ibis.memtable.

[jcrist]

Thanks for opening this! I agree that both the docs and error message here could be improved.

Replying to myself, I just discovered ibis.memtable which solves my original problem:

While this works and is one way to do it, you can also use con.register to register an external table with duckdb from multiple different common sources (pandas, pyarrow, parquet, …). Since this only registers an external table, it’s zero-copy (when possible), and won’t result in a new table being created in any persistent duckdb database.

import ibis
import pandas as pd

df = pd.DataFrame(
    {

        "g": ["a", "a", "a", "a", "a"],
        "x": [0, 1, 2, 3, 4],
        "y": [3, 2, 0, 1, 1],
    }
)
duckdb_conn = ibis.duckdb.connect()
t = duckdb_conn.register(df)  # you optionally also pass a table name, otherwise one is generated for you
t.execute()

[ogrisel]

Related UX issue:

• #4683