Ephemeral model bug: nested CTE cannot reference previous CTE

Given a query like:

with my_first_cte as (

  with my_sub_cte as (
  
    select 1 as fun
    
  )
  
  select * from my_sub_cte
  
),

my_second_cte as (

  with my_next_sub_cte as (
  
    select * from my_first_cte
    
  )
  
  select * from my_next_sub_cte
  
)

select * from my_second_cte

Returns the following error:

ERROR processing query/statement. Error Code: 0, SQL state: org.apache.spark.sql.AnalysisException: Table or view not found: my_first_cte; line 17 pos 18

This prevents us from being able to have multiple ephemeral models in a dependency line, since the second CTE (ephemeral model) includes sub CTEs that reference the first CTE (ephemeral model).

[jtcohen6]

This bug is actually resolved in Spark 3.x, which offers greater support for nested CTEs/subqueries. While it’s still an issue in Spark 2.x, I’m going to close this, as we don’t plan to make any code changes.

[drewbanin]

@aaronsteers fyi, check out https://github.com/fishtown-analytics/dbt/issues/1248 and https://github.com/fishtown-analytics/dbt/pull/1283

I think sql parsing + inlining CTEs is a cool idea - it would definitely give us the best of both worlds:

1. clean source SQL using CTEs
2. valid compiled SQL that actually runs on spark 😃

But, dbt doesn’t do very much SQL parsing at all, and we generally tend to bet on databases better supporting the SQL standard over time rather than trying to work around their current limitations.

I don’t think we have any immediate plans to do this type of SQL parsing/rewriting, but it could definitely be in scope for the future. For the moment, the shortest path to supporting ephemeral models is probably going to be a subquery-based implementation