custom data tests with CTEs fail

background

Per @alittlesliceoftom’s Slack thread.

Steps to reproduce:

1. write a data test that uses CTE clause,
2. dbt wraps the data test’s query into another CTE.
3. Synapse returns an error because it does not allow for nested CTEs.

Example

tests/example.sql

WITH cte_test AS (
    SELECT * FROM {{ref('interesting_moments')}}
)
select TOP 0 * FROM cte_test

target/compiled/ava_its_data/tests/example.sql

This doesn’t work as Synapse does not allow a single WITH clause inside of

with dbt__CTE__INTERNAL_test as (
WITH cte_test AS (
    SELECT * FROM "june_sql_pool"."ajs_mkt"."interesting_moments"
)
select TOP 0 * FROM cte_test
)select count(*) from dbt__CTE__INTERNAL_test

potential workarounds

haven’t thought much about the best solution yet…

add the wrapped CTE to the end of the CTE chain?

Synapse doesn’t support nested CTEs, but it does supported “chained referencing” of CTEs inside a single WITH clause. So the example below works, but I don’t know how I would conditionally do this only when the data test has a CTE

WITH
cte_test AS (
    SELECT * FROM "june_sql_pool"."ajs_mkt"."interesting_moments"
),
dbt__CTE__INTERNAL_test AS (
    select TOP 0 * FROM cte_test
)
select count(*) from dbt__CTE__INTERNAL_test

create and dispose of a view/ temp table instead of a CTE?

CREATE VIEW  dbt__CTE__INTERNAL_test as
WITH cte_test AS (
    SELECT * FROM "june_sql_pool"."ajs_mkt"."interesting_moments"
)
select TOP 0 * FROM cte_test;

select count(*) from dbt__CTE__INTERNAL_test

DROP VIEW dbt__CTE__INTERNAL_test

[jtcohen6]

@swanderz I had a chance to mess around with this, check out dbt-msft/dbt-synapse#29. I tried implementing your second idea for potential workarounds.

[dataders]

close by #167