Rerun dbt seed append data instead of refresh data if seed is stored in external table

Enviroment:

• Spark Standalone cluster 2.4.5 with Thrift JDBC/ODBC server
• dbt 0.18.0
• dbt-spark 0.18.0

I use example project from https://github.com/fishtown-analytics/jaffle_shop. Set location_root of seed so data is stored in external table.

seeds:
  jaffle_shop:
      raw_orders:
        location_root: hdfs:///user/qsbao/

While repeat run dbt seed -s raw_orders, I found the number of records of table raw_orders continues to grow.

Found sql from log in a run:

drop table if exists dbt_alice.raw_orders

create table dbt_alice.raw_orders (id bigint,user_id bigint,order_date date,status string)
location 'hdfs:///user/qsbao/raw_orders'

insert into dbt_alice.raw_orders values
            (%s,%s,%s,%s),(%s,%s,%s,%s),(%s,%s,%s,%s)...

Note that the first sql drop table removes only the metadata and not the data itself as it is an external table.

[jtcohen6]

@mv1742 That’s a really good point! It would be a pretty simple change to accomplish our desired behavior around truncate-then-insert, assuming it works with external tables.

The only thing we’ll need to be careful of is, dbt splits very large seeds into 10k-row chunks. We’d only want the first chunk to execute an insert overwrite, the rest all insert into.

So perhaps in the seed materialization: https://github.com/fishtown-analytics/dbt-spark/blob/dff1b613ddf87e4e72e8a47475bcfd1d55796a5c/dbt/include/spark/macros/materializations/seed.sql#L6-L14

The last line there could become:

insert {{ "overwrite" if loop.first else "into" }} {{ this.render() }} values

[qsbao]

And after the solution is confirmed, I am happy to work on this.