Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

DBT creates multiple tmp tables when updating incremental tables causing serialization isolation violations on Redshift

See original GitHub issue

Describe the bug

We encounter serializable isolation violation errors when updating incremental models. Upon debugging svl_statementtext in Redshift we can see that two temporary tables are created almost in parallel (20 secs apart) reading from the same source table. (The source table is frequently updated via AWS DMS, synchronizing changes from a MySQL DB to our Redshift DWH.) The creation of one of the temp tables is aborted due to concurrent access of the same source table (that has probably been changed in the meantime).

Steps To Reproduce

Model file in models/target_schema/test_table.sql

{{
    config(
        materialized='incremental'
    )
}}

SELECT 
    *
FROM {{ source('source_schema',
'test_table')}}

{% if is_incremental() %}
    where 
        updated > (select max(updated)
    from {{ this }})
{% endif %}

Run script in bin/dbt_run_pipeline.sh

echo "Installing dbt dependencies"
dbt deps
echo "---Running tests---"
echo ""
dbt test --target prod --profiles-dir . --models target_schema --full-refresh
echo ""
echo "---Running transformations---"
dbt run --target prod --profiles-dir . --models target_schema --full-refresh

The source table should be changed frequently in Redshift, e.g. via AWS DMS replication task in CDC mode.

Expected behavior

Create only one temporary table that is then used to update the target table incrementally.

Screenshots and log output

Airflow logs (abbreviated)

[2020-12-17T16:07:41.613000] 16:07:41 | 2 of 86 START incremental model target_schema.test_table [RUN]
...
[2020-12-17T16:08:01.123000] 16:08:01 | 2 of 86 ERROR creating incremental model staging_process_engine.appointment_appointment [ [31mERROR [0m in 19.51s]
...
[2020-12-17T16:16:07.066000] 16:16:07 | Finished running 16 incremental models, 70 table models, 6 hooks in 514.47s.
[2020-12-17T16:16:08.197000] [ 31mCompleted with 14 errors and 0 warnings: [0m
...
[2020-12-17T16:16:08.198000]  [33mDatabase Error in model test_table (models/target_schema/test_table.sql) [0m
[2020-12-17T16:16:08.198000]   1023
[2020-12-17T16:16:08.198000]   DETAIL:  Serializable isolation violation on table - 22123488, transactions forming the cycle are: 280864649, 280864380 (pid:2826)
[2020-12-17T16:16:08.198000]   compiled SQL at target/run/dwh/models/target_schema/test_table.sql

Concurrent transaction logs in Redshift svl_statementtext:

"xid","pid","starttime","endtime","sequence","type","text","aborted"
280864380,2809,2020-12-17 16:07:31,2020-12-17 16:07:31,0,UTILITY,BEGIN,0
280864380,2809,2020-12-17 16:07:31,2020-12-17 16:07:33,0,QUERY,"/* {""app"": ""dbt"", ""dbt_version"": ""0.18.1"", ""profile_name"": ""default"", ""target_name"": ""prod"", ""node_id"": ""model.dwh.test_table""} */\n\n    \n\n  create temporary table\n    ""test_table",0
280864380,2809,2020-12-17 16:07:31,2020-12-17 16:07:33,1,QUERY,"__dbt_tmp20201217160730952436""\n    \n    \n  as (\n    \n\nSELECT \n    *\nFROM ""dev"".""source_schema"".""test_table""\n\n\n    where \n        updated > (select max(updated)",0
280864380,2809,2020-12-17 16:07:31,2020-12-17 16:07:33,2,QUERY,"\n    from ""dev"".""target_schema"".""test_table"")\n\n  );\n",0
280864380,2809,2020-12-17 16:07:33,2020-12-17 16:07:34,0,UTILITY,Analyze test_table__dbt_tmp20201217160730952436,0
280864380,2809,2020-12-17 16:07:33,2020-12-17 16:07:33,0,QUERY,padb_fetch_sample: select * from test_table__dbt_tmp20201217160730952436,0
280864380,2809,2020-12-17 16:07:33,2020-12-17 16:07:34,0,QUERY,padb_fetch_sample: select * from test_table__dbt_tmp20201217160730952436,0
280864380,2809,2020-12-17 16:07:34,2020-12-17 16:07:34,0,UTILITY,BEGIN,0
280864380,2809,2020-12-17 16:07:36,2020-12-17 16:07:46,0,QUERY,"/* {""app"": ""dbt"", ""dbt_version"": ""0.18.1"", ""profile_name"": ""default"", ""target_name"": ""prod"", ""node_id"": ""model.dwh.test_table""} */\n\n      \n\n    insert into ""dev"".""target_schema",0
280864380,2809,2020-12-17 16:07:36,2020-12-17 16:07:46,1,QUERY,".""test_table"" (""id"", ""firstname"", ""lastname"", ""status"", ""starttime"", ""endtime"", ""created"", ""updated""\n       from ""test_table__dbt_tmp20201217160730952436""\n    );\n",0
280864649,2826,2020-12-17 16:07:43,2020-12-17 16:07:43,0,UTILITY,BEGIN,1
280864649,2826,2020-12-17 16:07:44,2020-12-17 16:07:51,0,QUERY,"/* {""app"": ""dbt"", ""dbt_version"": ""0.18.1"", ""profile_name"": ""default"", ""target_name"": ""prod"", ""node_id"": ""model.dwh.test_table""} */\n\n    \n\n  create temporary table\n    ""test_table",1
280864649,2826,2020-12-17 16:07:44,2020-12-17 16:07:51,1,QUERY,"__dbt_tmp20201217160742605919""\n    \n    \n  as (\n    \n\nSELECT \n    *\nFROM ""dev"".""source_schema"".""test_table""\n\n\n    where \n        updated > (select max(updated)",1
280864649,2826,2020-12-17 16:07:44,2020-12-17 16:07:51,2,QUERY,"\n    from ""dev"".""target_schema"".""test_table"")\n\n  );\n",1
280864380,2809,2020-12-17 16:07:46,2020-12-17 16:07:46,0,UTILITY,"/* {""app"": ""dbt"", ""dbt_version"": ""0.18.1"", ""profile_name"": ""default"", ""target_name"": ""prod"", ""node_id"": ""model.dwh.test_table""} */\n\n        GRANT SELECT ON ""dev"".""target_schema",0
280864380,2809,2020-12-17 16:07:46,2020-12-17 16:07:46,1,UTILITY,""".""test_table"" TO GROUP datamining\n",0
280864380,2809,2020-12-17 16:07:46,2020-12-17 16:07:48,0,UTILITY,"/* {""app"": ""dbt"", ""dbt_version"": ""0.18.1"", ""profile_name"": ""default"", ""target_name"": ""prod"", ""node_id"": ""model.dwh.test_table""} */\n\n        GRANT SELECT ON ""dev"".""target_schema",0
280864380,2809,2020-12-17 16:07:46,2020-12-17 16:07:48,1,UTILITY,""".""test_table"" TO etl\n",0
280864380,2809,2020-12-17 16:07:48,2020-12-17 16:07:48,0,UTILITY,"/* {""app"": ""dbt"", ""dbt_version"": ""0.18.1"", ""profile_name"": ""default"", ""target_name"": ""prod"", ""node_id"": ""model.dwh.test_table""} */\n\n        GRANT ALL ON ""dev"".""target_schema"".""",0
280864380,2809,2020-12-17 16:07:48,2020-12-17 16:07:48,1,UTILITY,"test_table"" TO dbt\n",0
280864380,2809,2020-12-17 16:07:48,2020-12-17 16:07:48,0,UTILITY,COMMIT,0
280864649,2826,2020-12-17 16:07:51,2020-12-17 16:07:53,0,UTILITY,Analyze test_table__dbt_tmp20201217160742605919,1
280864649,2826,2020-12-17 16:07:51,2020-12-17 16:07:52,0,QUERY,padb_fetch_sample: select * from test_table__dbt_tmp20201217160742605919,1
280864649,2826,2020-12-17 16:07:52,2020-12-17 16:07:52,0,QUERY,padb_fetch_sample: select * from test_table__dbt_tmp20201217160742605919,1
280864649,2826,2020-12-17 16:07:53,2020-12-17 16:07:53,0,UTILITY,BEGIN,1
280864649,2826,2020-12-17 16:08:01,2020-12-17 16:08:01,0,UTILITY,ROLLBACK,1

System information

Which database are you using dbt with?

  • postgres
  • redshift
  • bigquery
  • snowflake
  • other (specify: ____________)

The output of dbt --version:

installed version: 0.18.1
   latest version: 0.18.1

Up to date!

Plugins:
  - snowflake: 0.18.1
  - postgres: 0.18.1
  - redshift: 0.18.1
  - bigquery: 0.18.1

The operating system you’re using: Ubuntu 16.04

The output of python --version: 3.7.8

Additional context

The issue has been noticed in our production environment where we are triggering DBT runs via an Airflow ECSOperator to launch the runs as AWS Fargate tasks.

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
jtcohen6commented, Jan 4, 2021

Ah, got it, @dankleiman. I was just too excited about solving the mystery 😃

@n0rritt Happy to re-open this one if you’ve been able to debug it further. As I posted in the related Slack thread, I think there has to be something else happening here to cause two simultaneous dbt runs.

1reaction
dankleimancommented, Dec 26, 2020

@jtcohen6 I don’t want to go too off topic with this, but I’m tracking something similar and I think the output here is pointing to the model being executed twice.

The pattern I’m seeing involves two transactions that are colliding to produce the serializable isolation error.

From this query:

SELECT
xid,
starttime,
left(querytxt, 210) as text
FROM
stl_query
WHERE
xid IN (239456355, 239456273)
ORDER BY starttime

You can see:

    xid    |         starttime          |                                                                                                        text                                                                                                        
-----------+----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 239456987 | 2020-12-23 11:26:40.95652  | /* {"app": "dbt", "dbt_version": "0.18.0", "profile_name": "user", "target_name": "default", "node_id": "model.flipside_dbt.terra_prices"} */ create temporary table "terra_prices__dbt_tmp20201223112640827311" c
 239456987 | 2020-12-23 11:26:44.712805 | padb_fetch_sample: select * from terra_prices__dbt_tmp20201223112640827311
 239456987 | 2020-12-23 11:26:44.796814 | padb_fetch_sample: select * from terra_prices__dbt_tmp20201223112640827311
 239457044 | 2020-12-23 11:26:46.494805 | /* {"app": "dbt", "dbt_version": "0.18.0", "profile_name": "user", "target_name": "default", "node_id": "model.flipside_dbt.terra_prices"} */ create temporary table "terra_prices__dbt_tmp20201223112646129211" c
 239456987 | 2020-12-23 11:26:48.614373 | /* {"app": "dbt", "dbt_version": "0.18.0", "profile_name": "user", "target_name": "default", "node_id": "model.flipside_dbt.terra_prices"} */ delete from "prod_redshift_datosphere"."gold"."terra_prices" where (
 239456987 | 2020-12-23 11:26:49.387413 | insert into "prod_redshift_datosphere"."gold"."terra_prices" ("hour", "symbol", "price") ( select "hour", "symbol", "price" from "terra_prices__dbt_tmp20201223112640827311" );
 239457044 | 2020-12-23 11:26:51.268525 | padb_fetch_sample: select * from terra_prices__dbt_tmp20201223112646129211
 239457044 | 2020-12-23 11:26:51.285897 | padb_fetch_sample: select * from terra_prices__dbt_tmp20201223112646129211
(8 rows)

Now, that looks to me like the model model.flipside_dbt.terra_prices is getting built twice?

Transaction 239456987 makes it through the whole process:

  1. create temp table
  2. analyze (artifact of redshift creating temp table)
  3. delete from the target table
  4. insert from the temp table

We hit the conflict on transaction 239457044 before it can perform the merge operations.

~Interesting that the temp table is called terra_prices__dbt_tmp20201223112640827311 in both cases?~ It’s not.

{model}__dbt_tmp20201223112455620113
{model}__dbt_tmp20201223112447167833

EDIT: @jtcohen6 Apologies! This is a red-herring for being an issue with temp table creation in my case…I was able to use the dbt cloud api + debug logs to track down two different runs that were building overlapping models at the same time, causing the conflict above.

Read more comments on GitHub >

github_iconTop Results From Across the Web

DBT creates multiple tmp tables when updating incremental ...
We encounter serializable isolation violation errors when updating incremental models. Upon debugging svl_statementtext in Redshift we can see ...
Read more >
Resolve serializable isolation error in Amazon Redshift - AWS
Force serialization by locking all tables in each session; Use snapshot isolation for concurrent transactions. Resolution. Move operations that ...
Read more >
Serializable Isolation Violation on Table in Redshift Simplified
This article aims to highlight some of the methods that you can leverage to to resolve Serializable Isolation Violation on Table errors in ......
Read more >
Serializable Isolation Violation Errors in Amazon Redshift
INSERT; UPDATE; TRUNCATE; ALTER TABLE; CREATE TABLE; DROP TABLE; TRUNCATE TABLE. This snapshot prompts Redshift to monitor for violations. In ...
Read more >
Trials and Tribulations of Incremental Models - DBT
Incremental models are core to dbt, and are a key solution to how data ... So first it creates a temporary table and...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Review UX for --defer

Next page

Defining vars, folder-level configs outside dbt_project.yml