Standard `table` materialization does not work with ReplicateMergeTree engine
See original GitHub issueNot sure if this is related to #14 since I didn’t explore the attached PR, but I wanted to call out this problem specifically.
It is my understanding that the way dbt-clickhouse handles table materializations is to:
- Rename table (if exists) to
{{ this }}__dbt_tmp
or something similar (I don’t have the logs in front of me) - Create a new table with name
{{ this }}
- Drop temp table
With the Replication tables, this results in an error because there already exists a table with the replica path of /clickhouse/tables/{shard}/{database}/{table_name}
.
I am currently getting around this using an incremental materialization with a config like:
{{ config(
materialized="incremental",
engine="ReplicatedMergeTree('/clickhouse/tables/{database}/{table}', '{replica}')",
partition_by=[...],
order_by=[...],
unique_key=[...],
pre_hook="truncate table if exists {{ this }} on cluster '...'",
inserts_only=true
) }}
I’m not sure how we could handle a simple table materialization using the current rename-create-drop strategy since I don’t believe you can update the replica path after creation and I don’t believe there is a random value substitution provided which could potentially allow a replica path like /clickhouse/tables/{shard}/{database}/{table_name}__{random_uuid}
Issue Analytics
- State:
- Created a year ago
- Reactions:1
- Comments:5 (2 by maintainers)
Top GitHub Comments
Tests work with with Replicated database engine so explicit support for ReplicatedMergeTree engines will probably not be a priority.
@tema-popov I wasn’t aware of that template. That would be a much better solution than mine. I will test it out and update this issue if it works. Thanks!