question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Standard `table` materialization does not work with ReplicateMergeTree engine

See original GitHub issue

Not sure if this is related to #14 since I didn’t explore the attached PR, but I wanted to call out this problem specifically.

It is my understanding that the way dbt-clickhouse handles table materializations is to:

  1. Rename table (if exists) to {{ this }}__dbt_tmp or something similar (I don’t have the logs in front of me)
  2. Create a new table with name {{ this }}
  3. Drop temp table

With the Replication tables, this results in an error because there already exists a table with the replica path of /clickhouse/tables/{shard}/{database}/{table_name}.

I am currently getting around this using an incremental materialization with a config like:

{{ config(
    materialized="incremental",
    engine="ReplicatedMergeTree('/clickhouse/tables/{database}/{table}', '{replica}')",
    partition_by=[...],
    order_by=[...],
    unique_key=[...],
    pre_hook="truncate table if exists {{ this }} on cluster '...'",
    inserts_only=true
) }}

I’m not sure how we could handle a simple table materialization using the current rename-create-drop strategy since I don’t believe you can update the replica path after creation and I don’t believe there is a random value substitution provided which could potentially allow a replica path like /clickhouse/tables/{shard}/{database}/{table_name}__{random_uuid}

Issue Analytics

  • State:open
  • Created a year ago
  • Reactions:1
  • Comments:5 (2 by maintainers)

github_iconTop GitHub Comments

1reaction
genzgdcommented, Sep 19, 2022

Tests work with with Replicated database engine so explicit support for ReplicatedMergeTree engines will probably not be a priority.

0reactions
mharrisb1commented, Dec 4, 2022

@tema-popov I wasn’t aware of that template. That would be a much better solution than mine. I will test it out and update this issue if it works. Thanks!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Create MATERIALIZED VIEW against ReplicatedMergeTree ...
It never reads/selects a source table. The most used schema is to create at all nodes the same set of tables / MVs....
Read more >
Data Replication | ClickHouse Docs
Replication works at the level of an individual table, not the entire server. A server can store both replicated and non-replicated tables at...
Read more >
Materialized view with replication - clickhouse - Stack Overflow
My case is a source table with Null engine and 2 its' mv with replicatedMergeTree engine. Is it possible? I've read this man...
Read more >
Create materialized views in ClickHouse - Aiven documentation
A materialized view triggers a read on the table engine. The destination of the data (for example, a Merge Tree family table) is...
Read more >
Use materialized views | BigQuery - Google Cloud
Querying does not automatically trigger a materialized refresh. Note: If you delete a base table without first deleting the materialized view, queries over...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found