The database file was created with an newer version of DuckDB.

Thanks for developing both dbt-duckdb and target-duckdb!

Being new to the whole meltano, dbt, duckdb stack I tried to modify the official meltanto tutorial by replacing postgres with duckdb

cat meltano.yml 
version: 1
default_environment: dev
project_id: b1cb5b90-6956-4199-b7ed-e10092dfb5da
environments:
- name: dev
- name: staging
- name: prod
plugins:
  extractors:
  - name: tap-github
    variant: singer-io
    pip_url: tap-github
    config:
      repository: palaimon/ipyannotator palaimon/fastfm2
      start_date: ' 2016-11-28'
    select:
    - commits.url
    - commits.sha
    - commits.*
  loaders:
  - name: target-jsonl
    variant: andyh1203
    pip_url: target-jsonl
  - name: target-duckdb
    variant: jwills
    pip_url: target-duckdb~=0.4
    config:
      add_metadata_columns: true
      filepath: dump.duckdb
      default_target_schema: tap_github
  transformers:
  - name: dbt-duckdb
    variant: jwills
    pip_url: dbt-core~=1.2.0 dbt-duckdb~=1.2.0
    config:
      path: <my local path>/dump.duckdb

and run into the following error message.

dbt.exceptions.RuntimeException: Runtime Error
  IO Error: Trying to read a database file with version number 39, but we can only read version 38.
  The database file was created with an newer version of DuckDB.
  
  The storage of DuckDB is not yet stable; newer versions of DuckDB cannot read old database files and vice versa.
  The storage will be stabilized when version 1.0 releases.
  
  For now, we recommend that you load the database file in a supported version of DuckDB, and use the EXPORT DATABASE command followed by IMPORT DATABASE on the current version of DuckDB.

The error Trying to read a database file with version number 39, but we can only read version 38. seems pretty clear but I wasn’t able to find a way to pin the dbt-duckdb and target-duckdb version in such a way that they are compatible.

A pointer on how to make dbt-duckdb and target-duckdb work together in meltano would be great.

[jwills]

Jacob’s recommendation is to use target-parquet instead of target-duckdb to get around the issue, and I am thinking that is the best short-term strategy here until I actually build that MDS in a box project, which tbh I am warming up to as I sit here thinking about it 🤔

[matsonj]

@ibayer imo the advantage of meltano is the ecosystem of connectors - ironically the csv extractor is not required to use with dbt-duckdb. Think about the MDS in a box project as more like a frame in which you can swap pieces in and out as required. For me, I will be swapping out the data viz bits as well as some new extractors. Happy to chat here or in twitter DMs or what have you if you have questions!!