Running a SQL step after bulk insert into JDBC

Hello,

When using metorikku with a JDBC output, is it possible to add custom SQL after a bulk insert ?

We’d like to run some SQL on the target JDBC to add some indexes on the created tables :

CREATE INDEX … ON …;

First, we thought of adding the « truncate : true » option so that the tables are not dropped but wiped instead, so the indexes would not be deleted at each ingestion.

# job_config.yaml
output:
  jdbc:
    connectionUrl: ****
    options:
      truncate: true

However, this limits the schema evolution : if the schema in the input changes, the job will fail.

I was wondering if it could be possible to keep the default truncate: false, but add some SQL statement (creating indexes) that would be ran after the output is written. This would allow schema evolution while recreating the indexes at each ingestion.

Do you know if there is a way to achieve this with Metorikku ?

I hope the explanation is precise enough. Thank you in advance for your answer !

[lyogev]

I’ve opened this PR for this #414

[lyogev]

@cyrillay would you mind testing #414 and see if it fits what you need?