Redshift feature not supported

Hello guys,

I’m using Exposed with AWS Amazon Redshift and got exceptions when inserting data.

The code is as simple as that (I’m using a forEach on insert instead of batchInsert because I have another issue with batchInsert which will be issued in the github if this issue is fixed 😃 ) :

object FirstContacts : Table("some.table") {
    val id = varchar("contact_id", 36) references BaseContacts.id
    val appId = varchar("app_id", 128)
    val creationDate = datetime("creationdate")
    val firstname = varchar("firstname", 128).nullable()
    val lastname = varchar("lastname", 128).nullable()
    val phone = varchar("phone", 42).nullable()
    val gender = char("gender").nullable()
    val incomeRaw = integer("income_raw").nullable()
    val income = integer("income").nullable()
    val birthdateRaw = date("birthdate_raw").nullable()
    val birthdate = integer("birthdate").nullable()
}

notExistingFirstContacts.forEach { d ->
    @Suppress("RemoveRedundantQualifierName")
    FirstContacts.insert {
        it[FirstContacts.id] = d.first
        it[FirstContacts.appId] = appId
        it[FirstContacts.firstname] = d.second["firstname"]
        it[FirstContacts.lastname] = d.second["lastname"]
        it[FirstContacts.gender] = d.second["gender"]?.get(0)
    }
}

Here the exception thrown:

SQL: INSERT INTO some.table (app_id, birthdate, birthdate_raw, contact_id, firstname, gender, income, income_raw, lastname, phone) VALUES ('sensitivedata', NULL, NULL, 'sensitivedata', NULL, NULL, NULL, NULL, NULL, NULL)
[main] WARN Exposed - Transaction attempt #0 failed: java.sql.SQLFeatureNotSupportedException: [Amazon][JDBC](10220) Cette fonctionnalité optionnelle n’est pas prise en charge par le pilote.. Statement(s): INSERT INTO some.table (app_id, birthdate, birthdate_raw, contact_id, firstname, gender, income, income_raw, lastname, phone) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
org.jetbrains.exposed.exceptions.ExposedSQLException: java.sql.SQLFeatureNotSupportedException: [Amazon][JDBC](10220) Cette fonctionnalité optionnelle n’est pas prise en charge par le pilote.
SQL: [INSERT INTO some.table (app_id, birthdate, birthdate_raw, contact_id, firstname, gender, income, income_raw, lastname, phone) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]
	at org.jetbrains.exposed.sql.statements.Statement.executeIn$exposed_core(Statement.kt:50)
	at org.jetbrains.exposed.sql.Transaction.exec(Transaction.kt:122)
	at org.jetbrains.exposed.sql.Transaction.exec(Transaction.kt:108)
	at org.jetbrains.exposed.sql.statements.Statement.execute(Statement.kt:29)
	at org.jetbrains.exposed.sql.QueriesKt.insert(Queries.kt:45)
	at io.adfinitas.prometer.importProcess.Import$insertFirstContacts$1.invoke(Import.kt:127)
	at io.adfinitas.prometer.importProcess.Import$insertFirstContacts$1.invoke(Import.kt:27)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt$inTopLevelTransaction$1.invoke(ThreadLocalTransactionManager.kt:156)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt$inTopLevelTransaction$2.invoke(ThreadLocalTransactionManager.kt:197)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.keepAndRestoreTransactionRefAfterRun(ThreadLocalTransactionManager.kt:205)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.inTopLevelTransaction(ThreadLocalTransactionManager.kt:196)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt$transaction$1.invoke(ThreadLocalTransactionManager.kt:134)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.keepAndRestoreTransactionRefAfterRun(ThreadLocalTransactionManager.kt:205)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.transaction(ThreadLocalTransactionManager.kt:106)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.transaction(ThreadLocalTransactionManager.kt:104)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.transaction$default(ThreadLocalTransactionManager.kt:103)
	at io.adfinitas.prometer.importProcess.Import.insertFirstContacts(Import.kt:114)
	at io.adfinitas.prometer.importProcess.Import.access$insertFirstContacts(Import.kt:27)
	at io.adfinitas.prometer.importProcess.Import$doImport$1.invokeSuspend(Import.kt:78)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at arrow.typeclasses.MonadContinuation$bind$$inlined$suspendCoroutineUninterceptedOrReturn$lambda$1.invoke(MonadContinuations.kt:36)
	at arrow.typeclasses.MonadContinuation$bind$$inlined$suspendCoroutineUninterceptedOrReturn$lambda$1.invoke(MonadContinuations.kt:15)
	at arrow.fx.IO$flatMap$1.invoke(IO.kt:620)
	at arrow.fx.IO$flatMap$1.invoke(IO.kt:50)
	at arrow.fx.IORunLoop.loop(IORunLoop.kt:295)
	at arrow.fx.IORunLoop.access$loop(IORunLoop.kt:21)
	at arrow.fx.IORunLoop$RestartCallback.signal(IORunLoop.kt:414)
	at arrow.fx.IORunLoop$RestartCallback.resumeWith(IORunLoop.kt:445)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at kotlin.coroutines.ContinuationKt.startCoroutine(Continuation.kt:114)
	at arrow.fx.IORunLoop$RestartCallback.start(IORunLoop.kt:402)
	at arrow.fx.IORunLoop.loop(IORunLoop.kt:227)
	at arrow.fx.IORunLoop.access$loop(IORunLoop.kt:21)
	at arrow.fx.IORunLoop$RestartCallback.signal(IORunLoop.kt:414)
	at arrow.fx.IORunLoop$RestartCallback.resumeWith(IORunLoop.kt:445)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at kotlin.coroutines.ContinuationKt.startCoroutine(Continuation.kt:114)
	at arrow.fx.IORunLoop$RestartCallback.start(IORunLoop.kt:402)
	at arrow.fx.IORunLoop.loop(IORunLoop.kt:227)
	at arrow.fx.IORunLoop.access$loop(IORunLoop.kt:21)
	at arrow.fx.IORunLoop$RestartCallback.signal(IORunLoop.kt:414)
	at arrow.fx.IORunLoop$RestartCallback.resumeWith(IORunLoop.kt:445)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at kotlin.coroutines.ContinuationKt.startCoroutine(Continuation.kt:114)
	at arrow.fx.IORunLoop$RestartCallback.start(IORunLoop.kt:402)
	at arrow.fx.IORunLoop.loop(IORunLoop.kt:227)
	at arrow.fx.IORunLoop.access$loop(IORunLoop.kt:21)
	at arrow.fx.IORunLoop$suspendAsync$1.invoke(IORunLoop.kt:145)
	at arrow.fx.IORunLoop$suspendAsync$1.invoke(IORunLoop.kt:21)
	at arrow.fx.IORunLoop$RestartCallback.start(IORunLoop.kt:397)
	at arrow.fx.IORunLoop.loop(IORunLoop.kt:218)
	at arrow.fx.IORunLoop.start(IORunLoop.kt:24)
	at arrow.fx.IO.unsafeRunAsync(IO.kt:796)
	at arrow.fx.internal.Platform.unsafeResync(Utils.kt:156)
	at arrow.fx.IO$Async.unsafeRunTimedTotal$arrow_fx(IO.kt:1017)
	at arrow.fx.IO.unsafeRunTimed(IO.kt:862)
	at arrow.fx.IO.unsafeRunSync(IO.kt:851)
	at io.adfinitas.prometer.importProcess.MainKt.handleOrder(Main.kt:53)
	at io.adfinitas.prometer.importProcess.MainKt.main(Main.kt:62)
	at io.adfinitas.prometer.importProcess.MainKt.main(Main.kt)
Caused by: java.sql.SQLFeatureNotSupportedException: [Amazon][JDBC](10220) Cette fonctionnalité optionnelle n’est pas prise en charge par le pilote.
	at com.amazon.exceptions.ExceptionConverter.toSQLException(Unknown Source)
	at com.amazon.jdbc.common.SConnection.prepareStatement(Unknown Source)
	at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:323)
	at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java)
	at org.jetbrains.exposed.sql.statements.jdbc.JdbcConnectionImpl.prepareStatement(JdbcConnectionImpl.kt:54)
	at org.jetbrains.exposed.sql.statements.InsertStatement.prepared(InsertStatement.kt:137)
	at org.jetbrains.exposed.sql.statements.Statement.executeIn$exposed_core(Statement.kt:48)
	... 59 more

My gradle config (I’m using HikariCP over Exposed) :

// HIKARICP
compile "com.zaxxer:HikariCP:3.4.1"

// EXPOSED
compile "org.jetbrains.exposed:exposed-core:0.18.1"
compile "org.jetbrains.exposed:exposed-java-time:0.18.1"
compile "org.jetbrains.exposed:exposed-jdbc:0.18.1"

Do you have some idea how I can bypass that feature or fix that exception ?

Thank you

[Tapac]

@scorsi thank you for spending time on a deep investigation and attempt to workaround Redshift issues. I will leave that issue as is with a hope what Amazon will improve Redshift and jdbc-driver as well.

[scorsi]

We can see that Redshift is too far from standard SQL and from PostgreSQL and may be complicated to be compatible with Exposed by looking at this :

• List of unsupported features
• List of unsupported functions
• List of unsupported data types

I just tried all the tests in SamplesSQL.kt or SamplesDao.kt, but the more issues I fixed, the more other it comes… I made too much changements in the test files and other part of Exposed without being able to make the tests work successfully.

Redshift is definitely too hard to make it compatible with Exposed without a lot of changements and missing features inside the core of Exposed itself :

• CREATE TABLE have to be overridable by the dialect, so RedshiftDialect could remove ON DELETE (CASCADE/RESTRICT)/ON UPDATE (CASCADE/RESTRICT) for foreign keys constraints.
• We should find an alternative to RETURNING (which is too complicated), the only available things being : SELECT MAX(id) FROM schema.table; (which is absolutely unsafe) or to generate ids inside Exposed and don’t let Redshift auto-generate them like using UUIDs (which is not a solution but an alternative).

Btw, I think you’re agree with me, supporting Redshift may be too complicated for Exposed and my PR will not resolve all issues with Redshift… It’s a long hard work which implies a lot of changements… It requires a big workload to support Redshift.

We may close that issue and the PR #714 except if someone has the workload to handle that issue… I will stop using Exposed with Redshift and turn back to standalone/vanilla JDBC.

Thank you