Connection Terminated / Unable to release PostgreSQL advisory lock

Which version and edition of Flyway are you using?

Flyway Community Edition 6.4.2 by Redgate

If this is not the latest version, can you reproduce the issue with the latest one as well? (Many bugs are fixed in newer releases and upgrading will often resolve the issue)

Yes, test also with Flyway Community Edition 7.5.3 by Redgate

Which client are you using? (Command-line, Java API, Maven plugin, Gradle plugin)

CLI

Which database are you using? (Type & version)

Postgres 11.7

Which operating system are you using?

Flyway Docker

FLYWAY_URL='jdbc:postgresql://postgres/postgres?socketTimeout=0&tcpKeepAlive=true&options=-c%20statement_timeout=1h' 

What did you do? (Please include the content causing the issue, any relevant configuration settings, the SQL statement(s) that failed (if any), and the command you ran)

I am running a query to create an index concurrently on a table. When the query runs, flyway opens a connection, locks flyway_schema_history, and places an advisory lock. Then workers start a new connection and execute the script. While the script is running, flyway throw an error. The index is actually there, just the schema is not updated and becomes out of sync.

SQL: CREATE INDEX CONCURRENTLY idx_name ON table_name USING btree (c1, c2);

ERROR: Unable to release PostgreSQL advisory lock
org.postgresql.util.PSQLException: This connection has been closed.
	at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:865)
	at org.postgresql.jdbc.PgConnection.prepareStatement(PgConnection.java:1771)
	at org.postgresql.jdbc.PgConnection.prepareStatement(PgConnection.java:418)
	at org.flywaydb.core.internal.jdbc.JdbcTemplate.prepareStatement(JdbcTemplate.java:344)
	at org.flywaydb.core.internal.jdbc.JdbcTemplate.queryForBoolean(JdbcTemplate.java:170)
	at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.unlock(PostgreSQLAdvisoryLockTemplate.java:99)
	at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:80)
	at org.flywaydb.core.internal.database.postgresql.PostgreSQLConnection.lock(PostgreSQLConnection.java:99)
	at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:140)
	at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:191)
	at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:152)
	at org.flywaydb.core.Flyway$1.execute(Flyway.java:216)
	at org.flywaydb.core.Flyway$1.execute(Flyway.java:165)
	at org.flywaydb.core.Flyway.execute(Flyway.java:570)
	at org.flywaydb.core.Flyway.migrate(Flyway.java:165)
	at org.flywaydb.commandline.Main.executeOperation(Main.java:223)
	at org.flywaydb.commandline.Main.main(Main.java:144)
DEBUG: Memory usage: 15 of 126M
ERROR: Unexpected error
org.flywaydb.core.internal.exception.FlywaySqlException:
Unable to restore connection to its original state
--------------------------------------------------
SQL State  : 08003
Error Code : 0
Message    : This connection has been closed.

	at org.flywaydb.core.internal.database.base.Connection.restoreOriginalState(Connection.java:133)
	at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.exists(JdbcTableSchemaHistory.java:93)
	at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.allAppliedMigrations(JdbcTableSchemaHistory.java:184)
	at org.flywaydb.core.internal.schemahistory.SchemaHistory.calculateInstalledRank(SchemaHistory.java:191)
	at org.flywaydb.core.internal.schemahistory.SchemaHistory.addAppliedMigration(SchemaHistory.java:173)
	at org.flywaydb.core.internal.command.DbMigrate.applyMigrations(DbMigrate.java:345)
	at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:290)
	at org.flywaydb.core.internal.command.DbMigrate.access$100(DbMigrate.java:56)
	at org.flywaydb.core.internal.command.DbMigrate$2.call(DbMigrate.java:194)
	at org.flywaydb.core.internal.command.DbMigrate$2.call(DbMigrate.java:191)
	at org.flywaydb.core.internal.database.postgresql.PostgreSQLAdvisoryLockTemplate.execute(PostgreSQLAdvisoryLockTemplate.java:69)
	at org.flywaydb.core.internal.database.postgresql.PostgreSQLConnection.lock(PostgreSQLConnection.java:99)
	at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:140)
	at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:191)
	at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:152)
	at org.flywaydb.core.Flyway$1.execute(Flyway.java:216)
	at org.flywaydb.core.Flyway$1.execute(Flyway.java:165)
	at org.flywaydb.core.Flyway.execute(Flyway.java:570)
	at org.flywaydb.core.Flyway.migrate(Flyway.java:165)
	at org.flywaydb.commandline.Main.executeOperation(Main.java:223)
	at org.flywaydb.commandline.Main.main(Main.java:144)
Caused by: org.postgresql.util.PSQLException: This connection has been closed.
	at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:865)
	at org.postgresql.jdbc.PgConnection.prepareStatement(PgConnection.java:1771)
	at org.postgresql.jdbc.PgConnection.prepareStatement(PgConnection.java:418)
	at org.flywaydb.core.internal.jdbc.JdbcTemplate.prepareStatement(JdbcTemplate.java:344)
	at org.flywaydb.core.internal.jdbc.JdbcTemplate.execute(JdbcTemplate.java:221)
	at org.flywaydb.core.internal.database.postgresql.PostgreSQLConnection.doRestoreOriginalState(PostgreSQLConnection.java:47)
	at org.flywaydb.core.internal.database.base.Connection.restoreOriginalState(Connection.java:131)
	... 20 more

Looking on the connection lifecycle in PG server, the connection that actually placed the advisory lock is still active and there, also the advisory lock is still there. The queries below run after the error is thrown by flyway

SELECT pid, application_name, wait_event_type, wait_event, state FROM pg_stat_activity;
  pid  |    application_name    | wait_event_type | wait_event | state
-------+------------------------+-----------------+------------+--------
 10822 | PostgreSQL JDBC Driver |                 |            | active

SELECT locktype, pid, mode, granted, fastpath FROM pg_locks WHERE locktype='advisory';
 locktype |  pid  |     mode      | granted | fastpath
----------+-------+---------------+---------+----------
 advisory | 10822 | ExclusiveLock | t       | f

I could reproduce this with using pg_sleep instead of actually creating an index

SELECT
  now();

SELECT
  pg_sleep(600);

SELECT
  now();

It seems flyway is timeout even that the client settings disables socket timeout and enable keep alives.

What did you expect to see?

Migration script runs successfully with no errors and the schema is updated

What did you see instead?

Error thrown by flyway

ERROR: Unable to release PostgreSQL advisory lock
org.postgresql.util.PSQLException: This connection has been closed.

The schema also becomes out of sync

[DoodleBobBuffPants]

There isn’t any way to achieve this at the moment, we will be looking into a solution now that we have a reproduction

[rohityadav225]

@DoodleBobBuffPants Can you please let me know what else i could try ?