Remove_all_stubs migrations is failing

Bug Report

If you take a recent snapshot of staging data and run migrations 1.8.0 on it, you get errors on the “remove_all_stubs” change set. Happened to both Walt and me.

ERROR [2019-12-16 21:53:45,199] liquibase.changelog.ChangeSet: Change Set migrations.1.8.0.xml::remove_all_stubs::agduncan failed.  Error: ERROR: update or delete on table "workflow" violates foreign key constraint "fk_checkerid_with_tool" on table "tool"→  Detail: Key (id)=(7652) is still referenced from table "tool". [Failed SQL: DELETE FROM workflow WHERE mode='STUB']
liquibase.exception.MigrationFailedException: Migration failed for change set migrations.1.8.0.xml::remove_all_stubs::agduncan:
     Reason: liquibase.exception.DatabaseException: ERROR: update or delete on table "workflow" violates foreign key constraint "fk_checkerid_with_tool" on table "tool"
  Detail: Key (id)=(7652) is still referenced from table "tool". [Failed SQL: DELETE FROM workflow WHERE mode='STUB']
	at liquibase.changelog.ChangeSet.execute(ChangeSet.java:637)
	at liquibase.changelog.visitor.UpdateVisitor.visit(UpdateVisitor.java:53)
	at liquibase.changelog.ChangeLogIterator.run(ChangeLogIterator.java:83)
	at liquibase.Liquibase.update(Liquibase.java:202)
	at liquibase.Liquibase.update(Liquibase.java:179)
	at liquibase.Liquibase.update(Liquibase.java:175)
	at liquibase.Liquibase.update(Liquibase.java:168)
	at io.dropwizard.migrations.DbMigrateCommand.run(DbMigrateCommand.java:68)
	at io.dropwizard.migrations.DbCommand.run(DbCommand.java:55)
	at io.dropwizard.migrations.AbstractLiquibaseCommand.run(AbstractLiquibaseCommand.java:68)
	at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
	at io.dropwizard.cli.Cli.run(Cli.java:78)
	at io.dropwizard.Application.run(Application.java:93)
	at io.dockstore.webservice.DockstoreWebserviceApplication.main(DockstoreWebserviceApplication.java:153)
Caused by: liquibase.exception.DatabaseException: ERROR: update or delete on table "workflow" violates foreign key constraint "fk_checkerid_with_tool" on table "tool"
  Detail: Key (id)=(7652) is still referenced from table "tool". [Failed SQL: DELETE FROM workflow WHERE mode='STUB']
	at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:356)
	at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:57)
	at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:125)
	at liquibase.database.AbstractJdbcDatabase.execute(AbstractJdbcDatabase.java:1229)
	at liquibase.database.AbstractJdbcDatabase.executeStatements(AbstractJdbcDatabase.java:1211)
	at liquibase.changelog.ChangeSet.execute(ChangeSet.java:600)
	... 13 more
Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "workflow" violates foreign key constraint "fk_checkerid_with_tool" on table "tool"
  Detail: Key (id)=(7652) is still referenced from table "tool".
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
	at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
	at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:266)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114)
	at com.sun.proxy.$Proxy32.execute(Unknown Source)
	at liquibase.executor.jvm.JdbcExecutor$ExecuteStatementCallback.doInStatement(JdbcExecutor.java:352)
	... 18 more

┆Issue is synchronized with this Jira Story ┆Issue Type: Story ┆Fix Versions: Dockstore 1.8 ┆Sprint: Seabright Sprint 26 Abalone ┆Issue Number: DOCK-1120

[agduncan94]

Confirmed that the migrations work on the prod DB. Looks like there is a single unpublished checker workflow on production that was causing it to fail. All other checker workflows are not stubs.

[denis-yuen]

I guess there are two issues

1. The migration was failing (the workaround is linked above)
2. The migration is failing because of odd data (checker workflows that are stubs).

This is a bit of a special case, I think to remove the review label (which I just added), yeah, just see that things seem to be left in a good state using a prod DB with the new migration.

But the ticket should probably be left open until we have an explanation for the odd stub checker workflows.