ScopeManager can get confused when dealing with a heavy thread load.

Environment

Liquibase Version: 4.5

Liquibase Integration & Version: gradle

Liquibase Extension(s) & Version:

Database Vendor & Version: Sql Server 2018 at least

Operating System Type & Version: Linux inside Kubernetes pods

Description

We use Liquibase to perform migrations in a service hosted in a Kubernetes cluster, our service is multithreaded and we can updated several thousand databases at a time. We were on Liquibase 3.10 but were getting the occasional concurrentModification exception. I updated our service to utilize version 4.5 because it looked like the newer version handled that exception. I deployed the changes yesterday and saw 3557 migration failures when the processing window opened. This was the error (the scope names were all different)

liquibase.exception.LiquibaseException: java.lang.RuntimeException: Cannot end scope ehscwwaeie when currently at scope otgxsyxdtb at liquibase.Liquibase.runInScope(Liquibase.java:2409) at liquibase.Liquibase.update(Liquibase.java:211) at liquibase.Liquibase.update(Liquibase.java:197) at liquibase.Liquibase.update(Liquibase.java:193) at liquibase.Liquibase.update(Liquibase.java:185) at com.concur.reportdbservice.manager.ReportingSchemaMgr.update(ReportingSchemaMgr.java:66) at com.concur.reportdbservice.service.ReportingSchemaService.updateDatabase(ReportingSchemaService.java:46) at com.concur.reportdbservice.service.MigTaskService.entityMigrate(MigTaskService.java:48) at com.concur.reportdbservice.thread.MigMigrateJob.executeJob(MigMigrateJob.java:27) at com.concur.reportdbservice.thread.Job.run(Job.java:46) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.lang.RuntimeException: Cannot end scope ehscwwaeie when currently at scope otgxsyxdtb at liquibase.Scope.exit(Scope.java:226) at liquibase.Scope.child(Scope.java:188) at liquibase.Scope.child(Scope.java:176) at liquibase.Scope.child(Scope.java:155) at liquibase.Liquibase.runInScope(Liquibase.java:2404)

Steps To Reproduce

We have 8 processing pods that send a request to 3 pods which use liquibase to perform the schema updates. Each of the liquibase service pods runs a service that creates a new thread for each request coming in.

Actual Behavior

The error message was recorded above

Expected/Desired Behavior

We can run a heavy load of schema migrations successfully like was the case in version 3.10

[adamjones0101]

We are also experiencing this issue. We use Liquibase to perform upgrades in a Kubernetes environment on Linux. Our service is multithreaded and we usually upgrade only 10 to 50 databases in parallel yet this error is pervasive. We have used Liquibase version 3.8.5 for years without any issues but all of our attempts to upgrade to various levels of Version 4 have failed with this error. We use liquibase jar files, create a new Liquibase object and call the update method. PLEASE FIX THIS!!! This is a major regression for us. I am representing a fortune 500 company that depends on liquibase to maintain database integrity for our customers.

[chaitanyawho]

I’m facing this issue as well, is there a fix planned for this anytime soon?