Loading CSV data from file ist very slow due to bad logging performance

Environment

Liquibase Version: 4.4.3

Liquibase Integration & Version: spring boot

Liquibase Extension(s) & Version:

Database Vendor & Version: H2 in memory

Operating System Type & Version: Windows 10 Version 21H1 (Build 19043.1165)

Description

I have a project with multiple tables and lots of csv files to fill these tabels with unit-test-data. Some of these tables contain more than 10 columns and over 10.000 rows. Filling one of those tables lasts with the current liquibase version up to 13 seconds. Using an older liquibase version like 3.9.0 speeds up this task to last only round about 272 milliseconds.

Steps To Reproduce

• Create a new maven project with spring-boot-starter-parent version 2.5.4.

<project xmlns="http://maven.apache.org/POM/4.0.0" >
    <modelVersion>4.0.0</modelVersion>
    <groupId>test</groupId>
    <artifactId>liqbase-test</artifactId>
    <version>0.1-SNAPSHOT</version>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.5.4</version>
    </parent>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jdbc</artifactId>
        </dependency>
        <dependency>
            <groupId>org.liquibase</groupId>
            <artifactId>liquibase-core</artifactId>
            <version>4.4.3</version>
            <!-- <version>3.9.0</version> -->
        </dependency>
        <dependency>
            <groupId>com.h2database</groupId>
            <artifactId>h2</artifactId>
        </dependency>
    </dependencies>
</project>

• Create a new Spring Boot test.TestApplication-class:

package test;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class TestApplication {

    public static void main(String[] args) {
        SpringApplication.run(TestApplication.class, args);
    }
}

• Create an /src/main/resources/application.properties-file like this:

spring.datasource.url=jdbc:h2:mem:liquibase-test;MODE=Oracle;SCHEMA=PUBLIC;DB_CLOSE_DELAY=-1
spring.datasource.username=sa
spring.datasource.password=
spring.liquibase.change-log=classpath:/db/changelog.xml

• Create a /src/main/resources/db/changelock.yml-file:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.0.xsd">

    <changeSet id="1" author="chrlembeck">
        <createTable tableName="test-table">
            <column name="id" type="integer" ><constraints primaryKey="true" primaryKeyName="pk_test_table" nullable="false"/></column>
            <column name="col1" type="nvarchar2(80)"><constraints nullable="false"/></column>
            <column name="col2" type="nvarchar2(80)"><constraints nullable="false"/></column>
            <column name="col3" type="nvarchar2(80)"><constraints nullable="false"/></column>
            <column name="col4" type="nvarchar2(80)"><constraints nullable="false"/></column>
            <column name="col5" type="nvarchar2(80)"><constraints nullable="false"/></column>
            <column name="col6" type="nvarchar2(80)"><constraints nullable="false"/></column>
            <column name="col7" type="nvarchar2(80)"><constraints nullable="false"/></column>
            <column name="col8" type="nvarchar2(80)"><constraints nullable="false"/></column>
            <column name="col9" type="nvarchar2(80)"><constraints nullable="false"/></column>
            <column name="col10" type="nvarchar2(80)"><constraints nullable="false"/></column>
        </createTable>
    </changeSet>
    <changeSet id="2" author="chrlembeck">
        <loadData tableName="test-table" file="/db/test-table.csv"/>
    </changeSet>
</databaseChangeLog>

• Create a file named /src/main/resources/db/test-table.csv with the following content

id,col1,col2,col3,col4,col5,col6,col7,col8,col9,col10
1, value 1, value 1, value 1, value 1, value 1, value 1, value 1, value 1, value 1, value 1
2, value 2, value 2, value 2, value 2, value 2, value 2, value 2, value 2, value 2, value 2
3, value 3, value 3, value 3, value 3, value 3, value 3, value 3, value 3, value 3, value 3
  
 ... <code snipped here> ...
 
9998, value 9998, value 9998, value 9998, value 9998, value 9998, value 9998, value 9998, value 9998, value 9998, value 9998
9999, value 9999, value 9999, value 9999, value 9999, value 9999, value 9999, value 9999, value 9999, value 9999, value 9999
10000, value 10000, value 10000, value 10000, value 10000, value 10000, value 10000, value 10000, value 10000, value 10000, value 10000

• Start the application.

Actual Behavior

The Application starts and initializes the database. This process lasts round about 13 seconds:

2021-09-22 16:39:48.361  INFO 7640 --- [           main] liquibase.changelog                      : Table test-table created
2021-09-22 16:39:48.362  INFO 7640 --- [           main] liquibase.changelog                      : ChangeSet db/changelog.xml::1::chrlembeck ran successfully in 5ms
2021-09-22 16:40:01.513  INFO 7640 --- [           main] liquibase.statement                      : Executing JDBC DML batch was successful. 10000 operations were executed, 1 individual UPDATE events were confirmed by the database.
2021-09-22 16:40:01.513  INFO 7640 --- [           main] liquibase.changelog                      : Data loaded from '/db/test-table.csv' into table 'test-table'
2021-09-22 16:40:01.529  INFO 7640 --- [           main] liquibase.changelog                      : ChangeSet db/changelog.xml::2::chrlembeck ran successfully in 13155ms

First inspections with flight recorder and the IntelliJWindows Async Profiler show, that most of the time is being spent by the methods liquibase.logging.core.AbstractLogger#fine(java.lang.String), liquibase.logging.core.AbstractLogger#filterMessage and liquibase.configuration.AbstractMapConfigurationValueProvider#getProvidedValue even if logging is set to level INFO.

Expected/Desired Behavior

If you change the liquibase version in the maven pom file to 3.9.0, the database initialization lasts only 272ms. - That’s a lot faster 😉

2021-09-22 16:43:46.974  INFO 24444 --- [           main] liquibase.changelog.ChangeSet            : Table test-table created
2021-09-22 16:43:46.974  INFO 24444 --- [           main] liquibase.changelog.ChangeSet            : ChangeSet classpath:/db/changelog.xml::1::chrlembeck ran successfully in 2ms
2021-09-22 16:43:46.974  INFO 24444 --- [           main] liquibase.executor.jvm.JdbcExecutor      : SELECT MAX(ORDEREXECUTED) FROM PUBLIC.DATABASECHANGELOG
2021-09-22 16:43:46.977  INFO 24444 --- [           main] liquibase.executor.jvm.JdbcExecutor      : INSERT INTO PUBLIC.DATABASECHANGELOG (ID, AUTHOR, FILENAME, DATEEXECUTED, ORDEREXECUTED, MD5SUM, DESCRIPTION, COMMENTS, EXECTYPE, CONTEXTS, LABELS, LIQUIBASE, DEPLOYMENT_ID) VALUES ('1', 'chrlembeck', 'classpath:/db/changelog.xml', NOW(), 1, '8:4e74a102671cc4ccedba39942b7323d1', 'createTable tableName=test-table', '', 'EXECUTED', NULL, NULL, '3.9.0', '2321826914')
2021-09-22 16:43:47.090  INFO 24444 --- [           main] l.s.ExecutablePreparedStatementBase      : INSERT INTO PUBLIC."test-table"(id, col1, col2, col3, col4, col5, col6, col7, col8, col9, col10) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
2021-09-22 16:43:47.232  INFO 24444 --- [           main] l.s.BatchDmlExecutablePreparedStatement  : Executing JDBC DML batch was successful. 10000 operations were executed, 1 individual UPDATE events were confirmed by the database.
2021-09-22 16:43:47.232  INFO 24444 --- [           main] liquibase.changelog.ChangeSet            : Data loaded from /db/test-table.csv into test-table
2021-09-22 16:43:47.254  INFO 24444 --- [           main] liquibase.changelog.ChangeSet            : ChangeSet classpath:/db/changelog.xml::2::chrlembeck ran successfully in 272ms

Screenshots (if appropriate)

[Nimelrian]

Discovered this issue after profiling bad startup performance of our application even when run against H2.

This is a huge deal for us due to these unnecessary logs taking up to 80% of our application startup time to read 500 lines of CSV, which leads to timeouts and blocked runners in our CI environment.

[kataggart]

@chrlembeck we believe this has been addressed in the more recent version. Can you retry with ver4.12.0? Thanks!