after updating to version 3.10.0 import data from CSV (empty value) requires NULL

Environment

Liquibase Version: 3.10.0

Liquibase Integration & Version: spring boot

Liquibase Extension(s) & Version: liquibase-hibernate5 3.10.0

Database Vendor & Version: PostgreSQL 12.3

Operating System Type & Version: 18.04.1-Ubuntu - kernel 5.3.0-53-generic x86_64

Description

After upgrading from version 3.9.0 to 3.10.0, loading the same data from CSV, an empty column must be filled with NULL, otherwise the data is not loaded and an error is thrown (description below).

Steps To Reproduce

• Configure a new spring boot project, using liquibase 3.9.0
• Create the main changelog file (master.xml):

<?xml version="1.0" encoding="utf-8"?>
<databaseChangeLog
    xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.9.xsd">

    <property name="now" value="now()" dbms="h2"/>
    <property name="now" value="current_timestamp" dbms="postgresql"/>

    <property name="floatType" value="float4" dbms="postgresql, h2"/>
    <property name="floatType" value="float" dbms="mysql, oracle, mssql, mariadb"/>
    <property name="clobType" value="longvarchar" dbms="h2"/>
    <property name="clobType" value="clob" dbms="mysql, oracle, mssql, mariadb, postgresql"/>
    <property name="uuidType" value="uuid" dbms="h2, postgresql"/>

    <include file="config/liquibase/changelog/20200515191000_added_entity_Order.xml" relativeToChangelogFile="false"/>
    
</databaseChangeLog>

• Create the changelog:

<?xml version="1.0" encoding="utf-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd">

    <changeSet id="20200515191000-1" author="github">
        <createTable tableName="jhi_order">
            <column name="id" type="bigint">
                <constraints primaryKey="true" nullable="false"/>
            </column>
            <column name="selected_pickup_date" type="datetime">
                <constraints nullable="false" />
            </column>
            <column name="effective_pickup_date" type="datetime">
                <constraints nullable="true" />
            </column>
        </createTable>
    </changeSet>

    <changeSet id="20200515191000-1-data" author="github" context="faker">
        <loadData file="config/liquibase/fake-data/jhi_order.csv"
                  separator=";"
                  tableName="jhi_order">
            <column name="id" type="numeric"/>
            <column name="selected_pickup_date" type="datetime"/>
            <column name="effective_pickup_date" type="datetime"/>
        </loadData>
    </changeSet>

</databaseChangeLog>

• Load the following CSV (last column empty - NULL not used):

id;selected_pickup_date;effective_pickup_date
01;2020-09-01T07:00:00.000000;;
02;2020-09-01T07:00:00.000000;;

• Run the Spring boot application
• Database is created and data loaded without any error
• Now stop the application and delete the database
• Upgrade liquibase version to 3.10.0
• Try to launch the application again, you should get the error mentioned before
• Update the CSV data, specifying explicitly NULL for empty columns:

id;selected_pickup_date;effective_pickup_date
01;2020-09-01T07:00:00.000000;NULL;
02;2020-09-01T07:00:00.000000;NULL;

• Launch the application again
• Now you shouldn’t see any error reported

Actual Behavior

I can see that the generated SQL has the empty column, but it fails because it doesn’t replace the empty value with NULL automatically (check the third field - effective_pickup_date):

Caused by: liquibase.exception.DatabaseException: ERROR: syntax error at or near ","
  Position: 213 [Failed SQL: (0) INSERT INTO public.jhi_order (id, selected_pickup_date, effective_pickup_date, calculated_total_volume, status, pickup_point_id, delivery_point_id, user_account_id, vehicle_id) VALUES (01, '2020-09-01 07:00:00', , 40, 'INITIALIZED', 01, 0, 05, 1),(02, '2020-09-01 07:00:00', , 20, 'INITIALIZED', 02, 0, 10, 2);]

Expected/Desired Behavior

In the previous version NULL was not mandatory, leaving the column empty was enough

Additional Context

I don’t know if this behaviour has been intentionally adopted, I’m reporting the problem just to be sure that it’s a desired feature, and not a side-effect of another update.

Honestly I don’t know which SQL was generated by the previous version, it worked, so I didn’t see the SQL output in console.

Cheers, Federico

[nvoxland]

This has been fixed by PR #1431

[molivasdat]

Hi @TautvydasCerniauskas We are still working the list of issues to what can be done to address this issue, whether it was a result of another fix or just a change in how Liquibase processes empty data fields.